public inbox for [email protected]
help / color / mirror / Atom feedBUG #19519: REPACK can fail due to missing chunk for toast value
21+ messages / 8 participants
[nested] [flat]
* BUG #19519: REPACK can fail due to missing chunk for toast value
@ 2026-06-14 07:00 PG Bug reporting form <[email protected]>
2026-06-17 17:03 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: PG Bug reporting form @ 2026-06-14 07:00 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]
The following bug has been logged on the website:
Bug reference: 19519
Logged by: Alexander Lakhin
Email address: [email protected]
PostgreSQL version: 19beta1
Operating system: Ubuntu 24.04
Description:
The following script:
createdb db1
createdb db2
cat << EOF | psql db1
SET default_statistics_target = 1000;
CREATE TABLE t(i int, t text);
INSERT INTO t SELECT 1, g::text FROM generate_series(1, 50000) g;
ANALYZE t;
EOF
cat << EOF | psql db2 &
BEGIN;
CREATE TABLE t(i int);
SELECT pg_sleep(3);
EOF
sleep 1
cat << EOF | psql db1
DROP TABLE t;
VACUUM pg_toast.pg_toast_2619;
REPACK;
EOF
wait
triggers:
ERROR: missing chunk number 0 for toast value 16393 in pg_toast_2619
Reproduced starting from ac58465e0.
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: BUG #19519: REPACK can fail due to missing chunk for toast value
2026-06-14 07:00 BUG #19519: REPACK can fail due to missing chunk for toast value PG Bug reporting form <[email protected]>
@ 2026-06-17 17:03 ` Srinath Reddy Sadipiralla <[email protected]>
2026-07-03 06:45 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Imran Zaheer <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Srinath Reddy Sadipiralla @ 2026-06-17 17:03 UTC (permalink / raw)
To: [email protected]; [email protected]
Hi Alexander,
On Sun, Jun 14, 2026 at 10:37 PM PG Bug reporting form <
[email protected]> wrote:
> The following bug has been logged on the website:
>
> The following script:
> createdb db1
> createdb db2
>
> cat << EOF | psql db1
> SET default_statistics_target = 1000;
> CREATE TABLE t(i int, t text);
> INSERT INTO t SELECT 1, g::text FROM generate_series(1, 50000) g;
> ANALYZE t;
> EOF
>
> cat << EOF | psql db2 &
> BEGIN;
> CREATE TABLE t(i int);
> SELECT pg_sleep(3);
> EOF
>
> sleep 1
>
> cat << EOF | psql db1
> DROP TABLE t;
> VACUUM pg_toast.pg_toast_2619;
> REPACK;
> EOF
> wait
>
> triggers:
> ERROR: missing chunk number 0 for toast value 16393 in pg_toast_2619
>
Thanks for the report and clean reproducer , I can also reproduce this.
I looked into this. It is not REPACK-specific, it reproduces with
plain VACUUM FULL on back branches too, so it predates
the REPACK command. AFAIU The root cause is a disagreement about
the removal horizon between a table rewrite and an independent vacuum
of that table's TOAST relation.
A rewrite (cluster_rel -> copy_table_data, copy_heap_data) preserves live
and RECENTLY_DEAD tuples, and detoasts a preserved tuple's external
values to move them into the new TOAST table.Its removal cutoff comes
from GetOldestNonRemovableTransactionId(), i.e. ComputeXidHorizons(),
which includes the rewriting backend's own snapshot xmin.
The rewrite holds an ordinary MVCC snapshot (taken so index expressions
can be evaluated) and deliberately does not set PROC_IN_VACUUM. So
The rewrite backend's snapshot xmin is the cluster-wide oldest
running xid, which the db2 transaction pins. Since that backend is
in db1 and lacks PROC_IN_VACUUM, its own xmin folds into the
relation's data/catalog horizon -> OldestXmin sits behind the
DROP, so the dead pg_statistic tuple is RECENTLY_DEAD and gets
copied.
The lazy vacuum of pg_toast_2619 sets PROC_IN_VACUUM, which excludes
its own xmin from the horizon, and the db2 transaction, being in another
database,
does not constrain a non-shared relation's horizon. Its cutoff advances
past the DROP
and the chunks are removed.The two operations thus disagree about the same
tuple,
and the rewrite ends up fetching TOAST the lazy vacuum already reclaimed.
Attached Fix:
The rewrite is simply too conservative,AFAIK it's snapshot is only there to
evaluate index expressions against other relations, and it is not a
reader of the rewritten relation's historical rows, so its own xmin
should not hold back that relation's removal horizon. Excluding it
yields exactly the horizon the lazy vacuum uses.
We cannot set PROC_IN_VACUUM on the rewrite, that would broadcast
"ignore my xmin" to every backend and let other vacuums remove tuples in
other relations that the index expressions may need (the documented
reason VACUUM FULL does not set it). The exclusion has to be local to
the rewrite's own horizon computation, so the patch adds
GetOldestNonRemovableTransactionIdForRewrite(): ComputeXidHorizons()
gains a flag to skip the calling backend (and to not publish the
resulting, more-aggressive horizons into the shared GlobalVisState), and
copy_table_data() uses it for OldestXmin.
--
Thanks :)
Srinath Reddy Sadipiralla
EDB: https://www.enterprisedb.com/
Attachments:
[application/octet-stream] v1-0001-Fix-missing-chunk-errors-during-heap-rewrites-by-ign.patch (9.4K, ../../CAFC+b6rLNvDzzYxrfLdq7CWbbouHcuOxi-mjEOW1r3qy2uAPMQ@mail.gmail.com/3-v1-0001-Fix-missing-chunk-errors-during-heap-rewrites-by-ign.patch)
download | inline diff:
From fde0d4c4745d350ebf4b50e361683de532f2c283 Mon Sep 17 00:00:00 2001
From: Srinath Reddy Sadipiralla <[email protected]>
Date: Wed, 17 Jun 2026 20:58:43 +0530
Subject: [PATCH 1/1] Fix "missing chunk" errors during heap rewrites by
ignoring own xmin.
When performing a heap rewrite (CLUSTER, VACUUM FULL, REPACK),
the backend must hold an MVCC snapshot to evaluate
index expressions against other relations. However, this backend does
not use this snapshot to read the historical rows of the relation
currently being rewritten.
Previously, vacuum_get_cutoffs() used GetOldestNonRemovableTransactionId(),
which folds the backend's own xmin into the relation's OldestXmin horizon.
If this xmin is held back by a transaction that cannot even see the target
relation (e.g., a transaction in another database), the rewrite's horizon
becomes artificially conservative.
This creates a race condition with lazy vacuum. Lazy vacuum sets the
PROC_IN_VACUUM flag, allowing it to exclude its own xmin from the global
horizon. Consequently, a concurrent or prior lazy vacuum could calculate
a newer horizon and safely remove a recently-dead tuple's TOAST chunks.
When the heap rewrite later scans the heap, its artificially older horizon
dictates that the recently-dead tuple must be preserved. It attempts to
copy the tuple, fails to find the TOAST chunks, and raises a "missing
chunk number 0 for toast value" error.
This patch fixes the issue by introducing
GetOldestNonRemovableTransactionIdForRewrite(). This function calls
ComputeXidHorizons() with a new `excludeMyself` boolean parameter. By
excluding MyProc from the calculation, the rewrite generates the exact
same horizon that a lazy vacuum would use.
Also, we skip calling GlobalVisUpdateApply() when excludeMyself is true.
This prevents the artificially aggressive local horizon from being cached
in the backend's local GlobalVisState. If this fake horizon were cached,
it would poison the backend's own pruning logic. When the rewrite later
accesses other relations to evaluate index expressions, this poisoned
cache would cause the backend to physically prune tuples that its own
snapshot still strictly requires.
---
src/backend/commands/repack.c | 15 ++++++
src/backend/storage/ipc/procarray.c | 80 ++++++++++++++++++++++++++---
src/include/storage/procarray.h | 1 +
3 files changed, 89 insertions(+), 7 deletions(-)
diff --git a/src/backend/commands/repack.c b/src/backend/commands/repack.c
index ec100e3eef5..93ec67f8aaf 100644
--- a/src/backend/commands/repack.c
+++ b/src/backend/commands/repack.c
@@ -69,6 +69,7 @@
#include "storage/lmgr.h"
#include "storage/predicate.h"
#include "storage/proc.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -1380,6 +1381,20 @@ copy_table_data(Relation NewHeap, Relation OldHeap, Relation OldIndex,
memset(¶ms, 0, sizeof(VacuumParams));
vacuum_get_cutoffs(OldHeap, ¶ms, &cutoffs);
+ /*
+ * vacuum_get_cutoffs() folds our own backend's xmin into OldestXmin. For
+ * a rewrite that is too conservative: the snapshot we hold exists only to
+ * evaluate index expressions against other relations, not to read
+ * OldHeap's historical rows. If our xmin is held back by a transaction
+ * that cannot even see OldHeap (e.g. one in another database), we would
+ * preserve a recently-dead tuple whose TOAST chunks a concurrent or prior
+ * lazy vacuum was free to remove, and then fail with "missing chunk" while
+ * copying it. Recompute OldestXmin ignoring our own backend so it matches
+ * the horizon lazy vacuum uses. This can only move OldestXmin forward, so
+ * the freeze cutoffs derived above remain valid.
+ */
+ cutoffs.OldestXmin = GetOldestNonRemovableTransactionIdForRewrite(OldHeap);
+
/*
* FreezeXid will become the table's new relfrozenxid, and that mustn't go
* backwards, so take the max.
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index f540bb6b23f..e2f1ea08e84 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1671,7 +1671,7 @@ TransactionIdIsInProgress(TransactionId xid)
* code doesn't expect (breaking HOT).
*/
static void
-ComputeXidHorizons(ComputeXidHorizonsResult *h)
+ComputeXidHorizons(ComputeXidHorizonsResult *h, bool excludeMyself)
{
ProcArrayStruct *arrayP = procArray;
TransactionId kaxmin;
@@ -1770,6 +1770,21 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
if (statusFlags & (PROC_IN_VACUUM | PROC_IN_LOGICAL_DECODING))
continue;
+ /*
+ * Optionally ignore our own backend's xmin. A heap rewrite (CLUSTER /
+ * VACUUM FULL / REPACK) holds an MVCC snapshot only so that index
+ * expressions can be evaluated against other relations; it never
+ * reads the historical rows of the relation being rewritten through
+ * that snapshot. Letting our own xmin hold back that relation's
+ * removal horizon makes the rewrite more conservative than the lazy
+ * vacuum that may already have removed a recently-dead tuple's TOAST
+ * chunks, which would then fail with "missing chunk" while copying the
+ * tuple. This mirrors how PROC_IN_VACUUM excludes a lazy vacuum's own
+ * xmin. See GetOldestNonRemovableTransactionIdForRewrite().
+ */
+ if (excludeMyself && proc == MyProc)
+ continue;
+
/* shared tables need to take backends in all databases into account */
h->shared_oldest_nonremovable =
TransactionIdOlder(h->shared_oldest_nonremovable, xmin);
@@ -1898,8 +1913,15 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
TransactionIdPrecedesOrEquals(h->oldest_considered_running,
h->slot_catalog_xmin));
- /* update approximate horizons with the computed horizons */
- GlobalVisUpdateApply(h);
+ /*
+ * Update approximate horizons with the computed horizons. Skip this when
+ * excluding our own xmin: the resulting horizons are more aggressive than
+ * is globally safe and are only valid for the caller's own rewrite of a
+ * single relation, so they must not be published to the shared
+ * GlobalVisState.
+ */
+ if (!excludeMyself)
+ GlobalVisUpdateApply(h);
}
/*
@@ -1945,7 +1967,51 @@ GetOldestNonRemovableTransactionId(Relation rel)
{
ComputeXidHorizonsResult horizons;
- ComputeXidHorizons(&horizons);
+ ComputeXidHorizons(&horizons, false);
+
+ switch (GlobalVisHorizonKindForRel(rel))
+ {
+ case VISHORIZON_SHARED:
+ return horizons.shared_oldest_nonremovable;
+ case VISHORIZON_CATALOG:
+ return horizons.catalog_oldest_nonremovable;
+ case VISHORIZON_DATA:
+ return horizons.data_oldest_nonremovable;
+ case VISHORIZON_TEMP:
+ return horizons.temp_oldest_nonremovable;
+ }
+
+ /* just to prevent compiler warnings */
+ return InvalidTransactionId;
+}
+
+/*
+ * GetOldestNonRemovableTransactionIdForRewrite -- variant of
+ * GetOldestNonRemovableTransactionId() that ignores the calling backend's
+ * own xmin.
+ *
+ * A heap rewrite (CLUSTER / VACUUM FULL / REPACK) must preserve recently-dead
+ * tuples of the relation being rewritten so that other backends' snapshots
+ * still see them afterwards. It is not, however, a reader of that relation's
+ * historical rows itself: the MVCC snapshot it holds exists only to evaluate
+ * index expressions against *other* relations. Including our own xmin in the
+ * horizon therefore makes the rewrite needlessly conservative, and -- when our
+ * xmin is held back by a transaction that cannot even see the relation (for
+ * example one in another database) -- more conservative than the lazy vacuum
+ * which already removed a recently-dead tuple's TOAST chunks, leading to
+ * "missing chunk" errors while copying the tuple. Excluding our own xmin
+ * yields the same horizon a lazy vacuum would use (PROC_IN_VACUUM does the
+ * equivalent for lazy vacuum).
+ *
+ * The caller must hold a lock on rel strong enough that the set of backends
+ * that could read its rows cannot change underneath us.
+ */
+TransactionId
+GetOldestNonRemovableTransactionIdForRewrite(Relation rel)
+{
+ ComputeXidHorizonsResult horizons;
+
+ ComputeXidHorizons(&horizons, true);
switch (GlobalVisHorizonKindForRel(rel))
{
@@ -1974,7 +2040,7 @@ GetOldestTransactionIdConsideredRunning(void)
{
ComputeXidHorizonsResult horizons;
- ComputeXidHorizons(&horizons);
+ ComputeXidHorizons(&horizons, false);
return horizons.oldest_considered_running;
}
@@ -1987,7 +2053,7 @@ GetReplicationHorizons(TransactionId *xmin, TransactionId *catalog_xmin)
{
ComputeXidHorizonsResult horizons;
- ComputeXidHorizons(&horizons);
+ ComputeXidHorizons(&horizons, false);
/*
* Don't want to use shared_oldest_nonremovable here, as that contains the
@@ -4206,7 +4272,7 @@ GlobalVisUpdate(void)
ComputeXidHorizonsResult horizons;
/* updates the horizons as a side-effect */
- ComputeXidHorizons(&horizons);
+ ComputeXidHorizons(&horizons, false);
}
/*
diff --git a/src/include/storage/procarray.h b/src/include/storage/procarray.h
index d718a5b542f..ccfc31cdb5d 100644
--- a/src/include/storage/procarray.h
+++ b/src/include/storage/procarray.h
@@ -51,6 +51,7 @@ extern RunningTransactions GetRunningTransactionData(void);
extern bool TransactionIdIsInProgress(TransactionId xid);
extern TransactionId GetOldestNonRemovableTransactionId(Relation rel);
+extern TransactionId GetOldestNonRemovableTransactionIdForRewrite(Relation rel);
extern TransactionId GetOldestTransactionIdConsideredRunning(void);
extern TransactionId GetOldestActiveTransactionId(bool inCommitOnly,
bool allDbs);
--
2.43.0
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: BUG #19519: REPACK can fail due to missing chunk for toast value
2026-06-14 07:00 BUG #19519: REPACK can fail due to missing chunk for toast value PG Bug reporting form <[email protected]>
2026-06-17 17:03 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
@ 2026-07-03 06:45 ` Imran Zaheer <[email protected]>
2026-07-06 06:36 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Imran Zaheer @ 2026-07-03 06:45 UTC (permalink / raw)
To: Srinath Reddy Sadipiralla <[email protected]>; +Cc: [email protected]; [email protected]
Hi,
The proposed fix (skipping own xmin while computing oldest xmin) did
solve the problem, but while testing this, I found another bug report
showing the same error: "missing chunks."
For example, this report [1] reproduces the same bug with a different
repro attached, but interestingly, this thread discussion also pointed
out that the bug is also reproducible with the CREATE INDEX command.
The following trace was reported by Alexander Lakhin there.
> The backtrace of the heap_fetch_toast_slice() call emitting error is:
> 2024-03-05 17:41:07.786 UTC|law|db10|65e75933.208c6b|ERROR: missing chunk number 0 for toast value 17314 in pg_toast_17289
> 2024-03-05 17:41:07.786 UTC|law|db10|65e75933.208c6b|BACKTRACE:
> heap_fetch_toast_slice at heaptoast.c:784:3
> toast_fetch_datum at detoast.c:379:2
> detoast_external_attr at detoast.c:54:12
> index_form_tuple_context at indextuple.c:111:21
> tuplesort_putindextuplevalues at tuplesortvariants.c:678:15
> _bt_spool at nbtsort.c:537:1
> _bt_build_callback at nbtsort.c:610:24
> heapam_index_build_range_scan at heapam_handler.c:1329:22
> table_index_build_scan at tableam.h:1781:9
> (inlined by) _bt_spools_heapscan at nbtsort.c:483:15
> btbuild at nbtsort.c:329:14
> index_build at index.c:3042:10
> index_create at index.c:1265:3
> DefineIndex at indexcmds.c:1166:3
Index building, together with lazy vacuum, can create similar
conditions as we saw with VACUUM FULL / CLUSTER / REPACK. In
heapam_index_build_range_scan(), we compute the OldestXmin, which
later is used to decide which tuples to index (in repack we use it to
decide which tuples to copy). I think we should also consider fixing
the index building path while addressing this bug.
```
OldestXmin = GetOldestNonRemovableTransactionId(heapRelation);
```
Can we safely use the same fix in the index build path too? Can we use
GetOldestNonRemovableTransactionIdforRewrite or something similar
here? Normal serial index builds use SnapShotAny and concurrent index
builds use MVCC, but the bug only exists in the serial index build
path.
Other than that, after applying your patch, the bug was not
reproducible with either this repro or the other report's repro [1] in
the rewrite path. However, the create index bug is still there. You
can use the following repro as mentioned in the thread [1].
> I've discovered that not only VACUUM FULL can stumble over such missing
> toast values. CREATE INDEX behaves similarly, as the following script
> shows:
> for ((c=1;c<=20;c++)); do createdb db$c; done
>
>
> for ((i=1;i<=100;i++)); do
> echo "iteration $i"
> for ((c=1;c<=20;c++)); do
> echo "\\d sometable" | psql -d db$c >psql-1-$c.log 2>&1 &
> echo "DROP TABLE IF EXISTS tbl;
> CREATE TABLE tbl (i int, t text);
> ALTER TABLE tbl ALTER COLUMN t SET STORAGE EXTERNAL;
> INSERT INTO tbl(i, t) VALUES (1, repeat('1234567890', 250));
> DELETE FROM tbl;
>
>
> VACUUM (VERBOSE) tbl;
> CREATE INDEX t_idx ON tbl(t);
> " | psql -d db$c >psql-2-$c.log 2>&1 &
> done
> wait
> grep 'missing chunk number' server.log && break;
> done
Regards,
Imran Zaheer
[1]: https://www.postgresql.org/message-id/flat/18351-f6e06364b3a2e669%40postgresql.org
On Wed, Jun 17, 2026 at 10:03 PM Srinath Reddy Sadipiralla
<[email protected]> wrote:
>
> Hi Alexander,
>
> On Sun, Jun 14, 2026 at 10:37 PM PG Bug reporting form <[email protected]> wrote:
>>
>> The following bug has been logged on the website:
>>
>> The following script:
>> createdb db1
>> createdb db2
>>
>> cat << EOF | psql db1
>> SET default_statistics_target = 1000;
>> CREATE TABLE t(i int, t text);
>> INSERT INTO t SELECT 1, g::text FROM generate_series(1, 50000) g;
>> ANALYZE t;
>> EOF
>>
>> cat << EOF | psql db2 &
>> BEGIN;
>> CREATE TABLE t(i int);
>> SELECT pg_sleep(3);
>> EOF
>>
>> sleep 1
>>
>> cat << EOF | psql db1
>> DROP TABLE t;
>> VACUUM pg_toast.pg_toast_2619;
>> REPACK;
>> EOF
>> wait
>>
>> triggers:
>> ERROR: missing chunk number 0 for toast value 16393 in pg_toast_2619
>
>
> Thanks for the report and clean reproducer , I can also reproduce this.
>
> I looked into this. It is not REPACK-specific, it reproduces with
> plain VACUUM FULL on back branches too, so it predates
> the REPACK command. AFAIU The root cause is a disagreement about
> the removal horizon between a table rewrite and an independent vacuum
> of that table's TOAST relation.
>
> A rewrite (cluster_rel -> copy_table_data, copy_heap_data) preserves live
> and RECENTLY_DEAD tuples, and detoasts a preserved tuple's external
> values to move them into the new TOAST table.Its removal cutoff comes
> from GetOldestNonRemovableTransactionId(), i.e. ComputeXidHorizons(),
> which includes the rewriting backend's own snapshot xmin.
> The rewrite holds an ordinary MVCC snapshot (taken so index expressions
> can be evaluated) and deliberately does not set PROC_IN_VACUUM. So
> The rewrite backend's snapshot xmin is the cluster-wide oldest
> running xid, which the db2 transaction pins. Since that backend is
> in db1 and lacks PROC_IN_VACUUM, its own xmin folds into the
> relation's data/catalog horizon -> OldestXmin sits behind the
> DROP, so the dead pg_statistic tuple is RECENTLY_DEAD and gets
> copied.
> The lazy vacuum of pg_toast_2619 sets PROC_IN_VACUUM, which excludes
> its own xmin from the horizon, and the db2 transaction, being in another database,
> does not constrain a non-shared relation's horizon. Its cutoff advances past the DROP
> and the chunks are removed.The two operations thus disagree about the same tuple,
> and the rewrite ends up fetching TOAST the lazy vacuum already reclaimed.
>
> Attached Fix:
> The rewrite is simply too conservative,AFAIK it's snapshot is only there to
> evaluate index expressions against other relations, and it is not a
> reader of the rewritten relation's historical rows, so its own xmin
> should not hold back that relation's removal horizon. Excluding it
> yields exactly the horizon the lazy vacuum uses.
> We cannot set PROC_IN_VACUUM on the rewrite, that would broadcast
> "ignore my xmin" to every backend and let other vacuums remove tuples in
> other relations that the index expressions may need (the documented
> reason VACUUM FULL does not set it). The exclusion has to be local to
> the rewrite's own horizon computation, so the patch adds
> GetOldestNonRemovableTransactionIdForRewrite(): ComputeXidHorizons()
> gains a flag to skip the calling backend (and to not publish the
> resulting, more-aggressive horizons into the shared GlobalVisState), and
> copy_table_data() uses it for OldestXmin.
>
> --
> Thanks :)
> Srinath Reddy Sadipiralla
> EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: BUG #19519: REPACK can fail due to missing chunk for toast value
2026-06-14 07:00 BUG #19519: REPACK can fail due to missing chunk for toast value PG Bug reporting form <[email protected]>
2026-06-17 17:03 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-03 06:45 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Imran Zaheer <[email protected]>
@ 2026-07-06 06:36 ` Michael Paquier <[email protected]>
2026-07-06 13:38 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Konstantin Knizhnik <[email protected]>
2026-07-06 18:00 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Alexander Lakhin <[email protected]>
2026-07-06 18:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
0 siblings, 3 replies; 21+ messages in thread
From: Michael Paquier @ 2026-07-06 06:36 UTC (permalink / raw)
To: Imran Zaheer <[email protected]>; +Cc: Srinath Reddy Sadipiralla <[email protected]>; [email protected]; [email protected]
On Fri, Jul 03, 2026 at 11:45:42AM +0500, Imran Zaheer wrote:
> Can we safely use the same fix in the index build path too? Can we use
> GetOldestNonRemovableTransactionIdforRewrite or something similar
> here? Normal serial index builds use SnapShotAny and concurrent index
> builds use MVCC, but the bug only exists in the serial index build
> path.
Yeah, I think that we should be able to safely reuse your ForRewrite()
API for the index build case (SnapshotAny part) as well to make the
horizon computations more conservative, exclusing one own XID as we
do in the lazy VACUUM case.
> Other than that, after applying your patch, the bug was not
> reproducible with either this repro or the other report's repro [1] in
> the rewrite path. However, the create index bug is still there. You
> can use the following repro as mentioned in the thread [1].
The assymetry between the global xmin computation and the per-database
horizon is what's killing us here. All these patterns are complicated
enough that they warrant some tests, even if it is necessary to have
multiple databases to trigger the buggy horizon computations. TAP
tests would be an obvious choice for that. Now I have a set of tricks
in my sleeves to make an isolation test fully deterministic:
- dblink() that opens a transaction to a different database than the
one of the isolation regression database.
- Rename of a TOAST table using allow_system_table_mods, for a VACUUM.
The second one is one of the dirtiest tricks I've done in the past for
a REINDEX CONCURRENTLY test with TOAST. Quite useful. If the backend
side change is reverted, the tests fail.
The attached includes both test suites for all four cases reported
(REPACK, VACUUM, CLUSTER, non-MVCC index builds). We'll need to
remove one of them, just keeping both posted as I am not sure if the
isolation tests would be entirely stable in the buildfarm..
I still need to dive more into the code, but for now this bit stands
out:
vacuum_get_cutoffs(OldHeap, ¶ms, &cutoffs);
+ /*
+ * vacuum_get_cutoffs() folds our own backend's xmin into OldestXmin. For
+ * a rewrite that is too conservative: the snapshot we hold exists only to
+ * evaluate index expressions against other relations, not to read
+ * OldHeap's historical rows. If our xmin is held back by a transaction
+ * that cannot even see OldHeap (e.g. one in another database), we would
+ * preserve a recently-dead tuple whose TOAST chunks a concurrent or prior
+ * lazy vacuum was free to remove, and then fail with "missing chunk" while
+ * copying it. Recompute OldestXmin ignoring our own backend so it matches
+ * the horizon lazy vacuum uses. This can only move OldestXmin forward, so
+ * the freeze cutoffs derived above remain valid.
+ */
+ cutoffs.OldestXmin = GetOldestNonRemovableTransactionIdForRewrite(OldHeap);
In copy_table_data() (for the data copy with repack, cluster, vacuum
full), I think that this is incorrect. For one, this breaks the
FreezeLimit which should always be older than OldestXmin, and this
patch enforces a new recomputation of OldestXmin ignoring most of the
internals of vacuum_get_cutoffs(). That's brittle, to say the least
if we change the way the vacuum cutoffs are calculated in the future.
Thoughts and comments from others are welcome for now. My day is
almost out, at least I got all these scenarios working some tests.
--
Michael
From 36fdec282e5c48866bf915f767502ccc3c6bc0b2 Mon Sep 17 00:00:00 2001
From: Michael Paquier <[email protected]>
Date: Mon, 6 Jul 2026 15:27:54 +0900
Subject: [PATCH v2] Fix "missing chunk" errors during heap rewrites and index
builds
When performing a heap rewrite (CLUSTER, VACUUM FULL, REPACK) or a
non-concurrent index build, the backend holds an MVCC snapshot whose
xmin can be held back by a transaction in another database, since
GetSnapshotData() considers XIDs from all databases.
However, ComputeXidHorizons() only considers same-database backends for
data_oldest_nonremovable. This asymmetry means a concurrent VACUUM
can compute a more aggressive horizon and remove TOAST chunks for a
tuple that the rewrite/index-build backend still considers as recently
dead, leading to "missing chunk number 0 for toast value" errors when
attempting to copy or index a tuple.
This issue is fixed by introducing a ForRewrite() equivalent of
GetOldestNonRemovableTransactionId(). It calls ComputeXidHorizons()
with a new option called "excludeMyself", mirroring how PROC_IN_VACUUM
excludes a lazy VACUUM own's XID to make rewrites more conservative with
the computation of the XID horizon.
This problem has been reported recently by Alexander for REPACK, but it
is a much older issue (tested down to 9.5 for curiosity's sake, but I
suspect that it is much older than that).
Srinath has authored the C portion of the patch, while I got some tests
working for the four problematic cases reported.
XXX: This patch includes both TAP and isolation tests. We should remove
one pattern at the end.
Reported-by: Alexander Lakhin <[email protected]>
Author: Srinath Reddy Sadipiralla <[email protected]>
Co-authored-by: Michael Paquier <[email protected]>
Reviewed-by: Imran Zaheer <[email protected]>
Discussion: https://postgr.es/m/[email protected]
Discussion: https://postgr.es/m/[email protected]
Backpatch-through: 14
---
src/include/storage/procarray.h | 1 +
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/commands/repack.c | 15 ++
src/backend/storage/ipc/procarray.c | 78 +++++-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/055_rewrite_stale_xmin.pl | 124 +++++++++
contrib/dblink/.gitignore | 2 +
contrib/dblink/Makefile | 2 +
.../dblink/expected/rewrite_stale_xmin.out | 237 ++++++++++++++++++
contrib/dblink/meson.build | 5 +
contrib/dblink/specs/rewrite_stale_xmin.spec | 137 ++++++++++
11 files changed, 596 insertions(+), 8 deletions(-)
create mode 100644 src/test/recovery/t/055_rewrite_stale_xmin.pl
create mode 100644 contrib/dblink/expected/rewrite_stale_xmin.out
create mode 100644 contrib/dblink/specs/rewrite_stale_xmin.spec
diff --git a/src/include/storage/procarray.h b/src/include/storage/procarray.h
index d718a5b542f0..ccfc31cdb5d1 100644
--- a/src/include/storage/procarray.h
+++ b/src/include/storage/procarray.h
@@ -51,6 +51,7 @@ extern RunningTransactions GetRunningTransactionData(void);
extern bool TransactionIdIsInProgress(TransactionId xid);
extern TransactionId GetOldestNonRemovableTransactionId(Relation rel);
+extern TransactionId GetOldestNonRemovableTransactionIdForRewrite(Relation rel);
extern TransactionId GetOldestTransactionIdConsideredRunning(void);
extern TransactionId GetOldestActiveTransactionId(bool inCommitOnly,
bool allDbs);
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 2268cc277bce..fb1540784236 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1213,7 +1213,7 @@ heapam_index_build_range_scan(Relation heapRelation,
/* okay to ignore lazy VACUUMs here */
if (!IsBootstrapProcessingMode() && !indexInfo->ii_Concurrent)
- OldestXmin = GetOldestNonRemovableTransactionId(heapRelation);
+ OldestXmin = GetOldestNonRemovableTransactionIdForRewrite(heapRelation);
if (!scan)
{
diff --git a/src/backend/commands/repack.c b/src/backend/commands/repack.c
index faa07d1a118b..1585d67f0b34 100644
--- a/src/backend/commands/repack.c
+++ b/src/backend/commands/repack.c
@@ -71,6 +71,7 @@
#include "storage/lmgr.h"
#include "storage/predicate.h"
#include "storage/proc.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -1390,6 +1391,20 @@ copy_table_data(Relation NewHeap, Relation OldHeap, Relation OldIndex,
memset(¶ms, 0, sizeof(VacuumParams));
vacuum_get_cutoffs(OldHeap, ¶ms, &cutoffs);
+ /*
+ * vacuum_get_cutoffs() folds our own backend's xmin into OldestXmin. For
+ * a rewrite that is too conservative: the snapshot we hold exists only to
+ * evaluate index expressions against other relations, not to read
+ * OldHeap's historical rows. If our xmin is held back by a transaction
+ * that cannot even see OldHeap (e.g. one in another database), we would
+ * preserve a recently-dead tuple whose TOAST chunks a concurrent or prior
+ * lazy vacuum was free to remove, and then fail with "missing chunk"
+ * while copying it. Recompute OldestXmin ignoring our own backend so it
+ * matches the horizon lazy vacuum uses. This can only move OldestXmin
+ * forward, so the freeze cutoffs derived above remain valid.
+ */
+ cutoffs.OldestXmin = GetOldestNonRemovableTransactionIdForRewrite(OldHeap);
+
/*
* FreezeXid will become the table's new relfrozenxid, and that mustn't go
* backwards, so take the max.
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 60336b318036..a11c01eb9d42 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1671,7 +1671,7 @@ TransactionIdIsInProgress(TransactionId xid)
* code doesn't expect (breaking HOT).
*/
static void
-ComputeXidHorizons(ComputeXidHorizonsResult *h)
+ComputeXidHorizons(ComputeXidHorizonsResult *h, bool excludeMyself)
{
ProcArrayStruct *arrayP = procArray;
TransactionId kaxmin;
@@ -1770,6 +1770,22 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
if (statusFlags & (PROC_IN_VACUUM | PROC_IN_LOGICAL_DECODING))
continue;
+ /*
+ * Optionally ignore our own backend's xmin. A relation rewrite holds
+ * an MVCC snapshot only so that index expressions can be evaluated
+ * against other relations; it never reads the historical rows of the
+ * relation being rewritten through that snapshot. Letting our own
+ * xmin hold back that relation's removal horizon makes the rewrite
+ * more conservative than the lazy vacuum that may already have
+ * removed a recently-dead tuple's TOAST chunks, which would then fail
+ * with "missing chunk" while copying the tuple. This mirrors how
+ * PROC_IN_VACUUM excludes a lazy vacuum's own xmin.
+ *
+ * See also GetOldestNonRemovableTransactionIdForRewrite().
+ */
+ if (excludeMyself && proc == MyProc)
+ continue;
+
/* shared tables need to take backends in all databases into account */
h->shared_oldest_nonremovable =
TransactionIdOlder(h->shared_oldest_nonremovable, xmin);
@@ -1898,8 +1914,15 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
TransactionIdPrecedesOrEquals(h->oldest_considered_running,
h->slot_catalog_xmin));
- /* update approximate horizons with the computed horizons */
- GlobalVisUpdateApply(h);
+ /*
+ * Update approximate horizons with the computed horizons. Skip this when
+ * excluding our own xmin: the resulting horizons are more aggressive than
+ * what is globally safe and are only valid for the caller's own rewrite
+ * of a single relation, so they must not be published to the shared
+ * GlobalVisState.
+ */
+ if (!excludeMyself)
+ GlobalVisUpdateApply(h);
}
/*
@@ -1945,7 +1968,48 @@ GetOldestNonRemovableTransactionId(Relation rel)
{
ComputeXidHorizonsResult horizons;
- ComputeXidHorizons(&horizons);
+ ComputeXidHorizons(&horizons, false);
+
+ switch (GlobalVisHorizonKindForRel(rel))
+ {
+ case VISHORIZON_SHARED:
+ return horizons.shared_oldest_nonremovable;
+ case VISHORIZON_CATALOG:
+ return horizons.catalog_oldest_nonremovable;
+ case VISHORIZON_DATA:
+ return horizons.data_oldest_nonremovable;
+ case VISHORIZON_TEMP:
+ return horizons.temp_oldest_nonremovable;
+ }
+
+ /* just to prevent compiler warnings */
+ return InvalidTransactionId;
+}
+
+/*
+ * GetOldestNonRemovableTransactionIdForRewrite -- variant of
+ * GetOldestNonRemovableTransactionId() that ignores the calling backend's
+ * own xmin.
+ *
+ * A rewrite must preserve recently-dead tuples of the relation being
+ * rewritten so that other backends' snapshots still see them afterwards.
+ *
+ * It is not, however, a reader of that relation's
+ * historical rows itself: the MVCC snapshot it holds exists only to evaluate
+ * index expressions against *other* relations. Excluding our own xmin yields
+ * the same horizon a lazy vacuum would use (PROC_IN_VACUUM does the equivalent
+ * for lazy vacuum). That matters for TOAST relations to prevent the removal of
+ * recently-dead tuple's TOAST chunks.
+ *
+ * The caller must hold a lock on rel strong enough that the set of backends
+ * that could read its rows cannot change underneath us.
+ */
+TransactionId
+GetOldestNonRemovableTransactionIdForRewrite(Relation rel)
+{
+ ComputeXidHorizonsResult horizons;
+
+ ComputeXidHorizons(&horizons, true);
switch (GlobalVisHorizonKindForRel(rel))
{
@@ -1974,7 +2038,7 @@ GetOldestTransactionIdConsideredRunning(void)
{
ComputeXidHorizonsResult horizons;
- ComputeXidHorizons(&horizons);
+ ComputeXidHorizons(&horizons, false);
return horizons.oldest_considered_running;
}
@@ -1987,7 +2051,7 @@ GetReplicationHorizons(TransactionId *xmin, TransactionId *catalog_xmin)
{
ComputeXidHorizonsResult horizons;
- ComputeXidHorizons(&horizons);
+ ComputeXidHorizons(&horizons, false);
/*
* Don't want to use shared_oldest_nonremovable here, as that contains the
@@ -4214,7 +4278,7 @@ GlobalVisUpdate(void)
ComputeXidHorizonsResult horizons;
/* updates the horizons as a side-effect */
- ComputeXidHorizons(&horizons);
+ ComputeXidHorizons(&horizons, false);
}
/*
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index ad0d85f41897..7e583ee39a10 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -63,6 +63,7 @@ tests += {
't/052_checkpoint_segment_missing.pl',
't/053_standby_login_event_trigger.pl',
't/054_unlogged_sequence_promotion.pl',
+ 't/055_rewrite_stale_xmin.pl',
],
},
}
diff --git a/src/test/recovery/t/055_rewrite_stale_xmin.pl b/src/test/recovery/t/055_rewrite_stale_xmin.pl
new file mode 100644
index 000000000000..dad95c0cf967
--- /dev/null
+++ b/src/test/recovery/t/055_rewrite_stale_xmin.pl
@@ -0,0 +1,124 @@
+# Copyright (c) 2025-2026, PostgreSQL Global Development Group
+#
+# Test that heap rewrites (CLUSTER, VACUUM FULL, REPACK) and index builds
+# work in the case when the backend's own xmin is held back by a transaction
+# in another database.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+
+my $node = PostgreSQL::Test::Cluster->new('primary');
+$node->init;
+# No Autovacuum, no random interferences.
+$node->append_conf(
+ 'postgresql.conf', qq[
+autovacuum = off
+]);
+$node->start;
+
+# Create two databases.
+$node->safe_psql('postgres', 'CREATE DATABASE db_holder');
+$node->safe_psql('postgres', 'CREATE DATABASE db_target');
+
+# Set up a table with TOAST data. Returns the toast table name.
+sub setup_table
+{
+ my ($tblname, $extra_ddl) = @_;
+ $extra_ddl //= '';
+
+ $node->safe_psql(
+ 'db_target', qq{
+ DROP TABLE IF EXISTS $tblname;
+ CREATE TABLE $tblname (id int PRIMARY KEY, data text)
+ WITH (autovacuum_enabled = off);
+ ALTER TABLE $tblname ALTER COLUMN data SET STORAGE EXTERNAL;
+ INSERT INTO $tblname VALUES (1, repeat('x', 2500));
+ INSERT INTO $tblname VALUES (2, repeat('z', 2500));
+ $extra_ddl
+ });
+
+ my $toast = $node->safe_psql(
+ 'db_target', qq{
+ SELECT 'pg_toast.' || relname FROM pg_class
+ WHERE oid = (SELECT reltoastrelid FROM pg_class
+ WHERE relname = '$tblname');
+ });
+ chomp $toast;
+ return $toast;
+}
+
+# Create precondition state. Requires that $holder is already running with
+# a write XID.
+sub create_missing_toast_state
+{
+ my ($tblname, $toast_table) = @_;
+
+ # Hold xmin in same DB to prevent main tuple removal.
+ my $xmin_holder = $node->background_psql('db_target', on_error_stop => 1);
+ $xmin_holder->query_safe("BEGIN ISOLATION LEVEL REPEATABLE READ");
+ $xmin_holder->query_safe("SELECT 1");
+
+ # Delete the row with TOAST data.
+ $node->safe_psql('db_target', "DELETE FROM $tblname WHERE id = 1");
+
+ # VACUUM main table only. The xmin_holder keeps the dead tuple.
+ $node->safe_psql('db_target', "VACUUM (PROCESS_TOAST false) $tblname");
+
+ # Release same-DB xmin.
+ $xmin_holder->query_safe("COMMIT");
+ $xmin_holder->quit;
+
+ # Vacuum toast table: toast chunks now DEAD and removed.
+ $node->safe_psql('db_target', "VACUUM $toast_table");
+}
+
+# Open a transaction holding an XID before any deletes. This XID will hold
+# back GetSnapshotData() globally, on a different database than the one
+# running the various rewrite operations.
+my $holder = $node->background_psql('db_holder', on_error_stop => 1);
+$holder->query_safe("BEGIN");
+$holder->query_safe("CREATE TEMP TABLE xid_holder(x int)");
+
+# Test 1: CLUSTER
+my $toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+my ($ret, $stdout, $stderr) =
+ $node->psql('db_target', 'CLUSTER rewrite_test USING rewrite_test_pkey');
+is($ret, 0, "CLUSTER succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "CLUSTER: no missing-chunk error");
+
+# Test 2: VACUUM FULL
+$toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+($ret, $stdout, $stderr) =
+ $node->psql('db_target', 'VACUUM FULL rewrite_test');
+is($ret, 0, "VACUUM FULL succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "VACUUM FULL: no missing-chunk error");
+
+# Test 3: REPACK
+$toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+($ret, $stdout, $stderr) = $node->psql('db_target', 'REPACK rewrite_test');
+is($ret, 0, "REPACK succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "REPACK: no missing-chunk error");
+
+# Test 4: CREATE INDEX on toasted column
+$toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+($ret, $stdout, $stderr) = $node->psql('db_target',
+ 'CREATE INDEX rewrite_test_data_idx ON rewrite_test (data)');
+is($ret, 0, "CREATE INDEX succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "CREATE INDEX: no missing-chunk error");
+
+# Cleanup.
+$holder->query_safe("COMMIT");
+$holder->quit;
+
+$node->stop;
+done_testing();
diff --git a/contrib/dblink/.gitignore b/contrib/dblink/.gitignore
index 5dcb3ff97235..33bc8d8ffeb3 100644
--- a/contrib/dblink/.gitignore
+++ b/contrib/dblink/.gitignore
@@ -2,3 +2,5 @@
/log/
/results/
/tmp_check/
+/output_iso/
+/tmp_check_iso/
diff --git a/contrib/dblink/Makefile b/contrib/dblink/Makefile
index fde0b49ddbbd..a0ba41280813 100644
--- a/contrib/dblink/Makefile
+++ b/contrib/dblink/Makefile
@@ -13,6 +13,8 @@ PGFILEDESC = "dblink - connect to other PostgreSQL databases"
REGRESS = dblink
REGRESS_OPTS = --dlpath=$(top_builddir)/src/test/regress
+ISOLATION = rewrite_stale_xmin
+ISOLATION_OPTS = --load-extension=dblink
TAP_TESTS = 1
ifdef USE_PGXS
diff --git a/contrib/dblink/expected/rewrite_stale_xmin.out b/contrib/dblink/expected/rewrite_stale_xmin.out
new file mode 100644
index 000000000000..2e5a7940ea26
--- /dev/null
+++ b/contrib/dblink/expected/rewrite_stale_xmin.out
@@ -0,0 +1,237 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_cluster s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_cluster:
+ CLUSTER rewrite_test USING rewrite_test_pkey;
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_vacuum_full s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_vacuum_full:
+ VACUUM FULL rewrite_test;
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_repack s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_repack:
+ REPACK rewrite_test;
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_create_index s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_create_index:
+ CREATE INDEX rewrite_test_data_idx ON rewrite_test (data);
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
diff --git a/contrib/dblink/meson.build b/contrib/dblink/meson.build
index e2489f41229f..86fbd372b880 100644
--- a/contrib/dblink/meson.build
+++ b/contrib/dblink/meson.build
@@ -36,6 +36,11 @@ tests += {
],
'regress_args': ['--dlpath', meson.project_build_root() / 'src/test/regress'],
},
+ 'isolation': {
+ 'specs': [
+ 'rewrite_stale_xmin',
+ ],
+ },
'tap': {
'tests': [
't/001_auth_scram.pl',
diff --git a/contrib/dblink/specs/rewrite_stale_xmin.spec b/contrib/dblink/specs/rewrite_stale_xmin.spec
new file mode 100644
index 000000000000..fe88e789f854
--- /dev/null
+++ b/contrib/dblink/specs/rewrite_stale_xmin.spec
@@ -0,0 +1,137 @@
+# Test that heap rewrites work when the rewriting backend's own xmin is
+# held back by a transaction in another database.
+
+setup
+{
+ CREATE DATABASE regress_other_db;
+}
+
+setup
+{
+ CREATE EXTENSION IF NOT EXISTS dblink;
+
+ CREATE TABLE rewrite_test (id int PRIMARY KEY, data text)
+ WITH (autovacuum_enabled = off);
+ ALTER TABLE rewrite_test ALTER COLUMN data SET STORAGE EXTERNAL;
+ INSERT INTO rewrite_test VALUES (1, repeat('x', 2500));
+ INSERT INTO rewrite_test VALUES (2, repeat('y', 2500));
+
+ -- Rename the toast table to a known name so we can VACUUM it directly.
+ SET allow_system_table_mods TO true;
+ DO $$DECLARE r record;
+ BEGIN
+ SELECT INTO r reltoastrelid::regclass::text AS table_name
+ FROM pg_class WHERE oid = 'rewrite_test'::regclass;
+ EXECUTE 'ALTER TABLE ' || r.table_name || ' RENAME TO rewrite_test_toast;';
+ END$$;
+ RESET allow_system_table_mods;
+}
+
+teardown
+{
+ DROP DATABASE regress_other_db;
+}
+
+# Session s1: use dblink to hold a transaction in another database.
+# This must start before the DELETE so its XID is older, ensuring
+# GetSnapshotData() holds back any rewrites.
+session s1
+step s1_hold_xmin
+{
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+}
+step s1_release
+{
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+}
+
+# Session s2: holds xmin in the SAME database temporarily, so the
+# main-table vacuum preserves the dead tuple as RECENTLY_DEAD.
+session s2
+step s2_begin
+{
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+}
+step s2_commit { COMMIT; }
+
+# Session s3: performs DML, split vacuum, and the rewrite.
+session s3
+step s3_delete
+{
+ DELETE FROM rewrite_test WHERE id = 1;
+}
+step s3_vacuum_main_only
+{
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+}
+step s3_vacuum_toast
+{
+ VACUUM pg_toast.rewrite_test_toast;
+}
+step s3_cluster
+{
+ CLUSTER rewrite_test USING rewrite_test_pkey;
+}
+step s3_vacuum_full
+{
+ VACUUM FULL rewrite_test;
+}
+step s3_repack
+{
+ REPACK rewrite_test;
+}
+step s3_create_index
+{
+ CREATE INDEX rewrite_test_data_idx ON rewrite_test (data);
+}
+
+teardown { DROP TABLE IF EXISTS rewrite_test; }
+
+# CLUSTER
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_cluster
+ s1_release
+
+# VACUUM FULL
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_vacuum_full
+ s1_release
+
+# REPACK
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_repack
+ s1_release
+
+# CREATE INDEX
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_create_index
+ s1_release
--
2.55.0
Attachments:
[text/plain] v2-0001-Fix-missing-chunk-errors-during-heap-rewrites-and.patch (24.1K, ../../[email protected]/2-v2-0001-Fix-missing-chunk-errors-during-heap-rewrites-and.patch)
download | inline diff:
From 36fdec282e5c48866bf915f767502ccc3c6bc0b2 Mon Sep 17 00:00:00 2001
From: Michael Paquier <[email protected]>
Date: Mon, 6 Jul 2026 15:27:54 +0900
Subject: [PATCH v2] Fix "missing chunk" errors during heap rewrites and index
builds
When performing a heap rewrite (CLUSTER, VACUUM FULL, REPACK) or a
non-concurrent index build, the backend holds an MVCC snapshot whose
xmin can be held back by a transaction in another database, since
GetSnapshotData() considers XIDs from all databases.
However, ComputeXidHorizons() only considers same-database backends for
data_oldest_nonremovable. This asymmetry means a concurrent VACUUM
can compute a more aggressive horizon and remove TOAST chunks for a
tuple that the rewrite/index-build backend still considers as recently
dead, leading to "missing chunk number 0 for toast value" errors when
attempting to copy or index a tuple.
This issue is fixed by introducing a ForRewrite() equivalent of
GetOldestNonRemovableTransactionId(). It calls ComputeXidHorizons()
with a new option called "excludeMyself", mirroring how PROC_IN_VACUUM
excludes a lazy VACUUM own's XID to make rewrites more conservative with
the computation of the XID horizon.
This problem has been reported recently by Alexander for REPACK, but it
is a much older issue (tested down to 9.5 for curiosity's sake, but I
suspect that it is much older than that).
Srinath has authored the C portion of the patch, while I got some tests
working for the four problematic cases reported.
XXX: This patch includes both TAP and isolation tests. We should remove
one pattern at the end.
Reported-by: Alexander Lakhin <[email protected]>
Author: Srinath Reddy Sadipiralla <[email protected]>
Co-authored-by: Michael Paquier <[email protected]>
Reviewed-by: Imran Zaheer <[email protected]>
Discussion: https://postgr.es/m/[email protected]
Discussion: https://postgr.es/m/[email protected]
Backpatch-through: 14
---
src/include/storage/procarray.h | 1 +
src/backend/access/heap/heapam_handler.c | 2 +-
src/backend/commands/repack.c | 15 ++
src/backend/storage/ipc/procarray.c | 78 +++++-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/055_rewrite_stale_xmin.pl | 124 +++++++++
contrib/dblink/.gitignore | 2 +
contrib/dblink/Makefile | 2 +
.../dblink/expected/rewrite_stale_xmin.out | 237 ++++++++++++++++++
contrib/dblink/meson.build | 5 +
contrib/dblink/specs/rewrite_stale_xmin.spec | 137 ++++++++++
11 files changed, 596 insertions(+), 8 deletions(-)
create mode 100644 src/test/recovery/t/055_rewrite_stale_xmin.pl
create mode 100644 contrib/dblink/expected/rewrite_stale_xmin.out
create mode 100644 contrib/dblink/specs/rewrite_stale_xmin.spec
diff --git a/src/include/storage/procarray.h b/src/include/storage/procarray.h
index d718a5b542f0..ccfc31cdb5d1 100644
--- a/src/include/storage/procarray.h
+++ b/src/include/storage/procarray.h
@@ -51,6 +51,7 @@ extern RunningTransactions GetRunningTransactionData(void);
extern bool TransactionIdIsInProgress(TransactionId xid);
extern TransactionId GetOldestNonRemovableTransactionId(Relation rel);
+extern TransactionId GetOldestNonRemovableTransactionIdForRewrite(Relation rel);
extern TransactionId GetOldestTransactionIdConsideredRunning(void);
extern TransactionId GetOldestActiveTransactionId(bool inCommitOnly,
bool allDbs);
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 2268cc277bce..fb1540784236 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -1213,7 +1213,7 @@ heapam_index_build_range_scan(Relation heapRelation,
/* okay to ignore lazy VACUUMs here */
if (!IsBootstrapProcessingMode() && !indexInfo->ii_Concurrent)
- OldestXmin = GetOldestNonRemovableTransactionId(heapRelation);
+ OldestXmin = GetOldestNonRemovableTransactionIdForRewrite(heapRelation);
if (!scan)
{
diff --git a/src/backend/commands/repack.c b/src/backend/commands/repack.c
index faa07d1a118b..1585d67f0b34 100644
--- a/src/backend/commands/repack.c
+++ b/src/backend/commands/repack.c
@@ -71,6 +71,7 @@
#include "storage/lmgr.h"
#include "storage/predicate.h"
#include "storage/proc.h"
+#include "storage/procarray.h"
#include "utils/acl.h"
#include "utils/fmgroids.h"
#include "utils/guc.h"
@@ -1390,6 +1391,20 @@ copy_table_data(Relation NewHeap, Relation OldHeap, Relation OldIndex,
memset(¶ms, 0, sizeof(VacuumParams));
vacuum_get_cutoffs(OldHeap, ¶ms, &cutoffs);
+ /*
+ * vacuum_get_cutoffs() folds our own backend's xmin into OldestXmin. For
+ * a rewrite that is too conservative: the snapshot we hold exists only to
+ * evaluate index expressions against other relations, not to read
+ * OldHeap's historical rows. If our xmin is held back by a transaction
+ * that cannot even see OldHeap (e.g. one in another database), we would
+ * preserve a recently-dead tuple whose TOAST chunks a concurrent or prior
+ * lazy vacuum was free to remove, and then fail with "missing chunk"
+ * while copying it. Recompute OldestXmin ignoring our own backend so it
+ * matches the horizon lazy vacuum uses. This can only move OldestXmin
+ * forward, so the freeze cutoffs derived above remain valid.
+ */
+ cutoffs.OldestXmin = GetOldestNonRemovableTransactionIdForRewrite(OldHeap);
+
/*
* FreezeXid will become the table's new relfrozenxid, and that mustn't go
* backwards, so take the max.
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 60336b318036..a11c01eb9d42 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -1671,7 +1671,7 @@ TransactionIdIsInProgress(TransactionId xid)
* code doesn't expect (breaking HOT).
*/
static void
-ComputeXidHorizons(ComputeXidHorizonsResult *h)
+ComputeXidHorizons(ComputeXidHorizonsResult *h, bool excludeMyself)
{
ProcArrayStruct *arrayP = procArray;
TransactionId kaxmin;
@@ -1770,6 +1770,22 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
if (statusFlags & (PROC_IN_VACUUM | PROC_IN_LOGICAL_DECODING))
continue;
+ /*
+ * Optionally ignore our own backend's xmin. A relation rewrite holds
+ * an MVCC snapshot only so that index expressions can be evaluated
+ * against other relations; it never reads the historical rows of the
+ * relation being rewritten through that snapshot. Letting our own
+ * xmin hold back that relation's removal horizon makes the rewrite
+ * more conservative than the lazy vacuum that may already have
+ * removed a recently-dead tuple's TOAST chunks, which would then fail
+ * with "missing chunk" while copying the tuple. This mirrors how
+ * PROC_IN_VACUUM excludes a lazy vacuum's own xmin.
+ *
+ * See also GetOldestNonRemovableTransactionIdForRewrite().
+ */
+ if (excludeMyself && proc == MyProc)
+ continue;
+
/* shared tables need to take backends in all databases into account */
h->shared_oldest_nonremovable =
TransactionIdOlder(h->shared_oldest_nonremovable, xmin);
@@ -1898,8 +1914,15 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h)
TransactionIdPrecedesOrEquals(h->oldest_considered_running,
h->slot_catalog_xmin));
- /* update approximate horizons with the computed horizons */
- GlobalVisUpdateApply(h);
+ /*
+ * Update approximate horizons with the computed horizons. Skip this when
+ * excluding our own xmin: the resulting horizons are more aggressive than
+ * what is globally safe and are only valid for the caller's own rewrite
+ * of a single relation, so they must not be published to the shared
+ * GlobalVisState.
+ */
+ if (!excludeMyself)
+ GlobalVisUpdateApply(h);
}
/*
@@ -1945,7 +1968,48 @@ GetOldestNonRemovableTransactionId(Relation rel)
{
ComputeXidHorizonsResult horizons;
- ComputeXidHorizons(&horizons);
+ ComputeXidHorizons(&horizons, false);
+
+ switch (GlobalVisHorizonKindForRel(rel))
+ {
+ case VISHORIZON_SHARED:
+ return horizons.shared_oldest_nonremovable;
+ case VISHORIZON_CATALOG:
+ return horizons.catalog_oldest_nonremovable;
+ case VISHORIZON_DATA:
+ return horizons.data_oldest_nonremovable;
+ case VISHORIZON_TEMP:
+ return horizons.temp_oldest_nonremovable;
+ }
+
+ /* just to prevent compiler warnings */
+ return InvalidTransactionId;
+}
+
+/*
+ * GetOldestNonRemovableTransactionIdForRewrite -- variant of
+ * GetOldestNonRemovableTransactionId() that ignores the calling backend's
+ * own xmin.
+ *
+ * A rewrite must preserve recently-dead tuples of the relation being
+ * rewritten so that other backends' snapshots still see them afterwards.
+ *
+ * It is not, however, a reader of that relation's
+ * historical rows itself: the MVCC snapshot it holds exists only to evaluate
+ * index expressions against *other* relations. Excluding our own xmin yields
+ * the same horizon a lazy vacuum would use (PROC_IN_VACUUM does the equivalent
+ * for lazy vacuum). That matters for TOAST relations to prevent the removal of
+ * recently-dead tuple's TOAST chunks.
+ *
+ * The caller must hold a lock on rel strong enough that the set of backends
+ * that could read its rows cannot change underneath us.
+ */
+TransactionId
+GetOldestNonRemovableTransactionIdForRewrite(Relation rel)
+{
+ ComputeXidHorizonsResult horizons;
+
+ ComputeXidHorizons(&horizons, true);
switch (GlobalVisHorizonKindForRel(rel))
{
@@ -1974,7 +2038,7 @@ GetOldestTransactionIdConsideredRunning(void)
{
ComputeXidHorizonsResult horizons;
- ComputeXidHorizons(&horizons);
+ ComputeXidHorizons(&horizons, false);
return horizons.oldest_considered_running;
}
@@ -1987,7 +2051,7 @@ GetReplicationHorizons(TransactionId *xmin, TransactionId *catalog_xmin)
{
ComputeXidHorizonsResult horizons;
- ComputeXidHorizons(&horizons);
+ ComputeXidHorizons(&horizons, false);
/*
* Don't want to use shared_oldest_nonremovable here, as that contains the
@@ -4214,7 +4278,7 @@ GlobalVisUpdate(void)
ComputeXidHorizonsResult horizons;
/* updates the horizons as a side-effect */
- ComputeXidHorizons(&horizons);
+ ComputeXidHorizons(&horizons, false);
}
/*
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index ad0d85f41897..7e583ee39a10 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -63,6 +63,7 @@ tests += {
't/052_checkpoint_segment_missing.pl',
't/053_standby_login_event_trigger.pl',
't/054_unlogged_sequence_promotion.pl',
+ 't/055_rewrite_stale_xmin.pl',
],
},
}
diff --git a/src/test/recovery/t/055_rewrite_stale_xmin.pl b/src/test/recovery/t/055_rewrite_stale_xmin.pl
new file mode 100644
index 000000000000..dad95c0cf967
--- /dev/null
+++ b/src/test/recovery/t/055_rewrite_stale_xmin.pl
@@ -0,0 +1,124 @@
+# Copyright (c) 2025-2026, PostgreSQL Global Development Group
+#
+# Test that heap rewrites (CLUSTER, VACUUM FULL, REPACK) and index builds
+# work in the case when the backend's own xmin is held back by a transaction
+# in another database.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+
+my $node = PostgreSQL::Test::Cluster->new('primary');
+$node->init;
+# No Autovacuum, no random interferences.
+$node->append_conf(
+ 'postgresql.conf', qq[
+autovacuum = off
+]);
+$node->start;
+
+# Create two databases.
+$node->safe_psql('postgres', 'CREATE DATABASE db_holder');
+$node->safe_psql('postgres', 'CREATE DATABASE db_target');
+
+# Set up a table with TOAST data. Returns the toast table name.
+sub setup_table
+{
+ my ($tblname, $extra_ddl) = @_;
+ $extra_ddl //= '';
+
+ $node->safe_psql(
+ 'db_target', qq{
+ DROP TABLE IF EXISTS $tblname;
+ CREATE TABLE $tblname (id int PRIMARY KEY, data text)
+ WITH (autovacuum_enabled = off);
+ ALTER TABLE $tblname ALTER COLUMN data SET STORAGE EXTERNAL;
+ INSERT INTO $tblname VALUES (1, repeat('x', 2500));
+ INSERT INTO $tblname VALUES (2, repeat('z', 2500));
+ $extra_ddl
+ });
+
+ my $toast = $node->safe_psql(
+ 'db_target', qq{
+ SELECT 'pg_toast.' || relname FROM pg_class
+ WHERE oid = (SELECT reltoastrelid FROM pg_class
+ WHERE relname = '$tblname');
+ });
+ chomp $toast;
+ return $toast;
+}
+
+# Create precondition state. Requires that $holder is already running with
+# a write XID.
+sub create_missing_toast_state
+{
+ my ($tblname, $toast_table) = @_;
+
+ # Hold xmin in same DB to prevent main tuple removal.
+ my $xmin_holder = $node->background_psql('db_target', on_error_stop => 1);
+ $xmin_holder->query_safe("BEGIN ISOLATION LEVEL REPEATABLE READ");
+ $xmin_holder->query_safe("SELECT 1");
+
+ # Delete the row with TOAST data.
+ $node->safe_psql('db_target', "DELETE FROM $tblname WHERE id = 1");
+
+ # VACUUM main table only. The xmin_holder keeps the dead tuple.
+ $node->safe_psql('db_target', "VACUUM (PROCESS_TOAST false) $tblname");
+
+ # Release same-DB xmin.
+ $xmin_holder->query_safe("COMMIT");
+ $xmin_holder->quit;
+
+ # Vacuum toast table: toast chunks now DEAD and removed.
+ $node->safe_psql('db_target', "VACUUM $toast_table");
+}
+
+# Open a transaction holding an XID before any deletes. This XID will hold
+# back GetSnapshotData() globally, on a different database than the one
+# running the various rewrite operations.
+my $holder = $node->background_psql('db_holder', on_error_stop => 1);
+$holder->query_safe("BEGIN");
+$holder->query_safe("CREATE TEMP TABLE xid_holder(x int)");
+
+# Test 1: CLUSTER
+my $toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+my ($ret, $stdout, $stderr) =
+ $node->psql('db_target', 'CLUSTER rewrite_test USING rewrite_test_pkey');
+is($ret, 0, "CLUSTER succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "CLUSTER: no missing-chunk error");
+
+# Test 2: VACUUM FULL
+$toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+($ret, $stdout, $stderr) =
+ $node->psql('db_target', 'VACUUM FULL rewrite_test');
+is($ret, 0, "VACUUM FULL succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "VACUUM FULL: no missing-chunk error");
+
+# Test 3: REPACK
+$toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+($ret, $stdout, $stderr) = $node->psql('db_target', 'REPACK rewrite_test');
+is($ret, 0, "REPACK succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "REPACK: no missing-chunk error");
+
+# Test 4: CREATE INDEX on toasted column
+$toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+($ret, $stdout, $stderr) = $node->psql('db_target',
+ 'CREATE INDEX rewrite_test_data_idx ON rewrite_test (data)');
+is($ret, 0, "CREATE INDEX succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "CREATE INDEX: no missing-chunk error");
+
+# Cleanup.
+$holder->query_safe("COMMIT");
+$holder->quit;
+
+$node->stop;
+done_testing();
diff --git a/contrib/dblink/.gitignore b/contrib/dblink/.gitignore
index 5dcb3ff97235..33bc8d8ffeb3 100644
--- a/contrib/dblink/.gitignore
+++ b/contrib/dblink/.gitignore
@@ -2,3 +2,5 @@
/log/
/results/
/tmp_check/
+/output_iso/
+/tmp_check_iso/
diff --git a/contrib/dblink/Makefile b/contrib/dblink/Makefile
index fde0b49ddbbd..a0ba41280813 100644
--- a/contrib/dblink/Makefile
+++ b/contrib/dblink/Makefile
@@ -13,6 +13,8 @@ PGFILEDESC = "dblink - connect to other PostgreSQL databases"
REGRESS = dblink
REGRESS_OPTS = --dlpath=$(top_builddir)/src/test/regress
+ISOLATION = rewrite_stale_xmin
+ISOLATION_OPTS = --load-extension=dblink
TAP_TESTS = 1
ifdef USE_PGXS
diff --git a/contrib/dblink/expected/rewrite_stale_xmin.out b/contrib/dblink/expected/rewrite_stale_xmin.out
new file mode 100644
index 000000000000..2e5a7940ea26
--- /dev/null
+++ b/contrib/dblink/expected/rewrite_stale_xmin.out
@@ -0,0 +1,237 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_cluster s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_cluster:
+ CLUSTER rewrite_test USING rewrite_test_pkey;
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_vacuum_full s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_vacuum_full:
+ VACUUM FULL rewrite_test;
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_repack s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_repack:
+ REPACK rewrite_test;
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_create_index s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_create_index:
+ CREATE INDEX rewrite_test_data_idx ON rewrite_test (data);
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
diff --git a/contrib/dblink/meson.build b/contrib/dblink/meson.build
index e2489f41229f..86fbd372b880 100644
--- a/contrib/dblink/meson.build
+++ b/contrib/dblink/meson.build
@@ -36,6 +36,11 @@ tests += {
],
'regress_args': ['--dlpath', meson.project_build_root() / 'src/test/regress'],
},
+ 'isolation': {
+ 'specs': [
+ 'rewrite_stale_xmin',
+ ],
+ },
'tap': {
'tests': [
't/001_auth_scram.pl',
diff --git a/contrib/dblink/specs/rewrite_stale_xmin.spec b/contrib/dblink/specs/rewrite_stale_xmin.spec
new file mode 100644
index 000000000000..fe88e789f854
--- /dev/null
+++ b/contrib/dblink/specs/rewrite_stale_xmin.spec
@@ -0,0 +1,137 @@
+# Test that heap rewrites work when the rewriting backend's own xmin is
+# held back by a transaction in another database.
+
+setup
+{
+ CREATE DATABASE regress_other_db;
+}
+
+setup
+{
+ CREATE EXTENSION IF NOT EXISTS dblink;
+
+ CREATE TABLE rewrite_test (id int PRIMARY KEY, data text)
+ WITH (autovacuum_enabled = off);
+ ALTER TABLE rewrite_test ALTER COLUMN data SET STORAGE EXTERNAL;
+ INSERT INTO rewrite_test VALUES (1, repeat('x', 2500));
+ INSERT INTO rewrite_test VALUES (2, repeat('y', 2500));
+
+ -- Rename the toast table to a known name so we can VACUUM it directly.
+ SET allow_system_table_mods TO true;
+ DO $$DECLARE r record;
+ BEGIN
+ SELECT INTO r reltoastrelid::regclass::text AS table_name
+ FROM pg_class WHERE oid = 'rewrite_test'::regclass;
+ EXECUTE 'ALTER TABLE ' || r.table_name || ' RENAME TO rewrite_test_toast;';
+ END$$;
+ RESET allow_system_table_mods;
+}
+
+teardown
+{
+ DROP DATABASE regress_other_db;
+}
+
+# Session s1: use dblink to hold a transaction in another database.
+# This must start before the DELETE so its XID is older, ensuring
+# GetSnapshotData() holds back any rewrites.
+session s1
+step s1_hold_xmin
+{
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+}
+step s1_release
+{
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+}
+
+# Session s2: holds xmin in the SAME database temporarily, so the
+# main-table vacuum preserves the dead tuple as RECENTLY_DEAD.
+session s2
+step s2_begin
+{
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+}
+step s2_commit { COMMIT; }
+
+# Session s3: performs DML, split vacuum, and the rewrite.
+session s3
+step s3_delete
+{
+ DELETE FROM rewrite_test WHERE id = 1;
+}
+step s3_vacuum_main_only
+{
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+}
+step s3_vacuum_toast
+{
+ VACUUM pg_toast.rewrite_test_toast;
+}
+step s3_cluster
+{
+ CLUSTER rewrite_test USING rewrite_test_pkey;
+}
+step s3_vacuum_full
+{
+ VACUUM FULL rewrite_test;
+}
+step s3_repack
+{
+ REPACK rewrite_test;
+}
+step s3_create_index
+{
+ CREATE INDEX rewrite_test_data_idx ON rewrite_test (data);
+}
+
+teardown { DROP TABLE IF EXISTS rewrite_test; }
+
+# CLUSTER
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_cluster
+ s1_release
+
+# VACUUM FULL
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_vacuum_full
+ s1_release
+
+# REPACK
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_repack
+ s1_release
+
+# CREATE INDEX
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_create_index
+ s1_release
--
2.55.0
[application/pgp-signature] signature.asc (833B, ../../[email protected]/3-signature.asc)
download
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: BUG #19519: REPACK can fail due to missing chunk for toast value
2026-06-14 07:00 BUG #19519: REPACK can fail due to missing chunk for toast value PG Bug reporting form <[email protected]>
2026-06-17 17:03 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-03 06:45 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Imran Zaheer <[email protected]>
2026-07-06 06:36 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
@ 2026-07-06 13:38 ` Konstantin Knizhnik <[email protected]>
2 siblings, 0 replies; 21+ messages in thread
From: Konstantin Knizhnik @ 2026-07-06 13:38 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; Imran Zaheer <[email protected]>; +Cc: Srinath Reddy Sadipiralla <[email protected]>; [email protected]; [email protected]
On 06/07/2026 9:36 AM, Michael Paquier wrote:
> On Fri, Jul 03, 2026 at 11:45:42AM +0500, Imran Zaheer wrote:
>> Can we safely use the same fix in the index build path too? Can we use
>> GetOldestNonRemovableTransactionIdforRewrite or something similar
>> here? Normal serial index builds use SnapShotAny and concurrent index
>> builds use MVCC, but the bug only exists in the serial index build
>> path.
> Yeah, I think that we should be able to safely reuse your ForRewrite()
> API for the index build case (SnapshotAny part) as well to make the
> horizon computations more conservative, exclusing one own XID as we
> do in the lazy VACUUM case.
>
>> Other than that, after applying your patch, the bug was not
>> reproducible with either this repro or the other report's repro [1] in
>> the rewrite path. However, the create index bug is still there. You
>> can use the following repro as mentioned in the thread [1].
> The assymetry between the global xmin computation and the per-database
> horizon is what's killing us here. All these patterns are complicated
> enough that they warrant some tests, even if it is necessary to have
> multiple databases to trigger the buggy horizon computations. TAP
> tests would be an obvious choice for that. Now I have a set of tricks
> in my sleeves to make an isolation test fully deterministic:
> - dblink() that opens a transaction to a different database than the
> one of the isolation regression database.
> - Rename of a TOAST table using allow_system_table_mods, for a VACUUM.
>
> The second one is one of the dirtiest tricks I've done in the past for
> a REINDEX CONCURRENTLY test with TOAST. Quite useful. If the backend
> side change is reverted, the tests fail.
>
> The attached includes both test suites for all four cases reported
> (REPACK, VACUUM, CLUSTER, non-MVCC index builds). We'll need to
> remove one of them, just keeping both posted as I am not sure if the
> isolation tests would be entirely stable in the buildfarm..
>
> I still need to dive more into the code, but for now this bit stands
> out:
> vacuum_get_cutoffs(OldHeap, ¶ms, &cutoffs);
>
> + /*
> + * vacuum_get_cutoffs() folds our own backend's xmin into OldestXmin. For
> + * a rewrite that is too conservative: the snapshot we hold exists only to
> + * evaluate index expressions against other relations, not to read
> + * OldHeap's historical rows. If our xmin is held back by a transaction
> + * that cannot even see OldHeap (e.g. one in another database), we would
> + * preserve a recently-dead tuple whose TOAST chunks a concurrent or prior
> + * lazy vacuum was free to remove, and then fail with "missing chunk" while
> + * copying it. Recompute OldestXmin ignoring our own backend so it matches
> + * the horizon lazy vacuum uses. This can only move OldestXmin forward, so
> + * the freeze cutoffs derived above remain valid.
> + */
> + cutoffs.OldestXmin = GetOldestNonRemovableTransactionIdForRewrite(OldHeap);
>
> In copy_table_data() (for the data copy with repack, cluster, vacuum
> full), I think that this is incorrect. For one, this breaks the
> FreezeLimit which should always be older than OldestXmin, and this
> patch enforces a new recomputation of OldestXmin ignoring most of the
> internals of vacuum_get_cutoffs(). That's brittle, to say the least
> if we change the way the vacuum cutoffs are calculated in the future.
>
> Thoughts and comments from others are welcome for now. My day is
> almost out, at least I got all these scenarios working some tests.
> --
> Michael
I wonder if we also need to replace GetOldestNonRemovableTransactionId
with GetOldestNonRemovableTransactionIdForRewrite in vacuum.c:
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index be863db81cb..aaa7e6eeeaa 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -1149,7 +1149,7 @@ vacuum_get_cutoffs(Relation rel, const
VacuumParams *params,
* that only one vacuum process can be working on a particular
table at
* any time, and that each vacuum is always an independent
transaction.
*/
- cutoffs->OldestXmin = GetOldestNonRemovableTransactionId(rel);
+ cutoffs->OldestXmin =
GetOldestNonRemovableTransactionIdForRewrite(rel);
Assert(TransactionIdIsNormal(cutoffs->OldestXmin));
For some reasons the problem is not reproduced with current master, but
when I applied your patch to PG18 and run repro script with REPACK
replaced with VACUUM FULL, then I still get missing chunk error.
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: BUG #19519: REPACK can fail due to missing chunk for toast value
2026-06-14 07:00 BUG #19519: REPACK can fail due to missing chunk for toast value PG Bug reporting form <[email protected]>
2026-06-17 17:03 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-03 06:45 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Imran Zaheer <[email protected]>
2026-07-06 06:36 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
@ 2026-07-06 18:00 ` Alexander Lakhin <[email protected]>
2 siblings, 0 replies; 21+ messages in thread
From: Alexander Lakhin @ 2026-07-06 18:00 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Imran Zaheer <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; [email protected]
Hello Michael,
06.07.2026 09:36, Michael Paquier wrote:
> Thoughts and comments from others are welcome for now. My day is
> almost out, at least I got all these scenarios working some tests.
Unfortunately, I still can produce that error running my repro script,
with higher concurrency:
for ((c=1;c<=50;c++)); do createdb db$c; done
for ((i=1;i<=100;i++)); do
echo "iteration $i"
for ((c=1;c<=50;c++)); do
echo "\\d sometable" | psql -d db$c >psql-1-$c.log 2>&1 &
echo "DROP TABLE IF EXISTS tbl;
CREATE TABLE tbl (i int, t text);
ALTER TABLE tbl ALTER COLUMN t SET STORAGE EXTERNAL;
INSERT INTO tbl(i, t) VALUES (1, repeat('1234567890', 250));
DELETE FROM tbl;
VACUUM (TRUNCATE, VERBOSE) tbl;
VACUUM FULL tbl;
" | psql -d db$c >psql-$c.log 2>&1 &
done
wait
grep 'missing chunk number' server.log && break;
done
...
iteration 45
iteration 46
iteration 47
2026-07-06 17:38:19.952 UTC|law|db22|6a4be80b.3ab9a8|ERROR: missing chunk number 0 for toast value 41866 in pg_toast_41849
I had failures on iterations 47, 58, 79.
(Konstantin's patch doesn't help either.)
Best regards,
Alexander
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: BUG #19519: REPACK can fail due to missing chunk for toast value
2026-06-14 07:00 BUG #19519: REPACK can fail due to missing chunk for toast value PG Bug reporting form <[email protected]>
2026-06-17 17:03 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-03 06:45 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Imran Zaheer <[email protected]>
2026-07-06 06:36 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
@ 2026-07-06 18:21 ` Matthias van de Meent <[email protected]>
2026-07-06 23:18 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2 siblings, 1 reply; 21+ messages in thread
From: Matthias van de Meent @ 2026-07-06 18:21 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Imran Zaheer <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; [email protected]; [email protected]; Konstantin Knizhnik <[email protected]>
On Mon, 6 Jul 2026 at 18:40, Michael Paquier <[email protected]> wrote:
>
> On Fri, Jul 03, 2026 at 11:45:42AM +0500, Imran Zaheer wrote:
> > Can we safely use the same fix in the index build path too? Can we use
> > GetOldestNonRemovableTransactionIdforRewrite or something similar
> > here? Normal serial index builds use SnapShotAny and concurrent index
> > builds use MVCC, but the bug only exists in the serial index build
> > path.
>
> Yeah, I think that we should be able to safely reuse your ForRewrite()
> API for the index build case (SnapshotAny part) as well to make the
> horizon computations more conservative, exclusing one own XID as we
> do in the lazy VACUUM case.
>
> > Other than that, after applying your patch, the bug was not
> > reproducible with either this repro or the other report's repro [1] in
> > the rewrite path. However, the create index bug is still there. You
> > can use the following repro as mentioned in the thread [1].
>
> The assymetry between the global xmin computation and the per-database
> horizon is what's killing us here. All these patterns are complicated
> enough that they warrant some tests, even if it is necessary to have
> multiple databases to trigger the buggy horizon computations. TAP
> tests would be an obvious choice for that. Now I have a set of tricks
> in my sleeves to make an isolation test fully deterministic:
> - dblink() that opens a transaction to a different database than the
> one of the isolation regression database.
> - Rename of a TOAST table using allow_system_table_mods, for a VACUUM.
>
> The second one is one of the dirtiest tricks I've done in the past for
> a REINDEX CONCURRENTLY test with TOAST. Quite useful. If the backend
> side change is reverted, the tests fail.
>
> The attached includes both test suites for all four cases reported
> (REPACK, VACUUM, CLUSTER, non-MVCC index builds). We'll need to
> remove one of them, just keeping both posted as I am not sure if the
> isolation tests would be entirely stable in the buildfarm..
>
> I still need to dive more into the code, but for now this bit stands
> out:
> vacuum_get_cutoffs(OldHeap, ¶ms, &cutoffs);
>
> + /*
> + * vacuum_get_cutoffs() folds our own backend's xmin into OldestXmin. For
> + * a rewrite that is too conservative: the snapshot we hold exists only to
> + * evaluate index expressions against other relations, not to read
> + * OldHeap's historical rows. If our xmin is held back by a transaction
> + * that cannot even see OldHeap (e.g. one in another database), we would
> + * preserve a recently-dead tuple whose TOAST chunks a concurrent or prior
> + * lazy vacuum was free to remove, and then fail with "missing chunk" while
> + * copying it. Recompute OldestXmin ignoring our own backend so it matches
> + * the horizon lazy vacuum uses. This can only move OldestXmin forward, so
> + * the freeze cutoffs derived above remain valid.
> + */
> + cutoffs.OldestXmin = GetOldestNonRemovableTransactionIdForRewrite(OldHeap);
>
> In copy_table_data() (for the data copy with repack, cluster, vacuum
> full), I think that this is incorrect. For one, this breaks the
> FreezeLimit which should always be older than OldestXmin, and this
> patch enforces a new recomputation of OldestXmin ignoring most of the
> internals of vacuum_get_cutoffs(). That's brittle, to say the least
> if we change the way the vacuum cutoffs are calculated in the future.
>
> Thoughts and comments from others are welcome for now. My day is
> almost out, at least I got all these scenarios working some tests.
I don't think this approach actually fixes the underlying issue: As I
understand it; the problem is this:
1. OldestNonRemovableTransactionId() is based on the xmin state across
backends. It may increase at any point in time, could even decrease in
certain circumstances[0], and can be in an inconsistent state across
backends.
2. Data is only protected against concurrent cleanup while it's
visible to a MVCC snapshot in some backend, and this is advertised
through the xmin of the backend holding the snapshot being <= the xmin
of the snapshot.
This relation-level cleanup horizon is determined with
GlobalVisTestFor(), which uses the same data source as
OldestNonRemovableTransactionId().
3. All relevant cases of REPACK and CREATE INDEX use SnapshotAny + a
cached output of OldestNonRemovableTransactionId()
The issue here is that CREATE INDEX and REPACK (CI/R) can start with
an OldestXmin of (say) 100, then another backend that held back that
OldestXmin stops holding back the xmin (allowing the output of a new
invocation of OldestNonRemovableTransactionId() to return e.g. 200),
after which the cleanup starts for tuples deleted with transaction ID
< 200, rather than the <100 expected by INDEX/REPACK.
Because both CI and R use SnapshotAny, and stored the result of
OldestNonRemovableTransactionId() from the start of their command
processing, they'll consider any still-present tuple >= 100 to be
visible for their scan.
However, every other backend has stopped considering these tuples
visible for their MVCC, scans and therefore may have started cleaning
up the tuple data. For main relation's heap pages, this cleanup isn't
(much) of an issue, because the fully dead and removed tuples would
have to be cleaned up later anyway and are not interesting for the
scan, but the related toast tuples can also be cleaned up without
first having to wait for the main tuple to be cleaned up, which is
exactly what this issue shows: TOAST expects to be used in an MVCC
context, and thus errors when it fails to find the tuples that were
reclaimed. It's a de-sync between what this backend and what other
backends expect.
So, in effect, this is an issue that's quite similar to the CIC/RIC
bug of PG14<14.4, except that
1.) in this case it's SnapshotAny instead of a
registered-but-invisible MVCC snapshot;
2.) the issue in this case only happens when toast pointers were
cleaned up and then are dereferenced, rather than HOT chains updated
and reclaimed since the scan started; and finally
3.) this issue dates back a very long time, likely since TOAST, but
defininitely since PG14 introduced the GlobalVisTestFor apis.
So I think the only way to fix this is either
1.) we update every maintenance process that makes use of SnapshotAny
and could access TOAST tables to handle errors caused by missing TOAST
data when the base relation's tuple is RECENTLY_DEAD, and treat those
as "tuple cleanup has already started, we just weren't aware of it
yet, so ignore this tuple" (as I mentioned in [1]), or
2.) we set the xmin of the CI/R backend to the current OldestXmin for
the relation(s) we're currently processing, and so block all cleanup
of the relevant (toast) data for the duration of this statement or
whenever we check for a newer xmin for the relation (whilst ignoring
our current one).
Kind regards,
Matthias van de Meent
Databricks (https://www.databricks.com)
[0]: "BUG #17257: (auto)vacuum hangs within lazy_scan_prune()"
https://www.postgresql.org/message-id/flat/CAH2-WznTo_nkbTAoBO2mCR5TzWBj8fEa4%252Bk-K3x9YUYZ0AKVUQ%2...
[1]: "Re: VACUUM FULL or CREATE INDEX fails with error: missing chunk
number 0 for toast value XXX" at -hackers:
https://www.postgresql.org/message-id/CAEze2WgRUfoR%3DtSQQm8zexVNeBbEYi9ZSPztrK63ObyfPxKpBw%40mail.g...
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: BUG #19519: REPACK can fail due to missing chunk for toast value
2026-06-14 07:00 BUG #19519: REPACK can fail due to missing chunk for toast value PG Bug reporting form <[email protected]>
2026-06-17 17:03 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-03 06:45 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Imran Zaheer <[email protected]>
2026-07-06 06:36 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 18:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
@ 2026-07-06 23:18 ` Michael Paquier <[email protected]>
2026-07-06 23:51 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Michael Paquier @ 2026-07-06 23:18 UTC (permalink / raw)
To: Matthias van de Meent <[email protected]>; +Cc: Imran Zaheer <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; [email protected]; [email protected]; Konstantin Knizhnik <[email protected]>
On Mon, Jul 06, 2026 at 08:21:41PM +0200, Matthias van de Meent wrote:
> So, in effect, this is an issue that's quite similar to the CIC/RIC
> bug of PG14<14.4, except that
>
> 1.) in this case it's SnapshotAny instead of a
> registered-but-invisible MVCC snapshot;
> 2.) the issue in this case only happens when toast pointers were
> cleaned up and then are dereferenced, rather than HOT chains updated
> and reclaimed since the scan started; and finally
> 3.) this issue dates back a very long time, likely since TOAST, but
> defininitely since PG14 introduced the GlobalVisTestFor apis.
You mean 042b584c7f7d, revert of d9d076222f5b. That rings a bell.
> So I think the only way to fix this is either
> 1.) we update every maintenance process that makes use of SnapshotAny
> and could access TOAST tables to handle errors caused by missing TOAST
> data when the base relation's tuple is RECENTLY_DEAD, and treat those
> as "tuple cleanup has already started, we just weren't aware of it
> yet, so ignore this tuple" (as I mentioned in [1]), or
That seems like an option worth exploring to me, but I doubt that
we'll be able to get something that can be backpatched due to how
invasive it is. I strongly suspect that it would require at least one
table AM change to cope with the fact that we want to let the callers
be OK with TOAST chunks missing in some contexts when grabbing a toast
slice. That's perhaps for the best if we think about potential
regressions, this is scary and old enough that we may still be OK by
living without a backpatch.
> 2.) we set the xmin of the CI/R backend to the current OldestXmin for
> the relation(s) we're currently processing, and so block all cleanup
> of the relevant (toast) data for the duration of this statement or
> whenever we check for a newer xmin for the relation (whilst ignoring
> our current one).
Err, I don't think that would be safe anyway? It's rather uncommon in
practice in schemas, but we could trigger function calls, like index
expressions, domains, whatever, that access data of other tables while
processing one table with a SnapshotAny. The CIC revert of 14.4 was
about such cases.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: BUG #19519: REPACK can fail due to missing chunk for toast value
2026-06-14 07:00 BUG #19519: REPACK can fail due to missing chunk for toast value PG Bug reporting form <[email protected]>
2026-06-17 17:03 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-03 06:45 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Imran Zaheer <[email protected]>
2026-07-06 06:36 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 18:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
2026-07-06 23:18 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
@ 2026-07-06 23:51 ` Matthias van de Meent <[email protected]>
2026-07-07 01:42 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Matthias van de Meent @ 2026-07-06 23:51 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Imran Zaheer <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Alexander Lakhin <[email protected]>; PostgreSQL mailing lists <[email protected]>; Konstantin Knizhnik <[email protected]>
On Tue, 7 Jul 2026, 01:19 Michael Paquier, <[email protected]> wrote:
> On Mon, Jul 06, 2026 at 08:21:41PM +0200, Matthias van de Meent wrote:
> > So, in effect, this is an issue that's quite similar to the CIC/RIC
> > bug of PG14<14.4, except that
> >
> > 1.) in this case it's SnapshotAny instead of a
> > registered-but-invisible MVCC snapshot;
> > 2.) the issue in this case only happens when toast pointers were
> > cleaned up and then are dereferenced, rather than HOT chains updated
> > and reclaimed since the scan started; and finally
> > 3.) this issue dates back a very long time, likely since TOAST, but
> > defininitely since PG14 introduced the GlobalVisTestFor apis.
>
> You mean 042b584c7f7d, revert of d9d076222f5b. That rings a bell.
>
Indeed
> > So I think the only way to fix this is either
> > 1.) we update every maintenance process that makes use of SnapshotAny
> > and could access TOAST tables to handle errors caused by missing TOAST
> > data when the base relation's tuple is RECENTLY_DEAD, and treat those
> > as "tuple cleanup has already started, we just weren't aware of it
> > yet, so ignore this tuple" (as I mentioned in [1]), or
>
> That seems like an option worth exploring to me, but I doubt that
> we'll be able to get something that can be backpatched due to how
> invasive it is. I strongly suspect that it would require at least one
> table AM change to cope with the fact that we want to let the callers
> be OK with TOAST chunks missing in some contexts when grabbing a toast
> slice. That's perhaps for the best if we think about potential
> regressions, this is scary and old enough that we may still be OK by
> living without a backpatch.
>
I suspect that for indexing we're lucky, in that we have full control
internally over which tuple data get fed into IndexBuildCallback. We can
make sure only fully detoasted attributes get passed on, and can be
implemented with some (a lot) of TRY/CATCH work in or around
FormIndexDatum. That said, I don't think that's very nice, and I'm also a
bit concerned about leaking things (memory, buffers, ...) in such an
approach.
For REPACK/VACUUM FULL/CLUSTER, I think we can do something similar, given
that this code too has a central loop processing the tuple data, though I
suspect it'll need a bit more work than the reindex one.
> 2.) we set the xmin of the CI/R backend to the current OldestXmin for
> > the relation(s) we're currently processing, and so block all cleanup
> > of the relevant (toast) data for the duration of this statement or
> > whenever we check for a newer xmin for the relation (whilst ignoring
> > our current one).
>
> Err, I don't think that would be safe anyway? It's rather uncommon in
> practice in schemas, but we could trigger function calls, like index
> expressions, domains, whatever, that access data of other tables while
> processing one table with a SnapshotAny. The CIC revert of 14.4 was
> about such cases.
>
True, but in indexes that is forbidden: mislabeling of non-INMUTABLE
functions causes broken indexes every day, and we can't be expected to make
it work there. Similarly, for REPACK it shouldn't evaluate any
non-indexable expressions, right?
Either way, if we can register a snapshot in the REPACK/INDEX backend whose
xmin is (*correctly*) registered with the OldestNonRemovableTransactionId()
of the relation, then concurrent cleanup won't happen for toast, and we'd
also avoid dereferencing those dangling toast pointers. We won't need to
actually use that snapshot for any scans; it'll avoid the issue as long as
our backend advertises an xmin <= the OldestXmin used in maintenance to
hold back pruning/vacuum on the relation.
(we can adjust that xmin every once in a while with a newer
OldestNonRemovableTransactionId (excl. our own backend) as long as we also
adjust the OldestXmin cutoff for the SnapshotAny visibility tests, but I
don't suggest to backpatch that even newer system, it'd add significant
complications to an already complicated system)
- Matthias
ps. sorry for the formatting, I'm writing this on my phone.
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: BUG #19519: REPACK can fail due to missing chunk for toast value
2026-06-14 07:00 BUG #19519: REPACK can fail due to missing chunk for toast value PG Bug reporting form <[email protected]>
2026-06-17 17:03 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-03 06:45 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Imran Zaheer <[email protected]>
2026-07-06 06:36 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 18:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
2026-07-06 23:18 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 23:51 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
@ 2026-07-07 01:42 ` Michael Paquier <[email protected]>
2026-07-07 05:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Michael Paquier @ 2026-07-07 01:42 UTC (permalink / raw)
To: Matthias van de Meent <[email protected]>; +Cc: Imran Zaheer <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Alexander Lakhin <[email protected]>; PostgreSQL mailing lists <[email protected]>; Konstantin Knizhnik <[email protected]>
On Tue, Jul 07, 2026 at 01:51:32AM +0200, Matthias van de Meent wrote:
> On Tue, 7 Jul 2026, 01:19 Michael Paquier, <[email protected]> wrote:
>
>> On Mon, Jul 06, 2026 at 08:21:41PM +0200, Matthias van de Meent wrote:
>>> So, in effect, this is an issue that's quite similar to the CIC/RIC
>>> bug of PG14<14.4, except that
>>>
>>> 1.) in this case it's SnapshotAny instead of a
>>> registered-but-invisible MVCC snapshot;
>>> 2.) the issue in this case only happens when toast pointers were
>>> cleaned up and then are dereferenced, rather than HOT chains updated
>>> and reclaimed since the scan started; and finally
>>> 3.) this issue dates back a very long time, likely since TOAST, but
>>> defininitely since PG14 introduced the GlobalVisTestFor apis.
>>
>> You mean 042b584c7f7d, revert of d9d076222f5b. That rings a bell.
>>
>
> Indeed
>
>
>>> So I think the only way to fix this is either
>>> 1.) we update every maintenance process that makes use of SnapshotAny
>>> and could access TOAST tables to handle errors caused by missing TOAST
>>> data when the base relation's tuple is RECENTLY_DEAD, and treat those
>>> as "tuple cleanup has already started, we just weren't aware of it
>>> yet, so ignore this tuple" (as I mentioned in [1]), or
>>
>> That seems like an option worth exploring to me, but I doubt that
>> we'll be able to get something that can be backpatched due to how
>> invasive it is. I strongly suspect that it would require at least one
>> table AM change to cope with the fact that we want to let the callers
>> be OK with TOAST chunks missing in some contexts when grabbing a toast
>> slice. That's perhaps for the best if we think about potential
>> regressions, this is scary and old enough that we may still be OK by
>> living without a backpatch.
>>
>
> I suspect that for indexing we're lucky, in that we have full control
> internally over which tuple data get fed into IndexBuildCallback. We can
> make sure only fully detoasted attributes get passed on, and can be
> implemented with some (a lot) of TRY/CATCH work in or around
> FormIndexDatum. That said, I don't think that's very nice, and I'm also a
> bit concerned about leaking things (memory, buffers, ...) in such an
> approach.
I've quickly thought about some try/catch blocks while detoasting, but
I cannot really get how this would be entirely bullet-proof. It seems
to me that we'll live better if we bite the bullet and go for the
invasive change of passing a state across the stack to let detoasting
know that we are dealing with a rewrite.
> For REPACK/VACUUM FULL/CLUSTER, I think we can do something similar, given
> that this code too has a central loop processing the tuple data, though I
> suspect it'll need a bit more work than the reindex one.
While looking at the whole stack, the handling of dead tuples seems to
go through only index_build_range_scan() and
relation_copy_for_cluster() for heapam. A re-check for TOAST values
would be really bad if we have a lot of dead tuples.
>> 2.) we set the xmin of the CI/R backend to the current OldestXmin for
>>> the relation(s) we're currently processing, and so block all cleanup
>>> of the relevant (toast) data for the duration of this statement or
>>> whenever we check for a newer xmin for the relation (whilst ignoring
>>> our current one).
>>
>> Err, I don't think that would be safe anyway? It's rather uncommon in
>> practice in schemas, but we could trigger function calls, like index
>> expressions, domains, whatever, that access data of other tables while
>> processing one table with a SnapshotAny. The CIC revert of 14.4 was
>> about such cases.
>
> True, but in indexes that is forbidden: mislabeling of non-INMUTABLE
> functions causes broken indexes every day, and we can't be expected to make
> it work there. Similarly, for REPACK it shouldn't evaluate any
> non-indexable expressions, right?
Still I am pretty sure that the window would exist, users like fancy
definitions. It seems at least worth looking if we could make
something more stable across the board. Just playing with it now, and
it does not seem that bad in practice, actually..
> Either way, if we can register a snapshot in the REPACK/INDEX backend whose
> xmin is (*correctly*) registered with the OldestNonRemovableTransactionId()
> of the relation, then concurrent cleanup won't happen for toast, and we'd
> also avoid dereferencing those dangling toast pointers. We won't need to
> actually use that snapshot for any scans; it'll avoid the issue as long as
> our backend advertises an xmin <= the OldestXmin used in maintenance to
> hold back pruning/vacuum on the relation.
>
> (we can adjust that xmin every once in a while with a newer
> OldestNonRemovableTransactionId (excl. our own backend) as long as we also
> adjust the OldestXmin cutoff for the SnapshotAny visibility tests, but I
> don't suggest to backpatch that even newer system, it'd add significant
> complications to an already complicated system)
I doubt that anything would be backpatchable, based on how the
discussion is going. I'm OK to be proven wrong.
> ps. sorry for the formatting, I'm writing this on my phone.
No pb. Thanks.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: BUG #19519: REPACK can fail due to missing chunk for toast value
2026-06-14 07:00 BUG #19519: REPACK can fail due to missing chunk for toast value PG Bug reporting form <[email protected]>
2026-06-17 17:03 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-03 06:45 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Imran Zaheer <[email protected]>
2026-07-06 06:36 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 18:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
2026-07-06 23:18 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 23:51 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
2026-07-07 01:42 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
@ 2026-07-07 05:21 ` Michael Paquier <[email protected]>
2026-07-07 17:10 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Michael Paquier @ 2026-07-07 05:21 UTC (permalink / raw)
To: Matthias van de Meent <[email protected]>; +Cc: Imran Zaheer <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Alexander Lakhin <[email protected]>; PostgreSQL mailing lists <[email protected]>; Konstantin Knizhnik <[email protected]>
On Tue, Jul 07, 2026 at 10:42:33AM +0900, Michael Paquier wrote:
> I've quickly thought about some try/catch blocks while detoasting, but
> I cannot really get how this would be entirely bullet-proof. It seems
> to me that we'll live better if we bite the bullet and go for the
> invasive change of passing a state across the stack to let detoasting
> know that we are dealing with a rewrite.
I have been playing with this idea, with two approaches. For the
REPACK and VACUUM cases, things looked pretty straight for both
approaches, but the index build got hairy in the first case:
1) Attempt to shortcut missing TOAST chunks when doing index builds.
For hash and btree, things get straight with index_form_tuple(). But
I've quickly faced a wall with GIN (extractValue support function) and
GiST (compress function). So at the end I gave up on that, as it
would require more facilities than this looks worth for.
2) A more aggressive pre-detoast when building a range of values for
tuples that we are seeing as already dead, which should be able to
work across all index AMs transparently when we are dealing with dead
tuples (that should be kept in a per-tuple context). That's what I
guess you've mentioned upthread.
Option 2 leads to a much nicer result overall, with the attached
passing my regression tests and Alexander's case as well (highly
concurrent thing sent yesterday). The rest of the patch is kind of
boring, where I have been trying to get a missing_ok state across the
stack to let the toast slice fetch bypass the case of missing chunks
when we are OK with it due to the rewrites. That still feels crude,
but the simplicity is appealing here, as much as the rather low
invasiveness.
Thoughts or comments?
--
Michael
From 2e9cc62deda3de859ee389772471734a2f4f9e5d Mon Sep 17 00:00:00 2001
From: Michael Paquier <[email protected]>
Date: Tue, 7 Jul 2026 14:20:30 +0900
Subject: [PATCH v3] Fix "missing chunk" errors during heap rewrites and index
builds
Blah, blah.
This problem has been reported recently by Alexander for REPACK, but it
is a much older issue (tested down to 9.5 for curiosity's sake, but I
suspect that it is much older than that).
XXX: This patch includes both TAP and isolation tests. We should remove
one pattern at the end.
Reported-by: Alexander Lakhin <[email protected]>
Discussion: https://postgr.es/m/[email protected]
Discussion: https://postgr.es/m/[email protected]
---
src/include/access/detoast.h | 9 +
src/include/access/heaptoast.h | 13 +-
src/include/access/rewriteheap.h | 4 +-
src/include/access/tableam.h | 24 +-
src/include/access/toast_helper.h | 2 +-
src/backend/access/common/detoast.c | 41 ++-
src/backend/access/heap/heapam.c | 4 +-
src/backend/access/heap/heapam_handler.c | 94 ++++++-
src/backend/access/heap/heaptoast.c | 22 +-
src/backend/access/heap/rewriteheap.c | 31 ++-
src/backend/access/table/toast_helper.c | 17 +-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/055_rewrite_stale_xmin.pl | 124 +++++++++
contrib/dblink/.gitignore | 2 +
contrib/dblink/Makefile | 2 +
.../dblink/expected/rewrite_stale_xmin.out | 237 ++++++++++++++++++
contrib/dblink/meson.build | 5 +
contrib/dblink/specs/rewrite_stale_xmin.spec | 137 ++++++++++
18 files changed, 718 insertions(+), 51 deletions(-)
create mode 100644 src/test/recovery/t/055_rewrite_stale_xmin.pl
create mode 100644 contrib/dblink/expected/rewrite_stale_xmin.out
create mode 100644 contrib/dblink/specs/rewrite_stale_xmin.spec
diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h
index fbd98181a3a1..c5eeed04b011 100644
--- a/src/include/access/detoast.h
+++ b/src/include/access/detoast.h
@@ -43,6 +43,15 @@ do { \
*/
extern varlena *detoast_external_attr(varlena *attr);
+/* ----------
+ * detoast_external_attr_extended() -
+ *
+ * Like detoast_external_attr, but returns NULL if toast chunks
+ * are missing (instead of raising an error).
+ * ----------
+ */
+extern varlena *detoast_external_attr_extended(varlena *attr);
+
/* ----------
* detoast_attr() -
*
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index 631cb1836b96..1be85db6ddd2 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -92,10 +92,14 @@
* heap_toast_insert_or_update -
*
* Called by heap_insert() and heap_update().
+ *
+ * If missing_ok is true, returns NULL when failing to fetch external
+ * TOAST data, instead of throwing an error.
* ----------
*/
extern HeapTuple heap_toast_insert_or_update(Relation rel, HeapTuple newtup,
- HeapTuple oldtup, uint32 options);
+ HeapTuple oldtup, uint32 options,
+ bool missing_ok);
/* ----------
* heap_toast_delete -
@@ -140,10 +144,13 @@ extern HeapTuple toast_build_flattened_tuple(TupleDesc tupleDesc,
* heap_fetch_toast_slice
*
* Fetch a slice from a toast value stored in a heap table.
+ * If missing_ok is true, returns false when chunks are missing
+ * instead of raising an error. Returns true on success.
* ----------
*/
-extern void heap_fetch_toast_slice(Relation toastrel, Oid valueid,
+extern bool heap_fetch_toast_slice(Relation toastrel, Oid valueid,
int32 attrsize, int32 sliceoffset,
- int32 slicelength, varlena *result);
+ int32 slicelength, varlena *result,
+ bool missing_ok);
#endif /* HEAPTOAST_H */
diff --git a/src/include/access/rewriteheap.h b/src/include/access/rewriteheap.h
index 6ccf7b45c04e..df4fa4fde710 100644
--- a/src/include/access/rewriteheap.h
+++ b/src/include/access/rewriteheap.h
@@ -25,8 +25,8 @@ extern RewriteState begin_heap_rewrite(Relation old_heap, Relation new_heap,
TransactionId oldest_xmin, TransactionId freeze_xid,
MultiXactId cutoff_multi);
extern void end_heap_rewrite(RewriteState state);
-extern void rewrite_heap_tuple(RewriteState state, HeapTuple old_tuple,
- HeapTuple new_tuple);
+extern bool rewrite_heap_tuple(RewriteState state, HeapTuple old_tuple,
+ HeapTuple new_tuple, bool missing_ok);
extern bool rewrite_heap_dead_tuple(RewriteState state, HeapTuple old_tuple);
/*
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index f2c36696bcad..a90b15dd82b3 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -782,12 +782,16 @@ typedef struct TableAmRoutine
* This callback is invoked when detoasting a value stored in a toast
* table implemented by this AM. See table_relation_fetch_toast_slice()
* for more details.
+ *
+ * If missing_ok is true, returns false when chunks are missing instead of
+ * raising an error. Returns true on success.
*/
- void (*relation_fetch_toast_slice) (Relation toastrel, Oid valueid,
+ bool (*relation_fetch_toast_slice) (Relation toastrel, Oid valueid,
int32 attrsize,
int32 sliceoffset,
int32 slicelength,
- varlena *result);
+ varlena *result,
+ bool missing_ok);
/* ------------------------------------------------------------------------
@@ -1981,16 +1985,20 @@ table_relation_toast_am(Relation rel)
*
* result is caller-allocated space into which the fetched bytes should be
* stored.
+ *
+ * missing_ok: if true, return false when toast chunks are missing instead
+ * of raising an error. Returns true on success.
*/
-static inline void
+static inline bool
table_relation_fetch_toast_slice(Relation toastrel, Oid valueid,
int32 attrsize, int32 sliceoffset,
- int32 slicelength, varlena *result)
+ int32 slicelength, varlena *result,
+ bool missing_ok)
{
- toastrel->rd_tableam->relation_fetch_toast_slice(toastrel, valueid,
- attrsize,
- sliceoffset, slicelength,
- result);
+ return toastrel->rd_tableam->relation_fetch_toast_slice(toastrel, valueid,
+ attrsize,
+ sliceoffset, slicelength,
+ result, missing_ok);
}
diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h
index 2ec92397f267..e548ba5596b4 100644
--- a/src/include/access/toast_helper.h
+++ b/src/include/access/toast_helper.h
@@ -101,7 +101,7 @@ typedef struct
#define TOASTCOL_IGNORE 0x0010
#define TOASTCOL_INCOMPRESSIBLE 0x0020
-extern void toast_tuple_init(ToastTupleContext *ttc);
+extern bool toast_tuple_init(ToastTupleContext *ttc, bool missing_ok);
extern int toast_tuple_find_biggest_attribute(ToastTupleContext *ttc,
bool for_compression,
bool check_main);
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index a6c1f3a734b2..c16b61a1cb1c 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -22,7 +22,7 @@
#include "utils/expandeddatum.h"
#include "utils/rel.h"
-static varlena *toast_fetch_datum(varlena *attr);
+static varlena *toast_fetch_datum(varlena *attr, bool missing_ok);
static varlena *toast_fetch_datum_slice(varlena *attr,
int32 sliceoffset,
int32 slicelength);
@@ -51,7 +51,7 @@ detoast_external_attr(varlena *attr)
/*
* This is an external stored plain value
*/
- result = toast_fetch_datum(attr);
+ result = toast_fetch_datum(attr, false);
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
@@ -102,6 +102,23 @@ detoast_external_attr(varlena *attr)
}
+/* ----------
+ * detoast_external_attr_extended -
+ *
+ * Like detoast_external_attr, but returns NULL if toast chunks are
+ * missing for external pointers.
+ * ----------
+ */
+varlena *
+detoast_external_attr_extended(varlena *attr)
+{
+ if (VARATT_IS_EXTERNAL_ONDISK(attr))
+ return toast_fetch_datum(attr, true);
+ else
+ return detoast_external_attr(attr);
+}
+
+
/* ----------
* detoast_attr -
*
@@ -120,7 +137,7 @@ detoast_attr(varlena *attr)
/*
* This is an externally stored datum --- fetch it back from there
*/
- attr = toast_fetch_datum(attr);
+ attr = toast_fetch_datum(attr, false);
/* If it's compressed, decompress it */
if (VARATT_IS_COMPRESSED(attr))
{
@@ -262,7 +279,7 @@ detoast_attr_slice(varlena *attr,
preslice = toast_fetch_datum_slice(attr, 0, max_size);
}
else
- preslice = toast_fetch_datum(attr);
+ preslice = toast_fetch_datum(attr, false);
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
@@ -337,10 +354,12 @@ detoast_attr_slice(varlena *attr,
*
* Reconstruct an in memory Datum from the chunks saved
* in the toast relation
+ *
+ * If missing_ok is true, returns NULL when chunks are missing.
* ----------
*/
static varlena *
-toast_fetch_datum(varlena *attr)
+toast_fetch_datum(varlena *attr, bool missing_ok)
{
Relation toastrel;
varlena *result;
@@ -372,8 +391,14 @@ toast_fetch_datum(varlena *attr)
toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock);
/* Fetch all chunks */
- table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
- attrsize, 0, attrsize, result);
+ if (!table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
+ attrsize, 0, attrsize, result,
+ missing_ok))
+ {
+ pfree(result);
+ table_close(toastrel, AccessShareLock);
+ return NULL;
+ }
/* Close toast table */
table_close(toastrel, AccessShareLock);
@@ -454,7 +479,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
/* Fetch all chunks */
table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
attrsize, sliceoffset, slicelength,
- result);
+ result, false);
/* Close toast table */
table_close(toastrel, AccessShareLock);
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index abfd8e8970a6..1d4018931200 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -2236,7 +2236,7 @@ heap_prepare_insert(Relation relation, HeapTuple tup, TransactionId xid,
return tup;
}
else if (HeapTupleHasExternal(tup) || tup->t_len > TOAST_TUPLE_THRESHOLD)
- return heap_toast_insert_or_update(relation, tup, NULL, options);
+ return heap_toast_insert_or_update(relation, tup, NULL, options, false);
else
return tup;
}
@@ -3871,7 +3871,7 @@ l2:
if (need_toast)
{
/* Note we always use WAL and FSM during updates */
- heaptup = heap_toast_insert_or_update(relation, newtup, &oldtup, 0);
+ heaptup = heap_toast_insert_or_update(relation, newtup, &oldtup, 0, false);
newtupsize = MAXALIGN(heaptup->t_len);
}
else
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 2268cc277bce..2cd9d0f4aeb9 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -19,6 +19,7 @@
*/
#include "postgres.h"
+#include "access/detoast.h"
#include "access/genam.h"
#include "access/heapam.h"
#include "access/heaptoast.h"
@@ -48,9 +49,10 @@
#include "utils/rel.h"
#include "utils/tuplesort.h"
-static void reform_and_rewrite_tuple(HeapTuple tuple,
+static bool reform_and_rewrite_tuple(HeapTuple tuple,
Relation OldHeap, Relation NewHeap,
- Datum *values, bool *isnull, RewriteState rwstate);
+ Datum *values, bool *isnull, RewriteState rwstate,
+ bool missing_ok);
static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap,
Relation NewHeap, Datum *values, bool *isnull,
BulkInsertState bistate);
@@ -717,6 +719,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
HeapTuple tuple;
Buffer buf;
bool isdead;
+ bool recently_dead = false;
CHECK_FOR_INTERRUPTS();
@@ -802,6 +805,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
break;
case HEAPTUPLE_RECENTLY_DEAD:
*tups_recently_dead += 1;
+ recently_dead = true;
pg_fallthrough;
case HEAPTUPLE_LIVE:
/* Live or recently dead, must copy it */
@@ -835,6 +839,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
RelationGetRelationName(OldHeap));
/* treat as recently dead */
*tups_recently_dead += 1;
+ recently_dead = true;
isdead = false;
break;
default:
@@ -881,8 +886,22 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
int64 ct_val[2];
if (!concurrent)
- reform_and_rewrite_tuple(tuple, OldHeap, NewHeap,
- values, isnull, rwstate);
+ {
+ if (!reform_and_rewrite_tuple(tuple, OldHeap, NewHeap,
+ values, isnull, rwstate,
+ recently_dead))
+ {
+ /*
+ * Missing TOAST chunks for a recently-dead tuple. Treat
+ * it as dead.
+ */
+ *tups_vacuumed += 1;
+ *num_tuples -= 1;
+ if (recently_dead)
+ *tups_recently_dead -= 1;
+ continue;
+ }
+ }
else
heap_insert_for_repack(tuple, OldHeap, NewHeap,
values, isnull, bistate);
@@ -934,10 +953,17 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
n_tuples += 1;
if (!concurrent)
- reform_and_rewrite_tuple(tuple,
- OldHeap, NewHeap,
- values, isnull,
- rwstate);
+ {
+ if (!reform_and_rewrite_tuple(tuple,
+ OldHeap, NewHeap,
+ values, isnull,
+ rwstate, true))
+ {
+ /* TOAST gone for a recently dead tuple */
+ n_tuples -= 1;
+ continue;
+ }
+ }
else
heap_insert_for_repack(tuple, OldHeap, NewHeap,
values, isnull, bistate);
@@ -1613,6 +1639,43 @@ heapam_index_build_range_scan(Relation heapRelation,
continue;
}
+ /*
+ * For RECENTLY_DEAD tuples, pre-detoast external TOAST values. We do
+ * this to detect for missing chunks at later stages, feeding inline
+ * data to the downstream access method code.
+ */
+ if (!tupleIsAlive)
+ {
+ TupleDesc tdesc = RelationGetDescr(heapRelation);
+ bool skip = false;
+
+ slot_getallattrs(slot);
+ for (int i = 0; i < tdesc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(tdesc, i);
+
+ if (att->attlen != -1 || att->attisdropped)
+ continue;
+ if (slot->tts_isnull[i])
+ continue;
+ if (VARATT_IS_EXTERNAL_ONDISK(DatumGetPointer(slot->tts_values[i])))
+ {
+ varlena *detoasted;
+
+ detoasted = detoast_external_attr_extended(
+ (varlena *) DatumGetPointer(slot->tts_values[i]));
+ if (detoasted == NULL)
+ {
+ skip = true;
+ break;
+ }
+ slot->tts_values[i] = PointerGetDatum(detoasted);
+ }
+ }
+ if (skip)
+ continue;
+ }
+
/*
* For the current heap tuple, extract all the attributes we use in
* this index, and note which are null. This also performs evaluation
@@ -2347,20 +2410,29 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate,
* SET WITHOUT OIDS.
*
* So, we must reconstruct the tuple from component Datums.
+ *
+ * Returns true on success, and false on failure if missing_ok is
+ * true.
*/
-static void
+static bool
reform_and_rewrite_tuple(HeapTuple tuple,
Relation OldHeap, Relation NewHeap,
- Datum *values, bool *isnull, RewriteState rwstate)
+ Datum *values, bool *isnull, RewriteState rwstate,
+ bool missing_ok)
{
HeapTuple newtuple;
newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull);
/* The heap rewrite module does the rest */
- rewrite_heap_tuple(rwstate, tuple, newtuple);
+ if (!rewrite_heap_tuple(rwstate, tuple, newtuple, missing_ok))
+ {
+ heap_freetuple(newtuple);
+ return false;
+ }
heap_freetuple(newtuple);
+ return true;
}
/*
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 03f885a25b07..38896a0c7492 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -94,7 +94,7 @@ heap_toast_delete(Relation rel, HeapTuple oldtup, bool is_speculative)
*/
HeapTuple
heap_toast_insert_or_update(Relation rel, HeapTuple newtup, HeapTuple oldtup,
- uint32 options)
+ uint32 options, bool missing_ok)
{
HeapTuple result_tuple;
TupleDesc tupleDesc;
@@ -154,7 +154,8 @@ heap_toast_insert_or_update(Relation rel, HeapTuple newtup, HeapTuple oldtup,
ttc.ttc_oldisnull = toast_oldisnull;
}
ttc.ttc_attr = toast_attr;
- toast_tuple_init(&ttc);
+ if (!toast_tuple_init(&ttc, missing_ok))
+ return NULL;
/* ----------
* Compress and/or save external until data fits into target length
@@ -621,11 +622,14 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
* sliceoffset is the byte offset within the TOAST value from which to fetch.
* slicelength is the number of bytes to be fetched from the TOAST value.
* result is the varlena into which the results should be written.
+ * missing_ok if true, return false on missing chunks instead of error.
+ *
+ * Returns true on success, false if missing_ok and chunks are missing.
*/
-void
+bool
heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
int32 sliceoffset, int32 slicelength,
- varlena *result)
+ varlena *result, bool missing_ok)
{
Relation *toastidxs;
ScanKeyData toastkey[3];
@@ -779,13 +783,23 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
* Final checks that we successfully fetched the datum
*/
if (expectedchunk != (endchunk + 1))
+ {
+ if (missing_ok)
+ {
+ systable_endscan_ordered(toastscan);
+ toast_close_indexes(toastidxs, num_indexes, AccessShareLock);
+ return false;
+ }
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
errmsg_internal("missing chunk number %d for toast value %u in %s",
expectedchunk, valueid,
RelationGetRelationName(toastrel))));
+ }
/* End scan and close indexes. */
systable_endscan_ordered(toastscan);
toast_close_indexes(toastidxs, num_indexes, AccessShareLock);
+
+ return true;
}
diff --git a/src/backend/access/heap/rewriteheap.c b/src/backend/access/heap/rewriteheap.c
index 5a5398a76ae7..fbb6e6ca1f94 100644
--- a/src/backend/access/heap/rewriteheap.c
+++ b/src/backend/access/heap/rewriteheap.c
@@ -211,7 +211,8 @@ typedef struct RewriteMappingDataEntry
/* prototypes for internal functions */
-static void raw_heap_insert(RewriteState state, HeapTuple tup);
+static bool raw_heap_insert(RewriteState state, HeapTuple tup,
+ bool missing_ok);
/* internal logical remapping prototypes */
static void logical_begin_heap_rewrite(RewriteState state);
@@ -309,7 +310,7 @@ end_heap_rewrite(RewriteState state)
while ((unresolved = hash_seq_search(&seq_status)) != NULL)
{
ItemPointerSetInvalid(&unresolved->tuple->t_data->t_ctid);
- raw_heap_insert(state, unresolved->tuple);
+ raw_heap_insert(state, unresolved->tuple, false);
}
/* Write the last page, if any */
@@ -338,9 +339,10 @@ end_heap_rewrite(RewriteState state)
* old_tuple original tuple in the old heap
* new_tuple new, rewritten tuple to be inserted to new heap
*/
-void
+bool
rewrite_heap_tuple(RewriteState state,
- HeapTuple old_tuple, HeapTuple new_tuple)
+ HeapTuple old_tuple, HeapTuple new_tuple,
+ bool missing_ok)
{
MemoryContext old_cxt;
ItemPointerData old_tid;
@@ -437,7 +439,7 @@ rewrite_heap_tuple(RewriteState state,
* tuple will be written.
*/
MemoryContextSwitchTo(old_cxt);
- return;
+ return true;
}
}
@@ -455,7 +457,13 @@ rewrite_heap_tuple(RewriteState state,
ItemPointerData new_tid;
/* Insert the tuple and find out where it's put in new_heap */
- raw_heap_insert(state, new_tuple);
+ if (!raw_heap_insert(state, new_tuple, missing_ok))
+ {
+ if (free_new)
+ heap_freetuple(new_tuple);
+ MemoryContextSwitchTo(old_cxt);
+ return false;
+ }
new_tid = new_tuple->t_self;
logical_rewrite_heap_tuple(state, old_tid, new_tuple);
@@ -532,6 +540,7 @@ rewrite_heap_tuple(RewriteState state,
}
MemoryContextSwitchTo(old_cxt);
+ return true;
}
/*
@@ -593,8 +602,8 @@ rewrite_heap_dead_tuple(RewriteState state, HeapTuple old_tuple)
* tuple is invalid on entry, it's replaced with the new TID as well (in
* the inserted data only, not in the caller's copy).
*/
-static void
-raw_heap_insert(RewriteState state, HeapTuple tup)
+static bool
+raw_heap_insert(RewriteState state, HeapTuple tup, bool missing_ok)
{
Page page;
Size pageFreeSpace,
@@ -628,7 +637,9 @@ raw_heap_insert(RewriteState state, HeapTuple tup)
options |= HEAP_INSERT_NO_LOGICAL;
heaptup = heap_toast_insert_or_update(state->rs_new_rel, tup, NULL,
- options);
+ options, missing_ok);
+ if (heaptup == NULL)
+ return false;
}
else
heaptup = tup;
@@ -702,6 +713,8 @@ raw_heap_insert(RewriteState state, HeapTuple tup)
/* If heaptup is a private copy, release it. */
if (heaptup != tup)
heap_freetuple(heaptup);
+
+ return true;
}
/* ------------------------------------------------------------------------
diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c
index 2f2022d99510..7e98c01a5eee 100644
--- a/src/backend/access/table/toast_helper.c
+++ b/src/backend/access/table/toast_helper.c
@@ -36,9 +36,12 @@
* toast_flags is just a single uint8, but toast_attr is a caller-provided
* array with a length equal to tupleDesc->natts. The caller need not
* perform any initialization of the array before calling this function.
+ *
+ * If missing_ok is true and an external TOAST value cannot be fetched,
+ * returns false. Returns true on success.
*/
-void
-toast_tuple_init(ToastTupleContext *ttc)
+bool
+toast_tuple_init(ToastTupleContext *ttc, bool missing_ok)
{
TupleDesc tupleDesc = ttc->ttc_rel->rd_att;
int numAttrs = tupleDesc->natts;
@@ -137,7 +140,13 @@ toast_tuple_init(ToastTupleContext *ttc)
if (VARATT_IS_EXTERNAL(new_value))
{
ttc->ttc_attr[i].tai_oldexternal = new_value;
- if (att->attstorage == TYPSTORAGE_PLAIN)
+ if (missing_ok && VARATT_IS_EXTERNAL_ONDISK(new_value))
+ {
+ new_value = detoast_external_attr_extended(new_value);
+ if (new_value == NULL)
+ return false;
+ }
+ else if (att->attstorage == TYPSTORAGE_PLAIN)
new_value = detoast_attr(new_value);
else
new_value = detoast_external_attr(new_value);
@@ -159,6 +168,8 @@ toast_tuple_init(ToastTupleContext *ttc)
ttc->ttc_attr[i].tai_colflags |= TOASTCOL_IGNORE;
}
}
+
+ return true;
}
/*
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index ad0d85f41897..7e583ee39a10 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -63,6 +63,7 @@ tests += {
't/052_checkpoint_segment_missing.pl',
't/053_standby_login_event_trigger.pl',
't/054_unlogged_sequence_promotion.pl',
+ 't/055_rewrite_stale_xmin.pl',
],
},
}
diff --git a/src/test/recovery/t/055_rewrite_stale_xmin.pl b/src/test/recovery/t/055_rewrite_stale_xmin.pl
new file mode 100644
index 000000000000..dad95c0cf967
--- /dev/null
+++ b/src/test/recovery/t/055_rewrite_stale_xmin.pl
@@ -0,0 +1,124 @@
+# Copyright (c) 2025-2026, PostgreSQL Global Development Group
+#
+# Test that heap rewrites (CLUSTER, VACUUM FULL, REPACK) and index builds
+# work in the case when the backend's own xmin is held back by a transaction
+# in another database.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+
+my $node = PostgreSQL::Test::Cluster->new('primary');
+$node->init;
+# No Autovacuum, no random interferences.
+$node->append_conf(
+ 'postgresql.conf', qq[
+autovacuum = off
+]);
+$node->start;
+
+# Create two databases.
+$node->safe_psql('postgres', 'CREATE DATABASE db_holder');
+$node->safe_psql('postgres', 'CREATE DATABASE db_target');
+
+# Set up a table with TOAST data. Returns the toast table name.
+sub setup_table
+{
+ my ($tblname, $extra_ddl) = @_;
+ $extra_ddl //= '';
+
+ $node->safe_psql(
+ 'db_target', qq{
+ DROP TABLE IF EXISTS $tblname;
+ CREATE TABLE $tblname (id int PRIMARY KEY, data text)
+ WITH (autovacuum_enabled = off);
+ ALTER TABLE $tblname ALTER COLUMN data SET STORAGE EXTERNAL;
+ INSERT INTO $tblname VALUES (1, repeat('x', 2500));
+ INSERT INTO $tblname VALUES (2, repeat('z', 2500));
+ $extra_ddl
+ });
+
+ my $toast = $node->safe_psql(
+ 'db_target', qq{
+ SELECT 'pg_toast.' || relname FROM pg_class
+ WHERE oid = (SELECT reltoastrelid FROM pg_class
+ WHERE relname = '$tblname');
+ });
+ chomp $toast;
+ return $toast;
+}
+
+# Create precondition state. Requires that $holder is already running with
+# a write XID.
+sub create_missing_toast_state
+{
+ my ($tblname, $toast_table) = @_;
+
+ # Hold xmin in same DB to prevent main tuple removal.
+ my $xmin_holder = $node->background_psql('db_target', on_error_stop => 1);
+ $xmin_holder->query_safe("BEGIN ISOLATION LEVEL REPEATABLE READ");
+ $xmin_holder->query_safe("SELECT 1");
+
+ # Delete the row with TOAST data.
+ $node->safe_psql('db_target', "DELETE FROM $tblname WHERE id = 1");
+
+ # VACUUM main table only. The xmin_holder keeps the dead tuple.
+ $node->safe_psql('db_target', "VACUUM (PROCESS_TOAST false) $tblname");
+
+ # Release same-DB xmin.
+ $xmin_holder->query_safe("COMMIT");
+ $xmin_holder->quit;
+
+ # Vacuum toast table: toast chunks now DEAD and removed.
+ $node->safe_psql('db_target', "VACUUM $toast_table");
+}
+
+# Open a transaction holding an XID before any deletes. This XID will hold
+# back GetSnapshotData() globally, on a different database than the one
+# running the various rewrite operations.
+my $holder = $node->background_psql('db_holder', on_error_stop => 1);
+$holder->query_safe("BEGIN");
+$holder->query_safe("CREATE TEMP TABLE xid_holder(x int)");
+
+# Test 1: CLUSTER
+my $toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+my ($ret, $stdout, $stderr) =
+ $node->psql('db_target', 'CLUSTER rewrite_test USING rewrite_test_pkey');
+is($ret, 0, "CLUSTER succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "CLUSTER: no missing-chunk error");
+
+# Test 2: VACUUM FULL
+$toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+($ret, $stdout, $stderr) =
+ $node->psql('db_target', 'VACUUM FULL rewrite_test');
+is($ret, 0, "VACUUM FULL succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "VACUUM FULL: no missing-chunk error");
+
+# Test 3: REPACK
+$toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+($ret, $stdout, $stderr) = $node->psql('db_target', 'REPACK rewrite_test');
+is($ret, 0, "REPACK succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "REPACK: no missing-chunk error");
+
+# Test 4: CREATE INDEX on toasted column
+$toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+($ret, $stdout, $stderr) = $node->psql('db_target',
+ 'CREATE INDEX rewrite_test_data_idx ON rewrite_test (data)');
+is($ret, 0, "CREATE INDEX succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "CREATE INDEX: no missing-chunk error");
+
+# Cleanup.
+$holder->query_safe("COMMIT");
+$holder->quit;
+
+$node->stop;
+done_testing();
diff --git a/contrib/dblink/.gitignore b/contrib/dblink/.gitignore
index 5dcb3ff97235..33bc8d8ffeb3 100644
--- a/contrib/dblink/.gitignore
+++ b/contrib/dblink/.gitignore
@@ -2,3 +2,5 @@
/log/
/results/
/tmp_check/
+/output_iso/
+/tmp_check_iso/
diff --git a/contrib/dblink/Makefile b/contrib/dblink/Makefile
index fde0b49ddbbd..a0ba41280813 100644
--- a/contrib/dblink/Makefile
+++ b/contrib/dblink/Makefile
@@ -13,6 +13,8 @@ PGFILEDESC = "dblink - connect to other PostgreSQL databases"
REGRESS = dblink
REGRESS_OPTS = --dlpath=$(top_builddir)/src/test/regress
+ISOLATION = rewrite_stale_xmin
+ISOLATION_OPTS = --load-extension=dblink
TAP_TESTS = 1
ifdef USE_PGXS
diff --git a/contrib/dblink/expected/rewrite_stale_xmin.out b/contrib/dblink/expected/rewrite_stale_xmin.out
new file mode 100644
index 000000000000..2e5a7940ea26
--- /dev/null
+++ b/contrib/dblink/expected/rewrite_stale_xmin.out
@@ -0,0 +1,237 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_cluster s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_cluster:
+ CLUSTER rewrite_test USING rewrite_test_pkey;
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_vacuum_full s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_vacuum_full:
+ VACUUM FULL rewrite_test;
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_repack s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_repack:
+ REPACK rewrite_test;
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_create_index s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_create_index:
+ CREATE INDEX rewrite_test_data_idx ON rewrite_test (data);
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
diff --git a/contrib/dblink/meson.build b/contrib/dblink/meson.build
index e2489f41229f..86fbd372b880 100644
--- a/contrib/dblink/meson.build
+++ b/contrib/dblink/meson.build
@@ -36,6 +36,11 @@ tests += {
],
'regress_args': ['--dlpath', meson.project_build_root() / 'src/test/regress'],
},
+ 'isolation': {
+ 'specs': [
+ 'rewrite_stale_xmin',
+ ],
+ },
'tap': {
'tests': [
't/001_auth_scram.pl',
diff --git a/contrib/dblink/specs/rewrite_stale_xmin.spec b/contrib/dblink/specs/rewrite_stale_xmin.spec
new file mode 100644
index 000000000000..fe88e789f854
--- /dev/null
+++ b/contrib/dblink/specs/rewrite_stale_xmin.spec
@@ -0,0 +1,137 @@
+# Test that heap rewrites work when the rewriting backend's own xmin is
+# held back by a transaction in another database.
+
+setup
+{
+ CREATE DATABASE regress_other_db;
+}
+
+setup
+{
+ CREATE EXTENSION IF NOT EXISTS dblink;
+
+ CREATE TABLE rewrite_test (id int PRIMARY KEY, data text)
+ WITH (autovacuum_enabled = off);
+ ALTER TABLE rewrite_test ALTER COLUMN data SET STORAGE EXTERNAL;
+ INSERT INTO rewrite_test VALUES (1, repeat('x', 2500));
+ INSERT INTO rewrite_test VALUES (2, repeat('y', 2500));
+
+ -- Rename the toast table to a known name so we can VACUUM it directly.
+ SET allow_system_table_mods TO true;
+ DO $$DECLARE r record;
+ BEGIN
+ SELECT INTO r reltoastrelid::regclass::text AS table_name
+ FROM pg_class WHERE oid = 'rewrite_test'::regclass;
+ EXECUTE 'ALTER TABLE ' || r.table_name || ' RENAME TO rewrite_test_toast;';
+ END$$;
+ RESET allow_system_table_mods;
+}
+
+teardown
+{
+ DROP DATABASE regress_other_db;
+}
+
+# Session s1: use dblink to hold a transaction in another database.
+# This must start before the DELETE so its XID is older, ensuring
+# GetSnapshotData() holds back any rewrites.
+session s1
+step s1_hold_xmin
+{
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+}
+step s1_release
+{
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+}
+
+# Session s2: holds xmin in the SAME database temporarily, so the
+# main-table vacuum preserves the dead tuple as RECENTLY_DEAD.
+session s2
+step s2_begin
+{
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+}
+step s2_commit { COMMIT; }
+
+# Session s3: performs DML, split vacuum, and the rewrite.
+session s3
+step s3_delete
+{
+ DELETE FROM rewrite_test WHERE id = 1;
+}
+step s3_vacuum_main_only
+{
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+}
+step s3_vacuum_toast
+{
+ VACUUM pg_toast.rewrite_test_toast;
+}
+step s3_cluster
+{
+ CLUSTER rewrite_test USING rewrite_test_pkey;
+}
+step s3_vacuum_full
+{
+ VACUUM FULL rewrite_test;
+}
+step s3_repack
+{
+ REPACK rewrite_test;
+}
+step s3_create_index
+{
+ CREATE INDEX rewrite_test_data_idx ON rewrite_test (data);
+}
+
+teardown { DROP TABLE IF EXISTS rewrite_test; }
+
+# CLUSTER
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_cluster
+ s1_release
+
+# VACUUM FULL
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_vacuum_full
+ s1_release
+
+# REPACK
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_repack
+ s1_release
+
+# CREATE INDEX
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_create_index
+ s1_release
--
2.55.0
Attachments:
[text/plain] v3-0001-Fix-missing-chunk-errors-during-heap-rewrites-and.patch (36.6K, ../../[email protected]/2-v3-0001-Fix-missing-chunk-errors-during-heap-rewrites-and.patch)
download | inline diff:
From 2e9cc62deda3de859ee389772471734a2f4f9e5d Mon Sep 17 00:00:00 2001
From: Michael Paquier <[email protected]>
Date: Tue, 7 Jul 2026 14:20:30 +0900
Subject: [PATCH v3] Fix "missing chunk" errors during heap rewrites and index
builds
Blah, blah.
This problem has been reported recently by Alexander for REPACK, but it
is a much older issue (tested down to 9.5 for curiosity's sake, but I
suspect that it is much older than that).
XXX: This patch includes both TAP and isolation tests. We should remove
one pattern at the end.
Reported-by: Alexander Lakhin <[email protected]>
Discussion: https://postgr.es/m/[email protected]
Discussion: https://postgr.es/m/[email protected]
---
src/include/access/detoast.h | 9 +
src/include/access/heaptoast.h | 13 +-
src/include/access/rewriteheap.h | 4 +-
src/include/access/tableam.h | 24 +-
src/include/access/toast_helper.h | 2 +-
src/backend/access/common/detoast.c | 41 ++-
src/backend/access/heap/heapam.c | 4 +-
src/backend/access/heap/heapam_handler.c | 94 ++++++-
src/backend/access/heap/heaptoast.c | 22 +-
src/backend/access/heap/rewriteheap.c | 31 ++-
src/backend/access/table/toast_helper.c | 17 +-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/055_rewrite_stale_xmin.pl | 124 +++++++++
contrib/dblink/.gitignore | 2 +
contrib/dblink/Makefile | 2 +
.../dblink/expected/rewrite_stale_xmin.out | 237 ++++++++++++++++++
contrib/dblink/meson.build | 5 +
contrib/dblink/specs/rewrite_stale_xmin.spec | 137 ++++++++++
18 files changed, 718 insertions(+), 51 deletions(-)
create mode 100644 src/test/recovery/t/055_rewrite_stale_xmin.pl
create mode 100644 contrib/dblink/expected/rewrite_stale_xmin.out
create mode 100644 contrib/dblink/specs/rewrite_stale_xmin.spec
diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h
index fbd98181a3a1..c5eeed04b011 100644
--- a/src/include/access/detoast.h
+++ b/src/include/access/detoast.h
@@ -43,6 +43,15 @@ do { \
*/
extern varlena *detoast_external_attr(varlena *attr);
+/* ----------
+ * detoast_external_attr_extended() -
+ *
+ * Like detoast_external_attr, but returns NULL if toast chunks
+ * are missing (instead of raising an error).
+ * ----------
+ */
+extern varlena *detoast_external_attr_extended(varlena *attr);
+
/* ----------
* detoast_attr() -
*
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index 631cb1836b96..1be85db6ddd2 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -92,10 +92,14 @@
* heap_toast_insert_or_update -
*
* Called by heap_insert() and heap_update().
+ *
+ * If missing_ok is true, returns NULL when failing to fetch external
+ * TOAST data, instead of throwing an error.
* ----------
*/
extern HeapTuple heap_toast_insert_or_update(Relation rel, HeapTuple newtup,
- HeapTuple oldtup, uint32 options);
+ HeapTuple oldtup, uint32 options,
+ bool missing_ok);
/* ----------
* heap_toast_delete -
@@ -140,10 +144,13 @@ extern HeapTuple toast_build_flattened_tuple(TupleDesc tupleDesc,
* heap_fetch_toast_slice
*
* Fetch a slice from a toast value stored in a heap table.
+ * If missing_ok is true, returns false when chunks are missing
+ * instead of raising an error. Returns true on success.
* ----------
*/
-extern void heap_fetch_toast_slice(Relation toastrel, Oid valueid,
+extern bool heap_fetch_toast_slice(Relation toastrel, Oid valueid,
int32 attrsize, int32 sliceoffset,
- int32 slicelength, varlena *result);
+ int32 slicelength, varlena *result,
+ bool missing_ok);
#endif /* HEAPTOAST_H */
diff --git a/src/include/access/rewriteheap.h b/src/include/access/rewriteheap.h
index 6ccf7b45c04e..df4fa4fde710 100644
--- a/src/include/access/rewriteheap.h
+++ b/src/include/access/rewriteheap.h
@@ -25,8 +25,8 @@ extern RewriteState begin_heap_rewrite(Relation old_heap, Relation new_heap,
TransactionId oldest_xmin, TransactionId freeze_xid,
MultiXactId cutoff_multi);
extern void end_heap_rewrite(RewriteState state);
-extern void rewrite_heap_tuple(RewriteState state, HeapTuple old_tuple,
- HeapTuple new_tuple);
+extern bool rewrite_heap_tuple(RewriteState state, HeapTuple old_tuple,
+ HeapTuple new_tuple, bool missing_ok);
extern bool rewrite_heap_dead_tuple(RewriteState state, HeapTuple old_tuple);
/*
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index f2c36696bcad..a90b15dd82b3 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -782,12 +782,16 @@ typedef struct TableAmRoutine
* This callback is invoked when detoasting a value stored in a toast
* table implemented by this AM. See table_relation_fetch_toast_slice()
* for more details.
+ *
+ * If missing_ok is true, returns false when chunks are missing instead of
+ * raising an error. Returns true on success.
*/
- void (*relation_fetch_toast_slice) (Relation toastrel, Oid valueid,
+ bool (*relation_fetch_toast_slice) (Relation toastrel, Oid valueid,
int32 attrsize,
int32 sliceoffset,
int32 slicelength,
- varlena *result);
+ varlena *result,
+ bool missing_ok);
/* ------------------------------------------------------------------------
@@ -1981,16 +1985,20 @@ table_relation_toast_am(Relation rel)
*
* result is caller-allocated space into which the fetched bytes should be
* stored.
+ *
+ * missing_ok: if true, return false when toast chunks are missing instead
+ * of raising an error. Returns true on success.
*/
-static inline void
+static inline bool
table_relation_fetch_toast_slice(Relation toastrel, Oid valueid,
int32 attrsize, int32 sliceoffset,
- int32 slicelength, varlena *result)
+ int32 slicelength, varlena *result,
+ bool missing_ok)
{
- toastrel->rd_tableam->relation_fetch_toast_slice(toastrel, valueid,
- attrsize,
- sliceoffset, slicelength,
- result);
+ return toastrel->rd_tableam->relation_fetch_toast_slice(toastrel, valueid,
+ attrsize,
+ sliceoffset, slicelength,
+ result, missing_ok);
}
diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h
index 2ec92397f267..e548ba5596b4 100644
--- a/src/include/access/toast_helper.h
+++ b/src/include/access/toast_helper.h
@@ -101,7 +101,7 @@ typedef struct
#define TOASTCOL_IGNORE 0x0010
#define TOASTCOL_INCOMPRESSIBLE 0x0020
-extern void toast_tuple_init(ToastTupleContext *ttc);
+extern bool toast_tuple_init(ToastTupleContext *ttc, bool missing_ok);
extern int toast_tuple_find_biggest_attribute(ToastTupleContext *ttc,
bool for_compression,
bool check_main);
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index a6c1f3a734b2..c16b61a1cb1c 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -22,7 +22,7 @@
#include "utils/expandeddatum.h"
#include "utils/rel.h"
-static varlena *toast_fetch_datum(varlena *attr);
+static varlena *toast_fetch_datum(varlena *attr, bool missing_ok);
static varlena *toast_fetch_datum_slice(varlena *attr,
int32 sliceoffset,
int32 slicelength);
@@ -51,7 +51,7 @@ detoast_external_attr(varlena *attr)
/*
* This is an external stored plain value
*/
- result = toast_fetch_datum(attr);
+ result = toast_fetch_datum(attr, false);
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
@@ -102,6 +102,23 @@ detoast_external_attr(varlena *attr)
}
+/* ----------
+ * detoast_external_attr_extended -
+ *
+ * Like detoast_external_attr, but returns NULL if toast chunks are
+ * missing for external pointers.
+ * ----------
+ */
+varlena *
+detoast_external_attr_extended(varlena *attr)
+{
+ if (VARATT_IS_EXTERNAL_ONDISK(attr))
+ return toast_fetch_datum(attr, true);
+ else
+ return detoast_external_attr(attr);
+}
+
+
/* ----------
* detoast_attr -
*
@@ -120,7 +137,7 @@ detoast_attr(varlena *attr)
/*
* This is an externally stored datum --- fetch it back from there
*/
- attr = toast_fetch_datum(attr);
+ attr = toast_fetch_datum(attr, false);
/* If it's compressed, decompress it */
if (VARATT_IS_COMPRESSED(attr))
{
@@ -262,7 +279,7 @@ detoast_attr_slice(varlena *attr,
preslice = toast_fetch_datum_slice(attr, 0, max_size);
}
else
- preslice = toast_fetch_datum(attr);
+ preslice = toast_fetch_datum(attr, false);
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
@@ -337,10 +354,12 @@ detoast_attr_slice(varlena *attr,
*
* Reconstruct an in memory Datum from the chunks saved
* in the toast relation
+ *
+ * If missing_ok is true, returns NULL when chunks are missing.
* ----------
*/
static varlena *
-toast_fetch_datum(varlena *attr)
+toast_fetch_datum(varlena *attr, bool missing_ok)
{
Relation toastrel;
varlena *result;
@@ -372,8 +391,14 @@ toast_fetch_datum(varlena *attr)
toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock);
/* Fetch all chunks */
- table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
- attrsize, 0, attrsize, result);
+ if (!table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
+ attrsize, 0, attrsize, result,
+ missing_ok))
+ {
+ pfree(result);
+ table_close(toastrel, AccessShareLock);
+ return NULL;
+ }
/* Close toast table */
table_close(toastrel, AccessShareLock);
@@ -454,7 +479,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
/* Fetch all chunks */
table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
attrsize, sliceoffset, slicelength,
- result);
+ result, false);
/* Close toast table */
table_close(toastrel, AccessShareLock);
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index abfd8e8970a6..1d4018931200 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -2236,7 +2236,7 @@ heap_prepare_insert(Relation relation, HeapTuple tup, TransactionId xid,
return tup;
}
else if (HeapTupleHasExternal(tup) || tup->t_len > TOAST_TUPLE_THRESHOLD)
- return heap_toast_insert_or_update(relation, tup, NULL, options);
+ return heap_toast_insert_or_update(relation, tup, NULL, options, false);
else
return tup;
}
@@ -3871,7 +3871,7 @@ l2:
if (need_toast)
{
/* Note we always use WAL and FSM during updates */
- heaptup = heap_toast_insert_or_update(relation, newtup, &oldtup, 0);
+ heaptup = heap_toast_insert_or_update(relation, newtup, &oldtup, 0, false);
newtupsize = MAXALIGN(heaptup->t_len);
}
else
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 2268cc277bce..2cd9d0f4aeb9 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -19,6 +19,7 @@
*/
#include "postgres.h"
+#include "access/detoast.h"
#include "access/genam.h"
#include "access/heapam.h"
#include "access/heaptoast.h"
@@ -48,9 +49,10 @@
#include "utils/rel.h"
#include "utils/tuplesort.h"
-static void reform_and_rewrite_tuple(HeapTuple tuple,
+static bool reform_and_rewrite_tuple(HeapTuple tuple,
Relation OldHeap, Relation NewHeap,
- Datum *values, bool *isnull, RewriteState rwstate);
+ Datum *values, bool *isnull, RewriteState rwstate,
+ bool missing_ok);
static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap,
Relation NewHeap, Datum *values, bool *isnull,
BulkInsertState bistate);
@@ -717,6 +719,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
HeapTuple tuple;
Buffer buf;
bool isdead;
+ bool recently_dead = false;
CHECK_FOR_INTERRUPTS();
@@ -802,6 +805,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
break;
case HEAPTUPLE_RECENTLY_DEAD:
*tups_recently_dead += 1;
+ recently_dead = true;
pg_fallthrough;
case HEAPTUPLE_LIVE:
/* Live or recently dead, must copy it */
@@ -835,6 +839,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
RelationGetRelationName(OldHeap));
/* treat as recently dead */
*tups_recently_dead += 1;
+ recently_dead = true;
isdead = false;
break;
default:
@@ -881,8 +886,22 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
int64 ct_val[2];
if (!concurrent)
- reform_and_rewrite_tuple(tuple, OldHeap, NewHeap,
- values, isnull, rwstate);
+ {
+ if (!reform_and_rewrite_tuple(tuple, OldHeap, NewHeap,
+ values, isnull, rwstate,
+ recently_dead))
+ {
+ /*
+ * Missing TOAST chunks for a recently-dead tuple. Treat
+ * it as dead.
+ */
+ *tups_vacuumed += 1;
+ *num_tuples -= 1;
+ if (recently_dead)
+ *tups_recently_dead -= 1;
+ continue;
+ }
+ }
else
heap_insert_for_repack(tuple, OldHeap, NewHeap,
values, isnull, bistate);
@@ -934,10 +953,17 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
n_tuples += 1;
if (!concurrent)
- reform_and_rewrite_tuple(tuple,
- OldHeap, NewHeap,
- values, isnull,
- rwstate);
+ {
+ if (!reform_and_rewrite_tuple(tuple,
+ OldHeap, NewHeap,
+ values, isnull,
+ rwstate, true))
+ {
+ /* TOAST gone for a recently dead tuple */
+ n_tuples -= 1;
+ continue;
+ }
+ }
else
heap_insert_for_repack(tuple, OldHeap, NewHeap,
values, isnull, bistate);
@@ -1613,6 +1639,43 @@ heapam_index_build_range_scan(Relation heapRelation,
continue;
}
+ /*
+ * For RECENTLY_DEAD tuples, pre-detoast external TOAST values. We do
+ * this to detect for missing chunks at later stages, feeding inline
+ * data to the downstream access method code.
+ */
+ if (!tupleIsAlive)
+ {
+ TupleDesc tdesc = RelationGetDescr(heapRelation);
+ bool skip = false;
+
+ slot_getallattrs(slot);
+ for (int i = 0; i < tdesc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(tdesc, i);
+
+ if (att->attlen != -1 || att->attisdropped)
+ continue;
+ if (slot->tts_isnull[i])
+ continue;
+ if (VARATT_IS_EXTERNAL_ONDISK(DatumGetPointer(slot->tts_values[i])))
+ {
+ varlena *detoasted;
+
+ detoasted = detoast_external_attr_extended(
+ (varlena *) DatumGetPointer(slot->tts_values[i]));
+ if (detoasted == NULL)
+ {
+ skip = true;
+ break;
+ }
+ slot->tts_values[i] = PointerGetDatum(detoasted);
+ }
+ }
+ if (skip)
+ continue;
+ }
+
/*
* For the current heap tuple, extract all the attributes we use in
* this index, and note which are null. This also performs evaluation
@@ -2347,20 +2410,29 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate,
* SET WITHOUT OIDS.
*
* So, we must reconstruct the tuple from component Datums.
+ *
+ * Returns true on success, and false on failure if missing_ok is
+ * true.
*/
-static void
+static bool
reform_and_rewrite_tuple(HeapTuple tuple,
Relation OldHeap, Relation NewHeap,
- Datum *values, bool *isnull, RewriteState rwstate)
+ Datum *values, bool *isnull, RewriteState rwstate,
+ bool missing_ok)
{
HeapTuple newtuple;
newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull);
/* The heap rewrite module does the rest */
- rewrite_heap_tuple(rwstate, tuple, newtuple);
+ if (!rewrite_heap_tuple(rwstate, tuple, newtuple, missing_ok))
+ {
+ heap_freetuple(newtuple);
+ return false;
+ }
heap_freetuple(newtuple);
+ return true;
}
/*
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 03f885a25b07..38896a0c7492 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -94,7 +94,7 @@ heap_toast_delete(Relation rel, HeapTuple oldtup, bool is_speculative)
*/
HeapTuple
heap_toast_insert_or_update(Relation rel, HeapTuple newtup, HeapTuple oldtup,
- uint32 options)
+ uint32 options, bool missing_ok)
{
HeapTuple result_tuple;
TupleDesc tupleDesc;
@@ -154,7 +154,8 @@ heap_toast_insert_or_update(Relation rel, HeapTuple newtup, HeapTuple oldtup,
ttc.ttc_oldisnull = toast_oldisnull;
}
ttc.ttc_attr = toast_attr;
- toast_tuple_init(&ttc);
+ if (!toast_tuple_init(&ttc, missing_ok))
+ return NULL;
/* ----------
* Compress and/or save external until data fits into target length
@@ -621,11 +622,14 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
* sliceoffset is the byte offset within the TOAST value from which to fetch.
* slicelength is the number of bytes to be fetched from the TOAST value.
* result is the varlena into which the results should be written.
+ * missing_ok if true, return false on missing chunks instead of error.
+ *
+ * Returns true on success, false if missing_ok and chunks are missing.
*/
-void
+bool
heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
int32 sliceoffset, int32 slicelength,
- varlena *result)
+ varlena *result, bool missing_ok)
{
Relation *toastidxs;
ScanKeyData toastkey[3];
@@ -779,13 +783,23 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
* Final checks that we successfully fetched the datum
*/
if (expectedchunk != (endchunk + 1))
+ {
+ if (missing_ok)
+ {
+ systable_endscan_ordered(toastscan);
+ toast_close_indexes(toastidxs, num_indexes, AccessShareLock);
+ return false;
+ }
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
errmsg_internal("missing chunk number %d for toast value %u in %s",
expectedchunk, valueid,
RelationGetRelationName(toastrel))));
+ }
/* End scan and close indexes. */
systable_endscan_ordered(toastscan);
toast_close_indexes(toastidxs, num_indexes, AccessShareLock);
+
+ return true;
}
diff --git a/src/backend/access/heap/rewriteheap.c b/src/backend/access/heap/rewriteheap.c
index 5a5398a76ae7..fbb6e6ca1f94 100644
--- a/src/backend/access/heap/rewriteheap.c
+++ b/src/backend/access/heap/rewriteheap.c
@@ -211,7 +211,8 @@ typedef struct RewriteMappingDataEntry
/* prototypes for internal functions */
-static void raw_heap_insert(RewriteState state, HeapTuple tup);
+static bool raw_heap_insert(RewriteState state, HeapTuple tup,
+ bool missing_ok);
/* internal logical remapping prototypes */
static void logical_begin_heap_rewrite(RewriteState state);
@@ -309,7 +310,7 @@ end_heap_rewrite(RewriteState state)
while ((unresolved = hash_seq_search(&seq_status)) != NULL)
{
ItemPointerSetInvalid(&unresolved->tuple->t_data->t_ctid);
- raw_heap_insert(state, unresolved->tuple);
+ raw_heap_insert(state, unresolved->tuple, false);
}
/* Write the last page, if any */
@@ -338,9 +339,10 @@ end_heap_rewrite(RewriteState state)
* old_tuple original tuple in the old heap
* new_tuple new, rewritten tuple to be inserted to new heap
*/
-void
+bool
rewrite_heap_tuple(RewriteState state,
- HeapTuple old_tuple, HeapTuple new_tuple)
+ HeapTuple old_tuple, HeapTuple new_tuple,
+ bool missing_ok)
{
MemoryContext old_cxt;
ItemPointerData old_tid;
@@ -437,7 +439,7 @@ rewrite_heap_tuple(RewriteState state,
* tuple will be written.
*/
MemoryContextSwitchTo(old_cxt);
- return;
+ return true;
}
}
@@ -455,7 +457,13 @@ rewrite_heap_tuple(RewriteState state,
ItemPointerData new_tid;
/* Insert the tuple and find out where it's put in new_heap */
- raw_heap_insert(state, new_tuple);
+ if (!raw_heap_insert(state, new_tuple, missing_ok))
+ {
+ if (free_new)
+ heap_freetuple(new_tuple);
+ MemoryContextSwitchTo(old_cxt);
+ return false;
+ }
new_tid = new_tuple->t_self;
logical_rewrite_heap_tuple(state, old_tid, new_tuple);
@@ -532,6 +540,7 @@ rewrite_heap_tuple(RewriteState state,
}
MemoryContextSwitchTo(old_cxt);
+ return true;
}
/*
@@ -593,8 +602,8 @@ rewrite_heap_dead_tuple(RewriteState state, HeapTuple old_tuple)
* tuple is invalid on entry, it's replaced with the new TID as well (in
* the inserted data only, not in the caller's copy).
*/
-static void
-raw_heap_insert(RewriteState state, HeapTuple tup)
+static bool
+raw_heap_insert(RewriteState state, HeapTuple tup, bool missing_ok)
{
Page page;
Size pageFreeSpace,
@@ -628,7 +637,9 @@ raw_heap_insert(RewriteState state, HeapTuple tup)
options |= HEAP_INSERT_NO_LOGICAL;
heaptup = heap_toast_insert_or_update(state->rs_new_rel, tup, NULL,
- options);
+ options, missing_ok);
+ if (heaptup == NULL)
+ return false;
}
else
heaptup = tup;
@@ -702,6 +713,8 @@ raw_heap_insert(RewriteState state, HeapTuple tup)
/* If heaptup is a private copy, release it. */
if (heaptup != tup)
heap_freetuple(heaptup);
+
+ return true;
}
/* ------------------------------------------------------------------------
diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c
index 2f2022d99510..7e98c01a5eee 100644
--- a/src/backend/access/table/toast_helper.c
+++ b/src/backend/access/table/toast_helper.c
@@ -36,9 +36,12 @@
* toast_flags is just a single uint8, but toast_attr is a caller-provided
* array with a length equal to tupleDesc->natts. The caller need not
* perform any initialization of the array before calling this function.
+ *
+ * If missing_ok is true and an external TOAST value cannot be fetched,
+ * returns false. Returns true on success.
*/
-void
-toast_tuple_init(ToastTupleContext *ttc)
+bool
+toast_tuple_init(ToastTupleContext *ttc, bool missing_ok)
{
TupleDesc tupleDesc = ttc->ttc_rel->rd_att;
int numAttrs = tupleDesc->natts;
@@ -137,7 +140,13 @@ toast_tuple_init(ToastTupleContext *ttc)
if (VARATT_IS_EXTERNAL(new_value))
{
ttc->ttc_attr[i].tai_oldexternal = new_value;
- if (att->attstorage == TYPSTORAGE_PLAIN)
+ if (missing_ok && VARATT_IS_EXTERNAL_ONDISK(new_value))
+ {
+ new_value = detoast_external_attr_extended(new_value);
+ if (new_value == NULL)
+ return false;
+ }
+ else if (att->attstorage == TYPSTORAGE_PLAIN)
new_value = detoast_attr(new_value);
else
new_value = detoast_external_attr(new_value);
@@ -159,6 +168,8 @@ toast_tuple_init(ToastTupleContext *ttc)
ttc->ttc_attr[i].tai_colflags |= TOASTCOL_IGNORE;
}
}
+
+ return true;
}
/*
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index ad0d85f41897..7e583ee39a10 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -63,6 +63,7 @@ tests += {
't/052_checkpoint_segment_missing.pl',
't/053_standby_login_event_trigger.pl',
't/054_unlogged_sequence_promotion.pl',
+ 't/055_rewrite_stale_xmin.pl',
],
},
}
diff --git a/src/test/recovery/t/055_rewrite_stale_xmin.pl b/src/test/recovery/t/055_rewrite_stale_xmin.pl
new file mode 100644
index 000000000000..dad95c0cf967
--- /dev/null
+++ b/src/test/recovery/t/055_rewrite_stale_xmin.pl
@@ -0,0 +1,124 @@
+# Copyright (c) 2025-2026, PostgreSQL Global Development Group
+#
+# Test that heap rewrites (CLUSTER, VACUUM FULL, REPACK) and index builds
+# work in the case when the backend's own xmin is held back by a transaction
+# in another database.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+
+my $node = PostgreSQL::Test::Cluster->new('primary');
+$node->init;
+# No Autovacuum, no random interferences.
+$node->append_conf(
+ 'postgresql.conf', qq[
+autovacuum = off
+]);
+$node->start;
+
+# Create two databases.
+$node->safe_psql('postgres', 'CREATE DATABASE db_holder');
+$node->safe_psql('postgres', 'CREATE DATABASE db_target');
+
+# Set up a table with TOAST data. Returns the toast table name.
+sub setup_table
+{
+ my ($tblname, $extra_ddl) = @_;
+ $extra_ddl //= '';
+
+ $node->safe_psql(
+ 'db_target', qq{
+ DROP TABLE IF EXISTS $tblname;
+ CREATE TABLE $tblname (id int PRIMARY KEY, data text)
+ WITH (autovacuum_enabled = off);
+ ALTER TABLE $tblname ALTER COLUMN data SET STORAGE EXTERNAL;
+ INSERT INTO $tblname VALUES (1, repeat('x', 2500));
+ INSERT INTO $tblname VALUES (2, repeat('z', 2500));
+ $extra_ddl
+ });
+
+ my $toast = $node->safe_psql(
+ 'db_target', qq{
+ SELECT 'pg_toast.' || relname FROM pg_class
+ WHERE oid = (SELECT reltoastrelid FROM pg_class
+ WHERE relname = '$tblname');
+ });
+ chomp $toast;
+ return $toast;
+}
+
+# Create precondition state. Requires that $holder is already running with
+# a write XID.
+sub create_missing_toast_state
+{
+ my ($tblname, $toast_table) = @_;
+
+ # Hold xmin in same DB to prevent main tuple removal.
+ my $xmin_holder = $node->background_psql('db_target', on_error_stop => 1);
+ $xmin_holder->query_safe("BEGIN ISOLATION LEVEL REPEATABLE READ");
+ $xmin_holder->query_safe("SELECT 1");
+
+ # Delete the row with TOAST data.
+ $node->safe_psql('db_target', "DELETE FROM $tblname WHERE id = 1");
+
+ # VACUUM main table only. The xmin_holder keeps the dead tuple.
+ $node->safe_psql('db_target', "VACUUM (PROCESS_TOAST false) $tblname");
+
+ # Release same-DB xmin.
+ $xmin_holder->query_safe("COMMIT");
+ $xmin_holder->quit;
+
+ # Vacuum toast table: toast chunks now DEAD and removed.
+ $node->safe_psql('db_target', "VACUUM $toast_table");
+}
+
+# Open a transaction holding an XID before any deletes. This XID will hold
+# back GetSnapshotData() globally, on a different database than the one
+# running the various rewrite operations.
+my $holder = $node->background_psql('db_holder', on_error_stop => 1);
+$holder->query_safe("BEGIN");
+$holder->query_safe("CREATE TEMP TABLE xid_holder(x int)");
+
+# Test 1: CLUSTER
+my $toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+my ($ret, $stdout, $stderr) =
+ $node->psql('db_target', 'CLUSTER rewrite_test USING rewrite_test_pkey');
+is($ret, 0, "CLUSTER succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "CLUSTER: no missing-chunk error");
+
+# Test 2: VACUUM FULL
+$toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+($ret, $stdout, $stderr) =
+ $node->psql('db_target', 'VACUUM FULL rewrite_test');
+is($ret, 0, "VACUUM FULL succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "VACUUM FULL: no missing-chunk error");
+
+# Test 3: REPACK
+$toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+($ret, $stdout, $stderr) = $node->psql('db_target', 'REPACK rewrite_test');
+is($ret, 0, "REPACK succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "REPACK: no missing-chunk error");
+
+# Test 4: CREATE INDEX on toasted column
+$toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+($ret, $stdout, $stderr) = $node->psql('db_target',
+ 'CREATE INDEX rewrite_test_data_idx ON rewrite_test (data)');
+is($ret, 0, "CREATE INDEX succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "CREATE INDEX: no missing-chunk error");
+
+# Cleanup.
+$holder->query_safe("COMMIT");
+$holder->quit;
+
+$node->stop;
+done_testing();
diff --git a/contrib/dblink/.gitignore b/contrib/dblink/.gitignore
index 5dcb3ff97235..33bc8d8ffeb3 100644
--- a/contrib/dblink/.gitignore
+++ b/contrib/dblink/.gitignore
@@ -2,3 +2,5 @@
/log/
/results/
/tmp_check/
+/output_iso/
+/tmp_check_iso/
diff --git a/contrib/dblink/Makefile b/contrib/dblink/Makefile
index fde0b49ddbbd..a0ba41280813 100644
--- a/contrib/dblink/Makefile
+++ b/contrib/dblink/Makefile
@@ -13,6 +13,8 @@ PGFILEDESC = "dblink - connect to other PostgreSQL databases"
REGRESS = dblink
REGRESS_OPTS = --dlpath=$(top_builddir)/src/test/regress
+ISOLATION = rewrite_stale_xmin
+ISOLATION_OPTS = --load-extension=dblink
TAP_TESTS = 1
ifdef USE_PGXS
diff --git a/contrib/dblink/expected/rewrite_stale_xmin.out b/contrib/dblink/expected/rewrite_stale_xmin.out
new file mode 100644
index 000000000000..2e5a7940ea26
--- /dev/null
+++ b/contrib/dblink/expected/rewrite_stale_xmin.out
@@ -0,0 +1,237 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_cluster s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_cluster:
+ CLUSTER rewrite_test USING rewrite_test_pkey;
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_vacuum_full s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_vacuum_full:
+ VACUUM FULL rewrite_test;
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_repack s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_repack:
+ REPACK rewrite_test;
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_create_index s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_create_index:
+ CREATE INDEX rewrite_test_data_idx ON rewrite_test (data);
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
diff --git a/contrib/dblink/meson.build b/contrib/dblink/meson.build
index e2489f41229f..86fbd372b880 100644
--- a/contrib/dblink/meson.build
+++ b/contrib/dblink/meson.build
@@ -36,6 +36,11 @@ tests += {
],
'regress_args': ['--dlpath', meson.project_build_root() / 'src/test/regress'],
},
+ 'isolation': {
+ 'specs': [
+ 'rewrite_stale_xmin',
+ ],
+ },
'tap': {
'tests': [
't/001_auth_scram.pl',
diff --git a/contrib/dblink/specs/rewrite_stale_xmin.spec b/contrib/dblink/specs/rewrite_stale_xmin.spec
new file mode 100644
index 000000000000..fe88e789f854
--- /dev/null
+++ b/contrib/dblink/specs/rewrite_stale_xmin.spec
@@ -0,0 +1,137 @@
+# Test that heap rewrites work when the rewriting backend's own xmin is
+# held back by a transaction in another database.
+
+setup
+{
+ CREATE DATABASE regress_other_db;
+}
+
+setup
+{
+ CREATE EXTENSION IF NOT EXISTS dblink;
+
+ CREATE TABLE rewrite_test (id int PRIMARY KEY, data text)
+ WITH (autovacuum_enabled = off);
+ ALTER TABLE rewrite_test ALTER COLUMN data SET STORAGE EXTERNAL;
+ INSERT INTO rewrite_test VALUES (1, repeat('x', 2500));
+ INSERT INTO rewrite_test VALUES (2, repeat('y', 2500));
+
+ -- Rename the toast table to a known name so we can VACUUM it directly.
+ SET allow_system_table_mods TO true;
+ DO $$DECLARE r record;
+ BEGIN
+ SELECT INTO r reltoastrelid::regclass::text AS table_name
+ FROM pg_class WHERE oid = 'rewrite_test'::regclass;
+ EXECUTE 'ALTER TABLE ' || r.table_name || ' RENAME TO rewrite_test_toast;';
+ END$$;
+ RESET allow_system_table_mods;
+}
+
+teardown
+{
+ DROP DATABASE regress_other_db;
+}
+
+# Session s1: use dblink to hold a transaction in another database.
+# This must start before the DELETE so its XID is older, ensuring
+# GetSnapshotData() holds back any rewrites.
+session s1
+step s1_hold_xmin
+{
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+}
+step s1_release
+{
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+}
+
+# Session s2: holds xmin in the SAME database temporarily, so the
+# main-table vacuum preserves the dead tuple as RECENTLY_DEAD.
+session s2
+step s2_begin
+{
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+}
+step s2_commit { COMMIT; }
+
+# Session s3: performs DML, split vacuum, and the rewrite.
+session s3
+step s3_delete
+{
+ DELETE FROM rewrite_test WHERE id = 1;
+}
+step s3_vacuum_main_only
+{
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+}
+step s3_vacuum_toast
+{
+ VACUUM pg_toast.rewrite_test_toast;
+}
+step s3_cluster
+{
+ CLUSTER rewrite_test USING rewrite_test_pkey;
+}
+step s3_vacuum_full
+{
+ VACUUM FULL rewrite_test;
+}
+step s3_repack
+{
+ REPACK rewrite_test;
+}
+step s3_create_index
+{
+ CREATE INDEX rewrite_test_data_idx ON rewrite_test (data);
+}
+
+teardown { DROP TABLE IF EXISTS rewrite_test; }
+
+# CLUSTER
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_cluster
+ s1_release
+
+# VACUUM FULL
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_vacuum_full
+ s1_release
+
+# REPACK
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_repack
+ s1_release
+
+# CREATE INDEX
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_create_index
+ s1_release
--
2.55.0
[application/pgp-signature] signature.asc (833B, ../../[email protected]/3-signature.asc)
download
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: BUG #19519: REPACK can fail due to missing chunk for toast value
2026-06-14 07:00 BUG #19519: REPACK can fail due to missing chunk for toast value PG Bug reporting form <[email protected]>
2026-06-17 17:03 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-03 06:45 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Imran Zaheer <[email protected]>
2026-07-06 06:36 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 18:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
2026-07-06 23:18 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 23:51 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
2026-07-07 01:42 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-07 05:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
@ 2026-07-07 17:10 ` Srinath Reddy Sadipiralla <[email protected]>
2026-07-07 22:15 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Srinath Reddy Sadipiralla @ 2026-07-07 17:10 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Matthias van de Meent <[email protected]>; Imran Zaheer <[email protected]>; Alexander Lakhin <[email protected]>; PostgreSQL mailing lists <[email protected]>; Konstantin Knizhnik <[email protected]>
Hi,
On Tue, Jul 7, 2026 at 10:52 AM Michael Paquier <[email protected]> wrote:
>
> Option 2 leads to a much nicer result overall, with the attached
> passing my regression tests and Alexander's case as well (highly
> concurrent thing sent yesterday). The rest of the patch is kind of
> boring, where I have been trying to get a missing_ok state across the
> stack to let the toast slice fetch bypass the case of missing chunks
> when we are OK with it due to the rewrites. That still feels crude,
> but the simplicity is appealing here, as much as the rather low
> invasiveness.
>
> Thoughts or comments?
With the initial reproducer, I originally thought this bug was driven
entirely
by a long-running query in another database causing a self-inflicted horizon
drag on the rewrite operation. My theory was that the cross-database sleeper
drags down the global xmin, which infects the snapshot held by VACUUM FULL.
During ComputeXidHorizons, VACUUM FULL ends up dragging its own garbage
collection horizon backward. Lazy vacuum avoids this trap because it
utilizes the
PROC_IN_VACUUM flag to ignore its own snapshot.Since we cannot simply apply
PROC_IN_VACUUM to VACUUM FULL (it needs standard MVCC snapshots to
safely evaluate index expressions), and we can't remove it from lazy vacuum
(which would cause bloat), patching the rewrite operation to explicitly
ignore its
own snapshot during GC calculation (excludeMyself) seemed like the correct
fix.
The script proved that OldestXmin can be dragged backward by other
processes,
bypassing the patch entirely.If a completely separate process in the same
database
(such as a simple \d catalog lookup) requests a global snapshot, it gets
"infected" by
the long-running query in the other database. This new query now sits in
the local ProcArray.
Lazy Vacuum runs, ignores the cross-database sleeper, calculates an
aggressive horizon,
and physically removes the TOAST chunks.VACUUM FULL runs, sees the local
carrier
query, and is forced to respect its older snapshot. Its horizon is dragged
backward, it
assumes the main tuples are RECENTLY_DEAD, attempts to copy them, and
crashes
because the TOAST chunks are gone.
so my initial thoughts are that until there's dependency of
OldestXmin calculation on
global snapshot , the rewrite horizon getting dragged back seems
inevitable, so i think
compared to other approaches in the thread ,your approach of ignoring the
fact of
missing chunks seems good. I quickly reviewed the core logic of treating
the missing
chunks as dead and LGTM will further look into this and test.
--
Thanks :)
Srinath Reddy Sadipiralla
EDB: https://www.enterprisedb.com/
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: BUG #19519: REPACK can fail due to missing chunk for toast value
2026-06-14 07:00 BUG #19519: REPACK can fail due to missing chunk for toast value PG Bug reporting form <[email protected]>
2026-06-17 17:03 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-03 06:45 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Imran Zaheer <[email protected]>
2026-07-06 06:36 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 18:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
2026-07-06 23:18 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 23:51 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
2026-07-07 01:42 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-07 05:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-07 17:10 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
@ 2026-07-07 22:15 ` Michael Paquier <[email protected]>
2026-07-08 01:10 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Michael Paquier @ 2026-07-07 22:15 UTC (permalink / raw)
To: Srinath Reddy Sadipiralla <[email protected]>; +Cc: Matthias van de Meent <[email protected]>; Imran Zaheer <[email protected]>; Alexander Lakhin <[email protected]>; PostgreSQL mailing lists <[email protected]>; Konstantin Knizhnik <[email protected]>
On Tue, Jul 07, 2026 at 10:40:12PM +0530, Srinath Reddy Sadipiralla wrote:
> I quickly reviewed the core logic of treating the missing
> chunks as dead and LGTM will further look into this and test.
Thanks for that. This is tricky and invasive enough that this
warrants more eyes.
One thing that I still don't like much in the patch as written is my
use of a missing_ok argument, which feels super confusing as it
applies only to the underlying external TOAST, if the relation has
any. I'd be tempted to rewrite this portion of the patch with a
uint32 flags. Even if we assign one value for now (say MISSING_TOAST,
MISSING_TOAST_OK or whatever), it would allow more flexibility on ABI
grounds if we want more like states in the future across this portion
of the stack. Again, I strongly doubt that we will be able to
backpatch any of that.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: BUG #19519: REPACK can fail due to missing chunk for toast value
2026-06-14 07:00 BUG #19519: REPACK can fail due to missing chunk for toast value PG Bug reporting form <[email protected]>
2026-06-17 17:03 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-03 06:45 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Imran Zaheer <[email protected]>
2026-07-06 06:36 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 18:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
2026-07-06 23:18 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 23:51 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
2026-07-07 01:42 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-07 05:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-07 17:10 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-07 22:15 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
@ 2026-07-08 01:10 ` Michael Paquier <[email protected]>
2026-07-08 12:00 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Heikki Linnakangas <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Michael Paquier @ 2026-07-08 01:10 UTC (permalink / raw)
To: Srinath Reddy Sadipiralla <[email protected]>; +Cc: Matthias van de Meent <[email protected]>; Imran Zaheer <[email protected]>; Alexander Lakhin <[email protected]>; PostgreSQL mailing lists <[email protected]>; Konstantin Knizhnik <[email protected]>
On Wed, Jul 08, 2026 at 07:15:16AM +0900, Michael Paquier wrote:
> One thing that I still don't like much in the patch as written is my
> use of a missing_ok argument, which feels super confusing as it
> applies only to the underlying external TOAST, if the relation has
> any. I'd be tempted to rewrite this portion of the patch with a
> uint32 flags. Even if we assign one value for now (say MISSING_TOAST,
> MISSING_TOAST_OK or whatever), it would allow more flexibility on ABI
> grounds if we want more like states in the future across this portion
> of the stack. Again, I strongly doubt that we will be able to
> backpatch any of that.
A couple of extra notes while I do not forget about that stuff.. The
patch may be better split into two if we go with this approach, as the
index build and rewrite paths require different solutions:
- One for the rewrite path. It makes little sense to do any kind of
aggressive early detoasting because it may be wasteful due to the
tuple rewrites that update their data with only copies by reference
(main relation tuple is rewritten, reuses the same external TOAST
tuple). For workloads where UPDATEs do not touch the TOASTed
attributes, that would be a waste.
- One for the index build path, which is actually too aggressive with
its early detoasting, now that I think about it. There should be no
need to perform a detoast for anything else than the attributes that
are used in the index definition or the attributes that are used in
index expressions. So as written this patch would lead to a
regression.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: BUG #19519: REPACK can fail due to missing chunk for toast value
2026-06-14 07:00 BUG #19519: REPACK can fail due to missing chunk for toast value PG Bug reporting form <[email protected]>
2026-06-17 17:03 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-03 06:45 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Imran Zaheer <[email protected]>
2026-07-06 06:36 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 18:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
2026-07-06 23:18 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 23:51 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
2026-07-07 01:42 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-07 05:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-07 17:10 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-07 22:15 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-08 01:10 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
@ 2026-07-08 12:00 ` Heikki Linnakangas <[email protected]>
2026-07-08 15:00 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Imran Zaheer <[email protected]>
2026-07-09 02:47 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-09 05:17 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Konstantin Knizhnik <[email protected]>
0 siblings, 3 replies; 21+ messages in thread
From: Heikki Linnakangas @ 2026-07-08 12:00 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; +Cc: Matthias van de Meent <[email protected]>; Imran Zaheer <[email protected]>; Alexander Lakhin <[email protected]>; PostgreSQL mailing lists <[email protected]>; Konstantin Knizhnik <[email protected]>
On 08/07/2026 04:10, Michael Paquier wrote:
> On Wed, Jul 08, 2026 at 07:15:16AM +0900, Michael Paquier wrote:
>> One thing that I still don't like much in the patch as written is my
>> use of a missing_ok argument, which feels super confusing as it
>> applies only to the underlying external TOAST, if the relation has
>> any. I'd be tempted to rewrite this portion of the patch with a
>> uint32 flags. Even if we assign one value for now (say MISSING_TOAST,
>> MISSING_TOAST_OK or whatever), it would allow more flexibility on ABI
>> grounds if we want more like states in the future across this portion
>> of the stack. Again, I strongly doubt that we will be able to
>> backpatch any of that.
Yeah, boolean arguments in general can be confusing. Especially
arguments like "missing_ok" where it's hard to remember which behavior
is true and which is false. An IDE that shows the name of the argument
helps, but having the caller say "MISSING_TOAST_OK" rather than "true"
is much clearer.
For backpatching, I think we could smuggle the flag in a global
variable, to avoid changing the signature of the
relation_fetch_toast_slice() callback. See attached patch 0002 on top of
your 0001 patch (which is also attached for completeness). It doesn't
fix the problem for hypothetical 3rd party tableam implementations that
implement their own relation_fetch_toast_slice() callback. But do such
extensions even exist? If yes, they could be fixed too by also checking
the global variable.
I thought about adding an extra "extended" callback next to
relation_fetch_toast_slice(), but I don't see a way to add functions to
TableAmRoutine in an ABI-compatible way. For the future, we might want
to store sizeof(TableAmRoutine) in the struct itself, so that we could
add fields to it in minor versions without breaking the API, in case we
need something like this again.
> A couple of extra notes while I do not forget about that stuff.. The
> patch may be better split into two if we go with this approach, as the
> index build and rewrite paths require different solutions:
> - One for the rewrite path. It makes little sense to do any kind of
> aggressive early detoasting because it may be wasteful due to the
> tuple rewrites that update their data with only copies by reference
> (main relation tuple is rewritten, reuses the same external TOAST
> tuple). For workloads where UPDATEs do not touch the TOASTed
> attributes, that would be a waste.
I didn't understand this part. Do you mean when rebuilding the indexes
after rewriting the heap? I didn't think we support storing toasted
external datums in an index at all.
> - One for the index build path, which is actually too aggressive with
> its early detoasting, now that I think about it. There should be no
> need to perform a detoast for anything else than the attributes that
> are used in the index definition or the attributes that are used in
> index expressions. So as written this patch would lead to a
> regression.
Yeah, although I wouldn't be too worried about the performance here. The
penalty would be when building an index on values that are large enough
to be toasted, with lots of RECENTLY_DEAD tuples. That doesn't seem like
a very common case.
- Heikki
Attachments:
[text/x-patch] v3-0001-Fix-missing-chunk-errors-during-heap-rewrites-and.patch (36.6K, ../../[email protected]/2-v3-0001-Fix-missing-chunk-errors-during-heap-rewrites-and.patch)
download | inline diff:
From 2e9cc62deda3de859ee389772471734a2f4f9e5d Mon Sep 17 00:00:00 2001
From: Michael Paquier <[email protected]>
Date: Tue, 7 Jul 2026 14:20:30 +0900
Subject: [PATCH v3] Fix "missing chunk" errors during heap rewrites and index
builds
Blah, blah.
This problem has been reported recently by Alexander for REPACK, but it
is a much older issue (tested down to 9.5 for curiosity's sake, but I
suspect that it is much older than that).
XXX: This patch includes both TAP and isolation tests. We should remove
one pattern at the end.
Reported-by: Alexander Lakhin <[email protected]>
Discussion: https://postgr.es/m/[email protected]
Discussion: https://postgr.es/m/[email protected]
---
src/include/access/detoast.h | 9 +
src/include/access/heaptoast.h | 13 +-
src/include/access/rewriteheap.h | 4 +-
src/include/access/tableam.h | 24 +-
src/include/access/toast_helper.h | 2 +-
src/backend/access/common/detoast.c | 41 ++-
src/backend/access/heap/heapam.c | 4 +-
src/backend/access/heap/heapam_handler.c | 94 ++++++-
src/backend/access/heap/heaptoast.c | 22 +-
src/backend/access/heap/rewriteheap.c | 31 ++-
src/backend/access/table/toast_helper.c | 17 +-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/055_rewrite_stale_xmin.pl | 124 +++++++++
contrib/dblink/.gitignore | 2 +
contrib/dblink/Makefile | 2 +
.../dblink/expected/rewrite_stale_xmin.out | 237 ++++++++++++++++++
contrib/dblink/meson.build | 5 +
contrib/dblink/specs/rewrite_stale_xmin.spec | 137 ++++++++++
18 files changed, 718 insertions(+), 51 deletions(-)
create mode 100644 src/test/recovery/t/055_rewrite_stale_xmin.pl
create mode 100644 contrib/dblink/expected/rewrite_stale_xmin.out
create mode 100644 contrib/dblink/specs/rewrite_stale_xmin.spec
diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h
index fbd98181a3a1..c5eeed04b011 100644
--- a/src/include/access/detoast.h
+++ b/src/include/access/detoast.h
@@ -43,6 +43,15 @@ do { \
*/
extern varlena *detoast_external_attr(varlena *attr);
+/* ----------
+ * detoast_external_attr_extended() -
+ *
+ * Like detoast_external_attr, but returns NULL if toast chunks
+ * are missing (instead of raising an error).
+ * ----------
+ */
+extern varlena *detoast_external_attr_extended(varlena *attr);
+
/* ----------
* detoast_attr() -
*
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index 631cb1836b96..1be85db6ddd2 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -92,10 +92,14 @@
* heap_toast_insert_or_update -
*
* Called by heap_insert() and heap_update().
+ *
+ * If missing_ok is true, returns NULL when failing to fetch external
+ * TOAST data, instead of throwing an error.
* ----------
*/
extern HeapTuple heap_toast_insert_or_update(Relation rel, HeapTuple newtup,
- HeapTuple oldtup, uint32 options);
+ HeapTuple oldtup, uint32 options,
+ bool missing_ok);
/* ----------
* heap_toast_delete -
@@ -140,10 +144,13 @@ extern HeapTuple toast_build_flattened_tuple(TupleDesc tupleDesc,
* heap_fetch_toast_slice
*
* Fetch a slice from a toast value stored in a heap table.
+ * If missing_ok is true, returns false when chunks are missing
+ * instead of raising an error. Returns true on success.
* ----------
*/
-extern void heap_fetch_toast_slice(Relation toastrel, Oid valueid,
+extern bool heap_fetch_toast_slice(Relation toastrel, Oid valueid,
int32 attrsize, int32 sliceoffset,
- int32 slicelength, varlena *result);
+ int32 slicelength, varlena *result,
+ bool missing_ok);
#endif /* HEAPTOAST_H */
diff --git a/src/include/access/rewriteheap.h b/src/include/access/rewriteheap.h
index 6ccf7b45c04e..df4fa4fde710 100644
--- a/src/include/access/rewriteheap.h
+++ b/src/include/access/rewriteheap.h
@@ -25,8 +25,8 @@ extern RewriteState begin_heap_rewrite(Relation old_heap, Relation new_heap,
TransactionId oldest_xmin, TransactionId freeze_xid,
MultiXactId cutoff_multi);
extern void end_heap_rewrite(RewriteState state);
-extern void rewrite_heap_tuple(RewriteState state, HeapTuple old_tuple,
- HeapTuple new_tuple);
+extern bool rewrite_heap_tuple(RewriteState state, HeapTuple old_tuple,
+ HeapTuple new_tuple, bool missing_ok);
extern bool rewrite_heap_dead_tuple(RewriteState state, HeapTuple old_tuple);
/*
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index f2c36696bcad..a90b15dd82b3 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -782,12 +782,16 @@ typedef struct TableAmRoutine
* This callback is invoked when detoasting a value stored in a toast
* table implemented by this AM. See table_relation_fetch_toast_slice()
* for more details.
+ *
+ * If missing_ok is true, returns false when chunks are missing instead of
+ * raising an error. Returns true on success.
*/
- void (*relation_fetch_toast_slice) (Relation toastrel, Oid valueid,
+ bool (*relation_fetch_toast_slice) (Relation toastrel, Oid valueid,
int32 attrsize,
int32 sliceoffset,
int32 slicelength,
- varlena *result);
+ varlena *result,
+ bool missing_ok);
/* ------------------------------------------------------------------------
@@ -1981,16 +1985,20 @@ table_relation_toast_am(Relation rel)
*
* result is caller-allocated space into which the fetched bytes should be
* stored.
+ *
+ * missing_ok: if true, return false when toast chunks are missing instead
+ * of raising an error. Returns true on success.
*/
-static inline void
+static inline bool
table_relation_fetch_toast_slice(Relation toastrel, Oid valueid,
int32 attrsize, int32 sliceoffset,
- int32 slicelength, varlena *result)
+ int32 slicelength, varlena *result,
+ bool missing_ok)
{
- toastrel->rd_tableam->relation_fetch_toast_slice(toastrel, valueid,
- attrsize,
- sliceoffset, slicelength,
- result);
+ return toastrel->rd_tableam->relation_fetch_toast_slice(toastrel, valueid,
+ attrsize,
+ sliceoffset, slicelength,
+ result, missing_ok);
}
diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h
index 2ec92397f267..e548ba5596b4 100644
--- a/src/include/access/toast_helper.h
+++ b/src/include/access/toast_helper.h
@@ -101,7 +101,7 @@ typedef struct
#define TOASTCOL_IGNORE 0x0010
#define TOASTCOL_INCOMPRESSIBLE 0x0020
-extern void toast_tuple_init(ToastTupleContext *ttc);
+extern bool toast_tuple_init(ToastTupleContext *ttc, bool missing_ok);
extern int toast_tuple_find_biggest_attribute(ToastTupleContext *ttc,
bool for_compression,
bool check_main);
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index a6c1f3a734b2..c16b61a1cb1c 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -22,7 +22,7 @@
#include "utils/expandeddatum.h"
#include "utils/rel.h"
-static varlena *toast_fetch_datum(varlena *attr);
+static varlena *toast_fetch_datum(varlena *attr, bool missing_ok);
static varlena *toast_fetch_datum_slice(varlena *attr,
int32 sliceoffset,
int32 slicelength);
@@ -51,7 +51,7 @@ detoast_external_attr(varlena *attr)
/*
* This is an external stored plain value
*/
- result = toast_fetch_datum(attr);
+ result = toast_fetch_datum(attr, false);
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
@@ -102,6 +102,23 @@ detoast_external_attr(varlena *attr)
}
+/* ----------
+ * detoast_external_attr_extended -
+ *
+ * Like detoast_external_attr, but returns NULL if toast chunks are
+ * missing for external pointers.
+ * ----------
+ */
+varlena *
+detoast_external_attr_extended(varlena *attr)
+{
+ if (VARATT_IS_EXTERNAL_ONDISK(attr))
+ return toast_fetch_datum(attr, true);
+ else
+ return detoast_external_attr(attr);
+}
+
+
/* ----------
* detoast_attr -
*
@@ -120,7 +137,7 @@ detoast_attr(varlena *attr)
/*
* This is an externally stored datum --- fetch it back from there
*/
- attr = toast_fetch_datum(attr);
+ attr = toast_fetch_datum(attr, false);
/* If it's compressed, decompress it */
if (VARATT_IS_COMPRESSED(attr))
{
@@ -262,7 +279,7 @@ detoast_attr_slice(varlena *attr,
preslice = toast_fetch_datum_slice(attr, 0, max_size);
}
else
- preslice = toast_fetch_datum(attr);
+ preslice = toast_fetch_datum(attr, false);
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
@@ -337,10 +354,12 @@ detoast_attr_slice(varlena *attr,
*
* Reconstruct an in memory Datum from the chunks saved
* in the toast relation
+ *
+ * If missing_ok is true, returns NULL when chunks are missing.
* ----------
*/
static varlena *
-toast_fetch_datum(varlena *attr)
+toast_fetch_datum(varlena *attr, bool missing_ok)
{
Relation toastrel;
varlena *result;
@@ -372,8 +391,14 @@ toast_fetch_datum(varlena *attr)
toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock);
/* Fetch all chunks */
- table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
- attrsize, 0, attrsize, result);
+ if (!table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
+ attrsize, 0, attrsize, result,
+ missing_ok))
+ {
+ pfree(result);
+ table_close(toastrel, AccessShareLock);
+ return NULL;
+ }
/* Close toast table */
table_close(toastrel, AccessShareLock);
@@ -454,7 +479,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
/* Fetch all chunks */
table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
attrsize, sliceoffset, slicelength,
- result);
+ result, false);
/* Close toast table */
table_close(toastrel, AccessShareLock);
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index abfd8e8970a6..1d4018931200 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -2236,7 +2236,7 @@ heap_prepare_insert(Relation relation, HeapTuple tup, TransactionId xid,
return tup;
}
else if (HeapTupleHasExternal(tup) || tup->t_len > TOAST_TUPLE_THRESHOLD)
- return heap_toast_insert_or_update(relation, tup, NULL, options);
+ return heap_toast_insert_or_update(relation, tup, NULL, options, false);
else
return tup;
}
@@ -3871,7 +3871,7 @@ l2:
if (need_toast)
{
/* Note we always use WAL and FSM during updates */
- heaptup = heap_toast_insert_or_update(relation, newtup, &oldtup, 0);
+ heaptup = heap_toast_insert_or_update(relation, newtup, &oldtup, 0, false);
newtupsize = MAXALIGN(heaptup->t_len);
}
else
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 2268cc277bce..2cd9d0f4aeb9 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -19,6 +19,7 @@
*/
#include "postgres.h"
+#include "access/detoast.h"
#include "access/genam.h"
#include "access/heapam.h"
#include "access/heaptoast.h"
@@ -48,9 +49,10 @@
#include "utils/rel.h"
#include "utils/tuplesort.h"
-static void reform_and_rewrite_tuple(HeapTuple tuple,
+static bool reform_and_rewrite_tuple(HeapTuple tuple,
Relation OldHeap, Relation NewHeap,
- Datum *values, bool *isnull, RewriteState rwstate);
+ Datum *values, bool *isnull, RewriteState rwstate,
+ bool missing_ok);
static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap,
Relation NewHeap, Datum *values, bool *isnull,
BulkInsertState bistate);
@@ -717,6 +719,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
HeapTuple tuple;
Buffer buf;
bool isdead;
+ bool recently_dead = false;
CHECK_FOR_INTERRUPTS();
@@ -802,6 +805,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
break;
case HEAPTUPLE_RECENTLY_DEAD:
*tups_recently_dead += 1;
+ recently_dead = true;
pg_fallthrough;
case HEAPTUPLE_LIVE:
/* Live or recently dead, must copy it */
@@ -835,6 +839,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
RelationGetRelationName(OldHeap));
/* treat as recently dead */
*tups_recently_dead += 1;
+ recently_dead = true;
isdead = false;
break;
default:
@@ -881,8 +886,22 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
int64 ct_val[2];
if (!concurrent)
- reform_and_rewrite_tuple(tuple, OldHeap, NewHeap,
- values, isnull, rwstate);
+ {
+ if (!reform_and_rewrite_tuple(tuple, OldHeap, NewHeap,
+ values, isnull, rwstate,
+ recently_dead))
+ {
+ /*
+ * Missing TOAST chunks for a recently-dead tuple. Treat
+ * it as dead.
+ */
+ *tups_vacuumed += 1;
+ *num_tuples -= 1;
+ if (recently_dead)
+ *tups_recently_dead -= 1;
+ continue;
+ }
+ }
else
heap_insert_for_repack(tuple, OldHeap, NewHeap,
values, isnull, bistate);
@@ -934,10 +953,17 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
n_tuples += 1;
if (!concurrent)
- reform_and_rewrite_tuple(tuple,
- OldHeap, NewHeap,
- values, isnull,
- rwstate);
+ {
+ if (!reform_and_rewrite_tuple(tuple,
+ OldHeap, NewHeap,
+ values, isnull,
+ rwstate, true))
+ {
+ /* TOAST gone for a recently dead tuple */
+ n_tuples -= 1;
+ continue;
+ }
+ }
else
heap_insert_for_repack(tuple, OldHeap, NewHeap,
values, isnull, bistate);
@@ -1613,6 +1639,43 @@ heapam_index_build_range_scan(Relation heapRelation,
continue;
}
+ /*
+ * For RECENTLY_DEAD tuples, pre-detoast external TOAST values. We do
+ * this to detect for missing chunks at later stages, feeding inline
+ * data to the downstream access method code.
+ */
+ if (!tupleIsAlive)
+ {
+ TupleDesc tdesc = RelationGetDescr(heapRelation);
+ bool skip = false;
+
+ slot_getallattrs(slot);
+ for (int i = 0; i < tdesc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(tdesc, i);
+
+ if (att->attlen != -1 || att->attisdropped)
+ continue;
+ if (slot->tts_isnull[i])
+ continue;
+ if (VARATT_IS_EXTERNAL_ONDISK(DatumGetPointer(slot->tts_values[i])))
+ {
+ varlena *detoasted;
+
+ detoasted = detoast_external_attr_extended(
+ (varlena *) DatumGetPointer(slot->tts_values[i]));
+ if (detoasted == NULL)
+ {
+ skip = true;
+ break;
+ }
+ slot->tts_values[i] = PointerGetDatum(detoasted);
+ }
+ }
+ if (skip)
+ continue;
+ }
+
/*
* For the current heap tuple, extract all the attributes we use in
* this index, and note which are null. This also performs evaluation
@@ -2347,20 +2410,29 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate,
* SET WITHOUT OIDS.
*
* So, we must reconstruct the tuple from component Datums.
+ *
+ * Returns true on success, and false on failure if missing_ok is
+ * true.
*/
-static void
+static bool
reform_and_rewrite_tuple(HeapTuple tuple,
Relation OldHeap, Relation NewHeap,
- Datum *values, bool *isnull, RewriteState rwstate)
+ Datum *values, bool *isnull, RewriteState rwstate,
+ bool missing_ok)
{
HeapTuple newtuple;
newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull);
/* The heap rewrite module does the rest */
- rewrite_heap_tuple(rwstate, tuple, newtuple);
+ if (!rewrite_heap_tuple(rwstate, tuple, newtuple, missing_ok))
+ {
+ heap_freetuple(newtuple);
+ return false;
+ }
heap_freetuple(newtuple);
+ return true;
}
/*
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 03f885a25b07..38896a0c7492 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -94,7 +94,7 @@ heap_toast_delete(Relation rel, HeapTuple oldtup, bool is_speculative)
*/
HeapTuple
heap_toast_insert_or_update(Relation rel, HeapTuple newtup, HeapTuple oldtup,
- uint32 options)
+ uint32 options, bool missing_ok)
{
HeapTuple result_tuple;
TupleDesc tupleDesc;
@@ -154,7 +154,8 @@ heap_toast_insert_or_update(Relation rel, HeapTuple newtup, HeapTuple oldtup,
ttc.ttc_oldisnull = toast_oldisnull;
}
ttc.ttc_attr = toast_attr;
- toast_tuple_init(&ttc);
+ if (!toast_tuple_init(&ttc, missing_ok))
+ return NULL;
/* ----------
* Compress and/or save external until data fits into target length
@@ -621,11 +622,14 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
* sliceoffset is the byte offset within the TOAST value from which to fetch.
* slicelength is the number of bytes to be fetched from the TOAST value.
* result is the varlena into which the results should be written.
+ * missing_ok if true, return false on missing chunks instead of error.
+ *
+ * Returns true on success, false if missing_ok and chunks are missing.
*/
-void
+bool
heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
int32 sliceoffset, int32 slicelength,
- varlena *result)
+ varlena *result, bool missing_ok)
{
Relation *toastidxs;
ScanKeyData toastkey[3];
@@ -779,13 +783,23 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
* Final checks that we successfully fetched the datum
*/
if (expectedchunk != (endchunk + 1))
+ {
+ if (missing_ok)
+ {
+ systable_endscan_ordered(toastscan);
+ toast_close_indexes(toastidxs, num_indexes, AccessShareLock);
+ return false;
+ }
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
errmsg_internal("missing chunk number %d for toast value %u in %s",
expectedchunk, valueid,
RelationGetRelationName(toastrel))));
+ }
/* End scan and close indexes. */
systable_endscan_ordered(toastscan);
toast_close_indexes(toastidxs, num_indexes, AccessShareLock);
+
+ return true;
}
diff --git a/src/backend/access/heap/rewriteheap.c b/src/backend/access/heap/rewriteheap.c
index 5a5398a76ae7..fbb6e6ca1f94 100644
--- a/src/backend/access/heap/rewriteheap.c
+++ b/src/backend/access/heap/rewriteheap.c
@@ -211,7 +211,8 @@ typedef struct RewriteMappingDataEntry
/* prototypes for internal functions */
-static void raw_heap_insert(RewriteState state, HeapTuple tup);
+static bool raw_heap_insert(RewriteState state, HeapTuple tup,
+ bool missing_ok);
/* internal logical remapping prototypes */
static void logical_begin_heap_rewrite(RewriteState state);
@@ -309,7 +310,7 @@ end_heap_rewrite(RewriteState state)
while ((unresolved = hash_seq_search(&seq_status)) != NULL)
{
ItemPointerSetInvalid(&unresolved->tuple->t_data->t_ctid);
- raw_heap_insert(state, unresolved->tuple);
+ raw_heap_insert(state, unresolved->tuple, false);
}
/* Write the last page, if any */
@@ -338,9 +339,10 @@ end_heap_rewrite(RewriteState state)
* old_tuple original tuple in the old heap
* new_tuple new, rewritten tuple to be inserted to new heap
*/
-void
+bool
rewrite_heap_tuple(RewriteState state,
- HeapTuple old_tuple, HeapTuple new_tuple)
+ HeapTuple old_tuple, HeapTuple new_tuple,
+ bool missing_ok)
{
MemoryContext old_cxt;
ItemPointerData old_tid;
@@ -437,7 +439,7 @@ rewrite_heap_tuple(RewriteState state,
* tuple will be written.
*/
MemoryContextSwitchTo(old_cxt);
- return;
+ return true;
}
}
@@ -455,7 +457,13 @@ rewrite_heap_tuple(RewriteState state,
ItemPointerData new_tid;
/* Insert the tuple and find out where it's put in new_heap */
- raw_heap_insert(state, new_tuple);
+ if (!raw_heap_insert(state, new_tuple, missing_ok))
+ {
+ if (free_new)
+ heap_freetuple(new_tuple);
+ MemoryContextSwitchTo(old_cxt);
+ return false;
+ }
new_tid = new_tuple->t_self;
logical_rewrite_heap_tuple(state, old_tid, new_tuple);
@@ -532,6 +540,7 @@ rewrite_heap_tuple(RewriteState state,
}
MemoryContextSwitchTo(old_cxt);
+ return true;
}
/*
@@ -593,8 +602,8 @@ rewrite_heap_dead_tuple(RewriteState state, HeapTuple old_tuple)
* tuple is invalid on entry, it's replaced with the new TID as well (in
* the inserted data only, not in the caller's copy).
*/
-static void
-raw_heap_insert(RewriteState state, HeapTuple tup)
+static bool
+raw_heap_insert(RewriteState state, HeapTuple tup, bool missing_ok)
{
Page page;
Size pageFreeSpace,
@@ -628,7 +637,9 @@ raw_heap_insert(RewriteState state, HeapTuple tup)
options |= HEAP_INSERT_NO_LOGICAL;
heaptup = heap_toast_insert_or_update(state->rs_new_rel, tup, NULL,
- options);
+ options, missing_ok);
+ if (heaptup == NULL)
+ return false;
}
else
heaptup = tup;
@@ -702,6 +713,8 @@ raw_heap_insert(RewriteState state, HeapTuple tup)
/* If heaptup is a private copy, release it. */
if (heaptup != tup)
heap_freetuple(heaptup);
+
+ return true;
}
/* ------------------------------------------------------------------------
diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c
index 2f2022d99510..7e98c01a5eee 100644
--- a/src/backend/access/table/toast_helper.c
+++ b/src/backend/access/table/toast_helper.c
@@ -36,9 +36,12 @@
* toast_flags is just a single uint8, but toast_attr is a caller-provided
* array with a length equal to tupleDesc->natts. The caller need not
* perform any initialization of the array before calling this function.
+ *
+ * If missing_ok is true and an external TOAST value cannot be fetched,
+ * returns false. Returns true on success.
*/
-void
-toast_tuple_init(ToastTupleContext *ttc)
+bool
+toast_tuple_init(ToastTupleContext *ttc, bool missing_ok)
{
TupleDesc tupleDesc = ttc->ttc_rel->rd_att;
int numAttrs = tupleDesc->natts;
@@ -137,7 +140,13 @@ toast_tuple_init(ToastTupleContext *ttc)
if (VARATT_IS_EXTERNAL(new_value))
{
ttc->ttc_attr[i].tai_oldexternal = new_value;
- if (att->attstorage == TYPSTORAGE_PLAIN)
+ if (missing_ok && VARATT_IS_EXTERNAL_ONDISK(new_value))
+ {
+ new_value = detoast_external_attr_extended(new_value);
+ if (new_value == NULL)
+ return false;
+ }
+ else if (att->attstorage == TYPSTORAGE_PLAIN)
new_value = detoast_attr(new_value);
else
new_value = detoast_external_attr(new_value);
@@ -159,6 +168,8 @@ toast_tuple_init(ToastTupleContext *ttc)
ttc->ttc_attr[i].tai_colflags |= TOASTCOL_IGNORE;
}
}
+
+ return true;
}
/*
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index ad0d85f41897..7e583ee39a10 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -63,6 +63,7 @@ tests += {
't/052_checkpoint_segment_missing.pl',
't/053_standby_login_event_trigger.pl',
't/054_unlogged_sequence_promotion.pl',
+ 't/055_rewrite_stale_xmin.pl',
],
},
}
diff --git a/src/test/recovery/t/055_rewrite_stale_xmin.pl b/src/test/recovery/t/055_rewrite_stale_xmin.pl
new file mode 100644
index 000000000000..dad95c0cf967
--- /dev/null
+++ b/src/test/recovery/t/055_rewrite_stale_xmin.pl
@@ -0,0 +1,124 @@
+# Copyright (c) 2025-2026, PostgreSQL Global Development Group
+#
+# Test that heap rewrites (CLUSTER, VACUUM FULL, REPACK) and index builds
+# work in the case when the backend's own xmin is held back by a transaction
+# in another database.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+
+my $node = PostgreSQL::Test::Cluster->new('primary');
+$node->init;
+# No Autovacuum, no random interferences.
+$node->append_conf(
+ 'postgresql.conf', qq[
+autovacuum = off
+]);
+$node->start;
+
+# Create two databases.
+$node->safe_psql('postgres', 'CREATE DATABASE db_holder');
+$node->safe_psql('postgres', 'CREATE DATABASE db_target');
+
+# Set up a table with TOAST data. Returns the toast table name.
+sub setup_table
+{
+ my ($tblname, $extra_ddl) = @_;
+ $extra_ddl //= '';
+
+ $node->safe_psql(
+ 'db_target', qq{
+ DROP TABLE IF EXISTS $tblname;
+ CREATE TABLE $tblname (id int PRIMARY KEY, data text)
+ WITH (autovacuum_enabled = off);
+ ALTER TABLE $tblname ALTER COLUMN data SET STORAGE EXTERNAL;
+ INSERT INTO $tblname VALUES (1, repeat('x', 2500));
+ INSERT INTO $tblname VALUES (2, repeat('z', 2500));
+ $extra_ddl
+ });
+
+ my $toast = $node->safe_psql(
+ 'db_target', qq{
+ SELECT 'pg_toast.' || relname FROM pg_class
+ WHERE oid = (SELECT reltoastrelid FROM pg_class
+ WHERE relname = '$tblname');
+ });
+ chomp $toast;
+ return $toast;
+}
+
+# Create precondition state. Requires that $holder is already running with
+# a write XID.
+sub create_missing_toast_state
+{
+ my ($tblname, $toast_table) = @_;
+
+ # Hold xmin in same DB to prevent main tuple removal.
+ my $xmin_holder = $node->background_psql('db_target', on_error_stop => 1);
+ $xmin_holder->query_safe("BEGIN ISOLATION LEVEL REPEATABLE READ");
+ $xmin_holder->query_safe("SELECT 1");
+
+ # Delete the row with TOAST data.
+ $node->safe_psql('db_target', "DELETE FROM $tblname WHERE id = 1");
+
+ # VACUUM main table only. The xmin_holder keeps the dead tuple.
+ $node->safe_psql('db_target', "VACUUM (PROCESS_TOAST false) $tblname");
+
+ # Release same-DB xmin.
+ $xmin_holder->query_safe("COMMIT");
+ $xmin_holder->quit;
+
+ # Vacuum toast table: toast chunks now DEAD and removed.
+ $node->safe_psql('db_target', "VACUUM $toast_table");
+}
+
+# Open a transaction holding an XID before any deletes. This XID will hold
+# back GetSnapshotData() globally, on a different database than the one
+# running the various rewrite operations.
+my $holder = $node->background_psql('db_holder', on_error_stop => 1);
+$holder->query_safe("BEGIN");
+$holder->query_safe("CREATE TEMP TABLE xid_holder(x int)");
+
+# Test 1: CLUSTER
+my $toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+my ($ret, $stdout, $stderr) =
+ $node->psql('db_target', 'CLUSTER rewrite_test USING rewrite_test_pkey');
+is($ret, 0, "CLUSTER succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "CLUSTER: no missing-chunk error");
+
+# Test 2: VACUUM FULL
+$toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+($ret, $stdout, $stderr) =
+ $node->psql('db_target', 'VACUUM FULL rewrite_test');
+is($ret, 0, "VACUUM FULL succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "VACUUM FULL: no missing-chunk error");
+
+# Test 3: REPACK
+$toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+($ret, $stdout, $stderr) = $node->psql('db_target', 'REPACK rewrite_test');
+is($ret, 0, "REPACK succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "REPACK: no missing-chunk error");
+
+# Test 4: CREATE INDEX on toasted column
+$toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+($ret, $stdout, $stderr) = $node->psql('db_target',
+ 'CREATE INDEX rewrite_test_data_idx ON rewrite_test (data)');
+is($ret, 0, "CREATE INDEX succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "CREATE INDEX: no missing-chunk error");
+
+# Cleanup.
+$holder->query_safe("COMMIT");
+$holder->quit;
+
+$node->stop;
+done_testing();
diff --git a/contrib/dblink/.gitignore b/contrib/dblink/.gitignore
index 5dcb3ff97235..33bc8d8ffeb3 100644
--- a/contrib/dblink/.gitignore
+++ b/contrib/dblink/.gitignore
@@ -2,3 +2,5 @@
/log/
/results/
/tmp_check/
+/output_iso/
+/tmp_check_iso/
diff --git a/contrib/dblink/Makefile b/contrib/dblink/Makefile
index fde0b49ddbbd..a0ba41280813 100644
--- a/contrib/dblink/Makefile
+++ b/contrib/dblink/Makefile
@@ -13,6 +13,8 @@ PGFILEDESC = "dblink - connect to other PostgreSQL databases"
REGRESS = dblink
REGRESS_OPTS = --dlpath=$(top_builddir)/src/test/regress
+ISOLATION = rewrite_stale_xmin
+ISOLATION_OPTS = --load-extension=dblink
TAP_TESTS = 1
ifdef USE_PGXS
diff --git a/contrib/dblink/expected/rewrite_stale_xmin.out b/contrib/dblink/expected/rewrite_stale_xmin.out
new file mode 100644
index 000000000000..2e5a7940ea26
--- /dev/null
+++ b/contrib/dblink/expected/rewrite_stale_xmin.out
@@ -0,0 +1,237 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_cluster s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_cluster:
+ CLUSTER rewrite_test USING rewrite_test_pkey;
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_vacuum_full s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_vacuum_full:
+ VACUUM FULL rewrite_test;
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_repack s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_repack:
+ REPACK rewrite_test;
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_create_index s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_create_index:
+ CREATE INDEX rewrite_test_data_idx ON rewrite_test (data);
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
diff --git a/contrib/dblink/meson.build b/contrib/dblink/meson.build
index e2489f41229f..86fbd372b880 100644
--- a/contrib/dblink/meson.build
+++ b/contrib/dblink/meson.build
@@ -36,6 +36,11 @@ tests += {
],
'regress_args': ['--dlpath', meson.project_build_root() / 'src/test/regress'],
},
+ 'isolation': {
+ 'specs': [
+ 'rewrite_stale_xmin',
+ ],
+ },
'tap': {
'tests': [
't/001_auth_scram.pl',
diff --git a/contrib/dblink/specs/rewrite_stale_xmin.spec b/contrib/dblink/specs/rewrite_stale_xmin.spec
new file mode 100644
index 000000000000..fe88e789f854
--- /dev/null
+++ b/contrib/dblink/specs/rewrite_stale_xmin.spec
@@ -0,0 +1,137 @@
+# Test that heap rewrites work when the rewriting backend's own xmin is
+# held back by a transaction in another database.
+
+setup
+{
+ CREATE DATABASE regress_other_db;
+}
+
+setup
+{
+ CREATE EXTENSION IF NOT EXISTS dblink;
+
+ CREATE TABLE rewrite_test (id int PRIMARY KEY, data text)
+ WITH (autovacuum_enabled = off);
+ ALTER TABLE rewrite_test ALTER COLUMN data SET STORAGE EXTERNAL;
+ INSERT INTO rewrite_test VALUES (1, repeat('x', 2500));
+ INSERT INTO rewrite_test VALUES (2, repeat('y', 2500));
+
+ -- Rename the toast table to a known name so we can VACUUM it directly.
+ SET allow_system_table_mods TO true;
+ DO $$DECLARE r record;
+ BEGIN
+ SELECT INTO r reltoastrelid::regclass::text AS table_name
+ FROM pg_class WHERE oid = 'rewrite_test'::regclass;
+ EXECUTE 'ALTER TABLE ' || r.table_name || ' RENAME TO rewrite_test_toast;';
+ END$$;
+ RESET allow_system_table_mods;
+}
+
+teardown
+{
+ DROP DATABASE regress_other_db;
+}
+
+# Session s1: use dblink to hold a transaction in another database.
+# This must start before the DELETE so its XID is older, ensuring
+# GetSnapshotData() holds back any rewrites.
+session s1
+step s1_hold_xmin
+{
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+}
+step s1_release
+{
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+}
+
+# Session s2: holds xmin in the SAME database temporarily, so the
+# main-table vacuum preserves the dead tuple as RECENTLY_DEAD.
+session s2
+step s2_begin
+{
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+}
+step s2_commit { COMMIT; }
+
+# Session s3: performs DML, split vacuum, and the rewrite.
+session s3
+step s3_delete
+{
+ DELETE FROM rewrite_test WHERE id = 1;
+}
+step s3_vacuum_main_only
+{
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+}
+step s3_vacuum_toast
+{
+ VACUUM pg_toast.rewrite_test_toast;
+}
+step s3_cluster
+{
+ CLUSTER rewrite_test USING rewrite_test_pkey;
+}
+step s3_vacuum_full
+{
+ VACUUM FULL rewrite_test;
+}
+step s3_repack
+{
+ REPACK rewrite_test;
+}
+step s3_create_index
+{
+ CREATE INDEX rewrite_test_data_idx ON rewrite_test (data);
+}
+
+teardown { DROP TABLE IF EXISTS rewrite_test; }
+
+# CLUSTER
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_cluster
+ s1_release
+
+# VACUUM FULL
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_vacuum_full
+ s1_release
+
+# REPACK
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_repack
+ s1_release
+
+# CREATE INDEX
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_create_index
+ s1_release
--
2.55.0
[text/x-patch] v3-0002-Smuggle-the-missing_ok-arg-in-a-global-var-for-ba.patch (5.7K, ../../[email protected]/3-v3-0002-Smuggle-the-missing_ok-arg-in-a-global-var-for-ba.patch)
download | inline diff:
From ae8419d0a166372789cd94acc0b9c8a8d9c864cf Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Wed, 8 Jul 2026 12:36:48 +0300
Subject: [PATCH v3 2/2] Smuggle the 'missing_ok' arg in a global var, for
backpatching
---
src/backend/access/heap/heaptoast.c | 21 ++++++++++++++++---
src/backend/access/table/tableam.c | 4 ++++
src/include/access/heaptoast.h | 15 +++++++++++---
src/include/access/tableam.h | 31 ++++++++++++++++++++---------
4 files changed, 56 insertions(+), 15 deletions(-)
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 38896a0c749..7ab54225059 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -613,6 +613,21 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
return new_tuple;
}
+/*
+ * Adapter for the old API in backbranches, where the 'missing_ok' flag is
+ * passed in a global variable.
+ */
+void
+heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
+ int32 sliceoffset, int32 slicelength,
+ varlena *result)
+{
+ fetch_toast_slice_was_missing =
+ heap_fetch_toast_slice_ext(toastrel, valueid, attrsize,
+ sliceoffset, slicelength,
+ result, fetch_toast_slice_missing_ok);
+}
+
/*
* Fetch a TOAST slice from a heap table.
*
@@ -627,9 +642,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
* Returns true on success, false if missing_ok and chunks are missing.
*/
bool
-heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
- int32 sliceoffset, int32 slicelength,
- varlena *result, bool missing_ok)
+heap_fetch_toast_slice_ext(Relation toastrel, Oid valueid, int32 attrsize,
+ int32 sliceoffset, int32 slicelength,
+ varlena *result, bool missing_ok)
{
Relation *toastidxs;
ScanKeyData toastkey[3];
diff --git a/src/backend/access/table/tableam.c b/src/backend/access/table/tableam.c
index 68ff0966f1c..f3164c3d627 100644
--- a/src/backend/access/table/tableam.c
+++ b/src/backend/access/table/tableam.c
@@ -49,6 +49,10 @@
char *default_table_access_method = DEFAULT_TABLE_ACCESS_METHOD;
bool synchronize_seqscans = true;
+/* hack to pass extra argument and return value to/from relation_fetch_toast_slice */
+bool fetch_toast_slice_missing_ok = false;
+bool fetch_toast_slice_was_missing = false;
+
/* ----------------------------------------------------------------------------
* Slot functions.
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index 1be85db6ddd..4c763447d2b 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -148,9 +148,18 @@ extern HeapTuple toast_build_flattened_tuple(TupleDesc tupleDesc,
* instead of raising an error. Returns true on success.
* ----------
*/
-extern bool heap_fetch_toast_slice(Relation toastrel, Oid valueid,
+extern bool heap_fetch_toast_slice_ext(Relation toastrel, Oid valueid,
+ int32 attrsize, int32 sliceoffset,
+ int32 slicelength, varlena *result,
+ bool missing_ok);
+
+/*
+ * Legacy API used in the tableam. Instead of a 'missing_ok' arg, looks
+ * at the 'fetch_toast_slice_missing_ok' variable, and returns the return
+ * value 'fetch_toast_slice_was_missing'.
+ */
+extern void heap_fetch_toast_slice(Relation toastrel, Oid valueid,
int32 attrsize, int32 sliceoffset,
- int32 slicelength, varlena *result,
- bool missing_ok);
+ int32 slicelength, varlena *result);
#endif /* HEAPTOAST_H */
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index a90b15dd82b..979b3f63fb2 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -307,6 +307,9 @@ typedef void (*IndexBuildCallback) (Relation index,
bool tupleIsAlive,
void *state);
+extern bool fetch_toast_slice_missing_ok;
+extern bool fetch_toast_slice_was_missing;
+
/*
* API struct for a table AM. Note this must be allocated in a
* server-lifetime manner, typically as a static const struct, which then gets
@@ -783,15 +786,15 @@ typedef struct TableAmRoutine
* table implemented by this AM. See table_relation_fetch_toast_slice()
* for more details.
*
- * If missing_ok is true, returns false when chunks are missing instead of
- * raising an error. Returns true on success.
+ * If the 'fetch_toast_slice_missing_ok' global variable is true and some
+ * chunks are missing, the callback should set the
+ * 'fetch_toast_slice_was_missing' flag instead of raising an error
*/
- bool (*relation_fetch_toast_slice) (Relation toastrel, Oid valueid,
+ void (*relation_fetch_toast_slice) (Relation toastrel, Oid valueid,
int32 attrsize,
int32 sliceoffset,
int32 slicelength,
- varlena *result,
- bool missing_ok);
+ varlena *result);
/* ------------------------------------------------------------------------
@@ -1995,10 +1998,20 @@ table_relation_fetch_toast_slice(Relation toastrel, Oid valueid,
int32 slicelength, varlena *result,
bool missing_ok)
{
- return toastrel->rd_tableam->relation_fetch_toast_slice(toastrel, valueid,
- attrsize,
- sliceoffset, slicelength,
- result, missing_ok);
+ fetch_toast_slice_missing_ok = missing_ok;
+ fetch_toast_slice_was_missing = false;
+ toastrel->rd_tableam->relation_fetch_toast_slice(toastrel, valueid,
+ attrsize,
+ sliceoffset, slicelength,
+ result);
+ if (missing_ok)
+ {
+ /* reset the flag just in case there are other direct callers */
+ fetch_toast_slice_missing_ok = false;
+ return fetch_toast_slice_was_missing;
+ }
+ else
+ return true;
}
--
2.47.3
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: BUG #19519: REPACK can fail due to missing chunk for toast value
2026-06-14 07:00 BUG #19519: REPACK can fail due to missing chunk for toast value PG Bug reporting form <[email protected]>
2026-06-17 17:03 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-03 06:45 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Imran Zaheer <[email protected]>
2026-07-06 06:36 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 18:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
2026-07-06 23:18 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 23:51 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
2026-07-07 01:42 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-07 05:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-07 17:10 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-07 22:15 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-08 01:10 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-08 12:00 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Heikki Linnakangas <[email protected]>
@ 2026-07-08 15:00 ` Imran Zaheer <[email protected]>
2026-07-08 22:49 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2 siblings, 1 reply; 21+ messages in thread
From: Imran Zaheer @ 2026-07-08 15:00 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: Michael Paquier <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Alexander Lakhin <[email protected]>; PostgreSQL mailing lists <[email protected]>; Konstantin Knizhnik <[email protected]>
Hi.
@@ -934,10 +953,17 @@ heapam_relation_copy_for_cluster(Relation
OldHeap, Relation NewHeap,
- reform_and_rewrite_tuple(tuple,
- OldHeap, NewHeap,
- values, isnull,
- rwstate);
+ {
+ if (!reform_and_rewrite_tuple(tuple,
+ OldHeap, NewHeap,
+ values, isnull,
+ rwstate, true))
+ {
+ /* TOAST gone for a recently dead tuple */
+ n_tuples -= 1;
+ continue;
+ }
+ }
The tuplesort_getheaptuple() will contain both live and recently
deleted tuples. However, the proposed fix passes missing_ok=true for
all tuples coming out of the sort, which means a toast fetch failure
for a LIVE tuple would be acceptable and will be silently skipped. Or
am I missing something?
Thanks
Imran Zaheer
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: BUG #19519: REPACK can fail due to missing chunk for toast value
2026-06-14 07:00 BUG #19519: REPACK can fail due to missing chunk for toast value PG Bug reporting form <[email protected]>
2026-06-17 17:03 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-03 06:45 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Imran Zaheer <[email protected]>
2026-07-06 06:36 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 18:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
2026-07-06 23:18 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 23:51 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
2026-07-07 01:42 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-07 05:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-07 17:10 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-07 22:15 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-08 01:10 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-08 12:00 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Heikki Linnakangas <[email protected]>
2026-07-08 15:00 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Imran Zaheer <[email protected]>
@ 2026-07-08 22:49 ` Michael Paquier <[email protected]>
0 siblings, 0 replies; 21+ messages in thread
From: Michael Paquier @ 2026-07-08 22:49 UTC (permalink / raw)
To: Imran Zaheer <[email protected]>; +Cc: Heikki Linnakangas <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Alexander Lakhin <[email protected]>; PostgreSQL mailing lists <[email protected]>; Konstantin Knizhnik <[email protected]>
On Wed, Jul 08, 2026 at 08:00:28PM +0500, Imran Zaheer wrote:
> The tuplesort_getheaptuple() will contain both live and recently
> deleted tuples. However, the proposed fix passes missing_ok=true for
> all tuples coming out of the sort, which means a toast fetch failure
> for a LIVE tuple would be acceptable and will be silently skipped. Or
> am I missing something?
Argh, thanks. Using missing_ok=true for the sort path is broken. The
flag should be false, instead. When dealing with the sort of the
tuples, we would already have made sure that the tuples with missing
toast chunks have been discarded, so a plain error with a missing
chunk would point to an invalid case.
I'm taking some time today to rework the patch set. Will add this
adjustment in it.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: BUG #19519: REPACK can fail due to missing chunk for toast value
2026-06-14 07:00 BUG #19519: REPACK can fail due to missing chunk for toast value PG Bug reporting form <[email protected]>
2026-06-17 17:03 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-03 06:45 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Imran Zaheer <[email protected]>
2026-07-06 06:36 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 18:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
2026-07-06 23:18 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 23:51 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
2026-07-07 01:42 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-07 05:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-07 17:10 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-07 22:15 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-08 01:10 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-08 12:00 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Heikki Linnakangas <[email protected]>
@ 2026-07-09 02:47 ` Michael Paquier <[email protected]>
2 siblings, 0 replies; 21+ messages in thread
From: Michael Paquier @ 2026-07-09 02:47 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Imran Zaheer <[email protected]>; Alexander Lakhin <[email protected]>; PostgreSQL mailing lists <[email protected]>; Konstantin Knizhnik <[email protected]>
On Wed, Jul 08, 2026 at 03:00:33PM +0300, Heikki Linnakangas wrote:
> On 08/07/2026 04:10, Michael Paquier wrote:
> Yeah, boolean arguments in general can be confusing. Especially arguments
> like "missing_ok" where it's hard to remember which behavior is true and
> which is false. An IDE that shows the name of the argument helps, but having
> the caller say "MISSING_TOAST_OK" rather than "true" is much clearer.
>
> For backpatching, I think we could smuggle the flag in a global variable, to
> avoid changing the signature of the relation_fetch_toast_slice() callback.
> See attached patch 0002 on top of your 0001 patch (which is also attached
> for completeness). It doesn't fix the problem for hypothetical 3rd party
> tableam implementations that implement their own
> relation_fetch_toast_slice() callback. But do such extensions even exist? If
> yes, they could be fixed too by also checking the global variable.
One of my issues with that is that the static states can go out of
sync very easily. I'd like to think we don't have that many
extensions, but it's really hard to be sure. Silent breakages across
minor releases are never cool.
> I thought about adding an extra "extended" callback next to
> relation_fetch_toast_slice(), but I don't see a way to add functions to
> TableAmRoutine in an ABI-compatible way. For the future, we might want to
> store sizeof(TableAmRoutine) in the struct itself, so that we could add
> fields to it in minor versions without breaking the API, in case we need
> something like this again.
Right, I didn't consider this one.
>> - One for the rewrite path. It makes little sense to do any kind of
>> aggressive early detoasting because it may be wasteful due to the
>> tuple rewrites that update their data with only copies by reference
>> (main relation tuple is rewritten, reuses the same external TOAST
>> tuple). For workloads where UPDATEs do not touch the TOASTed
>> attributes, that would be a waste.
>
> I didn't understand this part. Do you mean when rebuilding the indexes after
> rewriting the heap? I didn't think we support storing toasted external
> datums in an index at all.
I was referring to toast_save_datum() (touched this area a few weeks
ago for the 8-byte TOAST thing), where we touch only the
varatt_external on the main relation in some rewrite case. Forcing a
detoast for all tuples in such cases would make the rewrite less
efficient, no?
>> - One for the index build path, which is actually too aggressive with
>> its early detoasting, now that I think about it. There should be no
>> need to perform a detoast for anything else than the attributes that
>> are used in the index definition or the attributes that are used in
>> index expressions. So as written this patch would lead to a
>> regression.
>
> Yeah, although I wouldn't be too worried about the performance here. The
> penalty would be when building an index on values that are large enough to
> be toasted, with lots of RECENTLY_DEAD tuples. That doesn't seem like a very
> common case.
Pulling the varnos from the index predicate and expressions makes that
possible, thanks to ii_NumIndexAttrs.
An updated patch is attached, with the "flags" business added, and
something for the other issue reported by Imran for the case where
reform_and_rewrite_tuple() was not handled correctly. If we have a
sort, that means going back to a more aggressive detoasting to check
if some chunks are missing if a tuple has been found as recently dead.
I was wondering about doing a split into two patches for
build_range_scan() and copy_for_cluster(), but refrained from that due
to the detoast_external_attr_extended() required in both cases.
Anyway, thoughts and reviews are welcome.
--
Michael
From 8d961035cdccfe9e76a859c5acd838e8f6969382 Mon Sep 17 00:00:00 2001
From: Michael Paquier <[email protected]>
Date: Thu, 9 Jul 2026 11:46:21 +0900
Subject: [PATCH v4] Fix "missing chunk" errors during heap rewrites and index
builds
Blah, blah.
This problem has been reported recently by Alexander for REPACK, but it
is a much older issue (tested down to 9.5 for curiosity's sake, but I
suspect that it is much older than that).
XXX: This patch includes both TAP and isolation tests. We should remove
one pattern at the end.
Reported-by: Alexander Lakhin <[email protected]>
Discussion: https://postgr.es/m/[email protected]
Discussion: https://postgr.es/m/[email protected]
---
src/include/access/detoast.h | 9 +
src/include/access/heaptoast.h | 18 +-
src/include/access/rewriteheap.h | 6 +-
src/include/access/tableam.h | 26 +-
src/include/access/toast_helper.h | 10 +-
src/backend/access/common/detoast.c | 42 +++-
src/backend/access/heap/heapam.c | 4 +-
src/backend/access/heap/heapam_handler.c | 148 ++++++++++-
src/backend/access/heap/heaptoast.c | 28 ++-
src/backend/access/heap/rewriteheap.c | 31 ++-
src/backend/access/table/toast_helper.c | 19 +-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/055_rewrite_stale_xmin.pl | 124 +++++++++
contrib/dblink/.gitignore | 2 +
contrib/dblink/Makefile | 2 +
.../dblink/expected/rewrite_stale_xmin.out | 237 ++++++++++++++++++
contrib/dblink/meson.build | 5 +
contrib/dblink/specs/rewrite_stale_xmin.spec | 137 ++++++++++
18 files changed, 794 insertions(+), 55 deletions(-)
create mode 100644 src/test/recovery/t/055_rewrite_stale_xmin.pl
create mode 100644 contrib/dblink/expected/rewrite_stale_xmin.out
create mode 100644 contrib/dblink/specs/rewrite_stale_xmin.spec
diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h
index fbd98181a3a1..c5eeed04b011 100644
--- a/src/include/access/detoast.h
+++ b/src/include/access/detoast.h
@@ -43,6 +43,15 @@ do { \
*/
extern varlena *detoast_external_attr(varlena *attr);
+/* ----------
+ * detoast_external_attr_extended() -
+ *
+ * Like detoast_external_attr, but returns NULL if toast chunks
+ * are missing (instead of raising an error).
+ * ----------
+ */
+extern varlena *detoast_external_attr_extended(varlena *attr);
+
/* ----------
* detoast_attr() -
*
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index 631cb1836b96..bc4087b3c95c 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -91,11 +91,15 @@
/* ----------
* heap_toast_insert_or_update -
*
- * Called by heap_insert() and heap_update().
+ * Called by heap_insert() and heap_update(). Returns NULL if the
+ * insert/update could not be completed.
+ *
+ * See toast_helper.h for the values of "flags".
* ----------
*/
extern HeapTuple heap_toast_insert_or_update(Relation rel, HeapTuple newtup,
- HeapTuple oldtup, uint32 options);
+ HeapTuple oldtup, uint32 options,
+ uint32 flags);
/* ----------
* heap_toast_delete -
@@ -139,11 +143,15 @@ extern HeapTuple toast_build_flattened_tuple(TupleDesc tupleDesc,
/* ----------
* heap_fetch_toast_slice
*
- * Fetch a slice from a toast value stored in a heap table.
+ * Fetch a slice from a toast value stored in a heap table. Returns
+ * false if the slice could not be fetched, otherwise true on success.
+ *
+ * See toast_helper.h for the values of "flags".
* ----------
*/
-extern void heap_fetch_toast_slice(Relation toastrel, Oid valueid,
+extern bool heap_fetch_toast_slice(Relation toastrel, Oid valueid,
int32 attrsize, int32 sliceoffset,
- int32 slicelength, varlena *result);
+ int32 slicelength, varlena *result,
+ uint32 flags);
#endif /* HEAPTOAST_H */
diff --git a/src/include/access/rewriteheap.h b/src/include/access/rewriteheap.h
index 6ccf7b45c04e..09110f6337a5 100644
--- a/src/include/access/rewriteheap.h
+++ b/src/include/access/rewriteheap.h
@@ -25,8 +25,10 @@ extern RewriteState begin_heap_rewrite(Relation old_heap, Relation new_heap,
TransactionId oldest_xmin, TransactionId freeze_xid,
MultiXactId cutoff_multi);
extern void end_heap_rewrite(RewriteState state);
-extern void rewrite_heap_tuple(RewriteState state, HeapTuple old_tuple,
- HeapTuple new_tuple);
+
+/* See toast_helper.h for the values of "flags" */
+extern bool rewrite_heap_tuple(RewriteState state, HeapTuple old_tuple,
+ HeapTuple new_tuple, uint32 flags);
extern bool rewrite_heap_dead_tuple(RewriteState state, HeapTuple old_tuple);
/*
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index f2c36696bcad..707baf804e3a 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -782,12 +782,17 @@ typedef struct TableAmRoutine
* This callback is invoked when detoasting a value stored in a toast
* table implemented by this AM. See table_relation_fetch_toast_slice()
* for more details.
+ *
+ * If TOAST_MISSING_OK is set in flags, returns false when chunks are
+ * missing instead of raising an error. Returns true on success. See
+ * toast_helper.h for possible values of "flags".
*/
- void (*relation_fetch_toast_slice) (Relation toastrel, Oid valueid,
+ bool (*relation_fetch_toast_slice) (Relation toastrel, Oid valueid,
int32 attrsize,
int32 sliceoffset,
int32 slicelength,
- varlena *result);
+ varlena *result,
+ uint32 flags);
/* ------------------------------------------------------------------------
@@ -1981,16 +1986,21 @@ table_relation_toast_am(Relation rel)
*
* result is caller-allocated space into which the fetched bytes should be
* stored.
+ *
+ * flags: if TOAST_MISSING_OK is set, return false when toast chunks are
+ * missing instead of raising an error. Returns true on success. See
+ * also toast_helper.h.
*/
-static inline void
+static inline bool
table_relation_fetch_toast_slice(Relation toastrel, Oid valueid,
int32 attrsize, int32 sliceoffset,
- int32 slicelength, varlena *result)
+ int32 slicelength, varlena *result,
+ uint32 flags)
{
- toastrel->rd_tableam->relation_fetch_toast_slice(toastrel, valueid,
- attrsize,
- sliceoffset, slicelength,
- result);
+ return toastrel->rd_tableam->relation_fetch_toast_slice(toastrel, valueid,
+ attrsize,
+ sliceoffset, slicelength,
+ result, flags);
}
diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h
index 2ec92397f267..4abebe82798f 100644
--- a/src/include/access/toast_helper.h
+++ b/src/include/access/toast_helper.h
@@ -101,7 +101,15 @@ typedef struct
#define TOASTCOL_IGNORE 0x0010
#define TOASTCOL_INCOMPRESSIBLE 0x0020
-extern void toast_tuple_init(ToastTupleContext *ttc);
+/*
+ * Flags for TOAST fetch operations.
+ *
+ * TOAST_MISSING_OK, when set, means that a TOAST fetch tolerates missing
+ * chunks and returns a failure state instead of raising an error.
+ */
+#define TOAST_MISSING_OK (1 << 1)
+
+extern bool toast_tuple_init(ToastTupleContext *ttc, uint32 flags);
extern int toast_tuple_find_biggest_attribute(ToastTupleContext *ttc,
bool for_compression,
bool check_main);
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index a6c1f3a734b2..3195d77db4b0 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -16,13 +16,14 @@
#include "access/detoast.h"
#include "access/table.h"
#include "access/tableam.h"
+#include "access/toast_helper.h"
#include "access/toast_internals.h"
#include "common/int.h"
#include "common/pg_lzcompress.h"
#include "utils/expandeddatum.h"
#include "utils/rel.h"
-static varlena *toast_fetch_datum(varlena *attr);
+static varlena *toast_fetch_datum(varlena *attr, uint32 flags);
static varlena *toast_fetch_datum_slice(varlena *attr,
int32 sliceoffset,
int32 slicelength);
@@ -51,7 +52,7 @@ detoast_external_attr(varlena *attr)
/*
* This is an external stored plain value
*/
- result = toast_fetch_datum(attr);
+ result = toast_fetch_datum(attr, 0);
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
@@ -102,6 +103,23 @@ detoast_external_attr(varlena *attr)
}
+/* ----------
+ * detoast_external_attr_extended -
+ *
+ * Like detoast_external_attr, but returns NULL if toast chunks are
+ * missing for external pointers.
+ * ----------
+ */
+varlena *
+detoast_external_attr_extended(varlena *attr)
+{
+ if (VARATT_IS_EXTERNAL_ONDISK(attr))
+ return toast_fetch_datum(attr, TOAST_MISSING_OK);
+ else
+ return detoast_external_attr(attr);
+}
+
+
/* ----------
* detoast_attr -
*
@@ -120,7 +138,7 @@ detoast_attr(varlena *attr)
/*
* This is an externally stored datum --- fetch it back from there
*/
- attr = toast_fetch_datum(attr);
+ attr = toast_fetch_datum(attr, 0);
/* If it's compressed, decompress it */
if (VARATT_IS_COMPRESSED(attr))
{
@@ -262,7 +280,7 @@ detoast_attr_slice(varlena *attr,
preslice = toast_fetch_datum_slice(attr, 0, max_size);
}
else
- preslice = toast_fetch_datum(attr);
+ preslice = toast_fetch_datum(attr, 0);
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
@@ -337,10 +355,12 @@ detoast_attr_slice(varlena *attr,
*
* Reconstruct an in memory Datum from the chunks saved
* in the toast relation
+ *
+ * If flags uses TOAST_MISSING_OK, returns NULL when chunks are missing.
* ----------
*/
static varlena *
-toast_fetch_datum(varlena *attr)
+toast_fetch_datum(varlena *attr, uint32 flags)
{
Relation toastrel;
varlena *result;
@@ -372,8 +392,14 @@ toast_fetch_datum(varlena *attr)
toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock);
/* Fetch all chunks */
- table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
- attrsize, 0, attrsize, result);
+ if (!table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
+ attrsize, 0, attrsize, result,
+ flags))
+ {
+ pfree(result);
+ table_close(toastrel, AccessShareLock);
+ return NULL;
+ }
/* Close toast table */
table_close(toastrel, AccessShareLock);
@@ -454,7 +480,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
/* Fetch all chunks */
table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
attrsize, sliceoffset, slicelength,
- result);
+ result, 0);
/* Close toast table */
table_close(toastrel, AccessShareLock);
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index abfd8e8970a6..c962393aab22 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -2236,7 +2236,7 @@ heap_prepare_insert(Relation relation, HeapTuple tup, TransactionId xid,
return tup;
}
else if (HeapTupleHasExternal(tup) || tup->t_len > TOAST_TUPLE_THRESHOLD)
- return heap_toast_insert_or_update(relation, tup, NULL, options);
+ return heap_toast_insert_or_update(relation, tup, NULL, options, 0);
else
return tup;
}
@@ -3871,7 +3871,7 @@ l2:
if (need_toast)
{
/* Note we always use WAL and FSM during updates */
- heaptup = heap_toast_insert_or_update(relation, newtup, &oldtup, 0);
+ heaptup = heap_toast_insert_or_update(relation, newtup, &oldtup, 0, 0);
newtupsize = MAXALIGN(heaptup->t_len);
}
else
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index bf87430cf017..e900ea0c0673 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -19,6 +19,7 @@
*/
#include "postgres.h"
+#include "access/detoast.h"
#include "access/genam.h"
#include "access/heapam.h"
#include "access/heaptoast.h"
@@ -26,6 +27,7 @@
#include "access/rewriteheap.h"
#include "access/syncscan.h"
#include "access/tableam.h"
+#include "access/toast_helper.h"
#include "access/tsmapi.h"
#include "access/visibilitymap.h"
#include "access/xact.h"
@@ -36,6 +38,7 @@
#include "commands/progress.h"
#include "executor/executor.h"
#include "miscadmin.h"
+#include "optimizer/optimizer.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "storage/bufpage.h"
@@ -48,9 +51,11 @@
#include "utils/rel.h"
#include "utils/tuplesort.h"
-static void reform_and_rewrite_tuple(HeapTuple tuple,
+/* See toast_helper.h for values of "flags" */
+static bool reform_and_rewrite_tuple(HeapTuple tuple,
Relation OldHeap, Relation NewHeap,
- Datum *values, bool *isnull, RewriteState rwstate);
+ Datum *values, bool *isnull, RewriteState rwstate,
+ uint32 flags);
static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap,
Relation NewHeap, Datum *values, bool *isnull,
BulkInsertState bistate);
@@ -711,6 +716,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
HeapTuple tuple;
Buffer buf;
bool isdead;
+ bool recently_dead = false;
CHECK_FOR_INTERRUPTS();
@@ -796,6 +802,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
break;
case HEAPTUPLE_RECENTLY_DEAD:
*tups_recently_dead += 1;
+ recently_dead = true;
pg_fallthrough;
case HEAPTUPLE_LIVE:
/* Live or recently dead, must copy it */
@@ -829,6 +836,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
RelationGetRelationName(OldHeap));
/* treat as recently dead */
*tups_recently_dead += 1;
+ recently_dead = true;
isdead = false;
break;
default:
@@ -857,6 +865,45 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
*num_tuples += 1;
if (tuplesort != NULL)
{
+ /*
+ * For RECENTLY_DEAD tuples, verify TOAST data is still available
+ * before putting the tuple in the sort. If a concurrent VACUUM
+ * already reclaimed it, skip the tuple.
+ */
+ if (recently_dead)
+ {
+ bool skip = false;
+
+ heap_deform_tuple(tuple, oldTupDesc, values, isnull);
+ for (int i = 0; i < oldTupDesc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(oldTupDesc, i);
+ varlena *d;
+
+ if (att->attlen != -1 || att->attisdropped)
+ continue;
+ if (isnull[i])
+ continue;
+ if (!VARATT_IS_EXTERNAL_ONDISK(DatumGetPointer(values[i])))
+ continue;
+ d = detoast_external_attr_extended(
+ (varlena *) DatumGetPointer(values[i]));
+ if (d == NULL)
+ {
+ skip = true;
+ break;
+ }
+ pfree(d);
+ }
+ if (skip)
+ {
+ *num_tuples -= 1;
+ *tups_vacuumed += 1;
+ *tups_recently_dead -= 1;
+ continue;
+ }
+ }
+
tuplesort_putheaptuple(tuplesort, tuple);
/*
@@ -875,8 +922,22 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
int64 ct_val[2];
if (!concurrent)
- reform_and_rewrite_tuple(tuple, OldHeap, NewHeap,
- values, isnull, rwstate);
+ {
+ if (!reform_and_rewrite_tuple(tuple, OldHeap, NewHeap,
+ values, isnull, rwstate,
+ recently_dead ? TOAST_MISSING_OK : 0))
+ {
+ /*
+ * Missing TOAST chunks for a recently-dead tuple. Treat
+ * it as dead.
+ */
+ *tups_vacuumed += 1;
+ *num_tuples -= 1;
+ if (recently_dead)
+ *tups_recently_dead -= 1;
+ continue;
+ }
+ }
else
heap_insert_for_repack(tuple, OldHeap, NewHeap,
values, isnull, bistate);
@@ -928,10 +989,12 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
n_tuples += 1;
if (!concurrent)
- reform_and_rewrite_tuple(tuple,
- OldHeap, NewHeap,
- values, isnull,
- rwstate);
+ {
+ (void) reform_and_rewrite_tuple(tuple,
+ OldHeap, NewHeap,
+ values, isnull,
+ rwstate, 0);
+ }
else
heap_insert_for_repack(tuple, OldHeap, NewHeap,
values, isnull, bistate);
@@ -1163,6 +1226,7 @@ heapam_index_build_range_scan(Relation heapRelation,
BlockNumber previous_blkno = InvalidBlockNumber;
BlockNumber root_blkno = InvalidBlockNumber;
OffsetNumber root_offsets[MaxHeapTuplesPerPage];
+ Bitmapset *index_attrs = NULL;
/*
* sanity checks
@@ -1259,6 +1323,21 @@ heapam_index_build_range_scan(Relation heapRelation,
!TransactionIdIsValid(OldestXmin));
Assert(snapshot == SnapshotAny || !anyvisible);
+ /*
+ * Extract the set of attributes an index may need to detoast, including
+ * expressions and predicates.
+ */
+ for (int i = 0; i < indexInfo->ii_NumIndexAttrs; i++)
+ {
+ int attnum = indexInfo->ii_IndexAttrNumbers[i];
+
+ if (attnum != 0)
+ index_attrs = bms_add_member(index_attrs,
+ attnum - FirstLowInvalidHeapAttributeNumber);
+ }
+ pull_varattnos((Node *) indexInfo->ii_Expressions, 1, &index_attrs);
+ pull_varattnos((Node *) indexInfo->ii_Predicate, 1, &index_attrs);
+
/* Publish number of blocks to scan */
if (progress)
{
@@ -1607,6 +1686,44 @@ heapam_index_build_range_scan(Relation heapRelation,
continue;
}
+ /* For RECENTLY_DEAD tuples, pre-detoast external TOAST values. */
+ if (!tupleIsAlive)
+ {
+ bool skip = false;
+ int attno;
+
+ slot_getallattrs(slot);
+ attno = -1;
+ while ((attno = bms_next_member(index_attrs, attno)) >= 0)
+ {
+ int attnum = attno + FirstLowInvalidHeapAttributeNumber;
+ Form_pg_attribute att;
+
+ if (attnum <= 0)
+ continue; /* system column */
+ att = TupleDescAttr(RelationGetDescr(heapRelation), attnum - 1);
+ if (att->attlen != -1 || att->attisdropped)
+ continue;
+ if (slot->tts_isnull[attnum - 1])
+ continue;
+ if (VARATT_IS_EXTERNAL_ONDISK(DatumGetPointer(slot->tts_values[attnum - 1])))
+ {
+ varlena *detoasted;
+
+ detoasted = detoast_external_attr_extended(
+ (varlena *) DatumGetPointer(slot->tts_values[attnum - 1]));
+ if (detoasted == NULL)
+ {
+ skip = true;
+ break;
+ }
+ slot->tts_values[attnum - 1] = PointerGetDatum(detoasted);
+ }
+ }
+ if (skip)
+ continue;
+ }
+
/*
* For the current heap tuple, extract all the attributes we use in
* this index, and note which are null. This also performs evaluation
@@ -2341,20 +2458,29 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate,
* SET WITHOUT OIDS.
*
* So, we must reconstruct the tuple from component Datums.
+ *
+ * Returns true on success, and false on failure if flags uses
+ * TOAST_MISSING_OK. See toast_helper.h.
*/
-static void
+static bool
reform_and_rewrite_tuple(HeapTuple tuple,
Relation OldHeap, Relation NewHeap,
- Datum *values, bool *isnull, RewriteState rwstate)
+ Datum *values, bool *isnull, RewriteState rwstate,
+ uint32 flags)
{
HeapTuple newtuple;
newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull);
/* The heap rewrite module does the rest */
- rewrite_heap_tuple(rwstate, tuple, newtuple);
+ if (!rewrite_heap_tuple(rwstate, tuple, newtuple, flags))
+ {
+ heap_freetuple(newtuple);
+ return false;
+ }
heap_freetuple(newtuple);
+ return true;
}
/*
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 03f885a25b07..ba5942a94953 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -84,9 +84,10 @@ heap_toast_delete(Relation rel, HeapTuple oldtup, bool is_speculative)
* newtup: the candidate new tuple to be inserted
* oldtup: the old row version for UPDATE, or NULL for INSERT
* options: options to be passed to heap_insert() for toast rows
+ * flags: additional control flags, see heaptoast.h.
* Result:
- * either newtup if no toasting is needed, or a palloc'd modified tuple
- * that is what should actually get stored
+ * either newtup if no toasting is needed, a palloc'd modified tuple
+ * that is what should actually get stored, or NULL.
*
* NOTE: neither newtup nor oldtup will be modified. This is a change
* from the pre-8.1 API of this routine.
@@ -94,7 +95,7 @@ heap_toast_delete(Relation rel, HeapTuple oldtup, bool is_speculative)
*/
HeapTuple
heap_toast_insert_or_update(Relation rel, HeapTuple newtup, HeapTuple oldtup,
- uint32 options)
+ uint32 options, uint32 flags)
{
HeapTuple result_tuple;
TupleDesc tupleDesc;
@@ -154,7 +155,8 @@ heap_toast_insert_or_update(Relation rel, HeapTuple newtup, HeapTuple oldtup,
ttc.ttc_oldisnull = toast_oldisnull;
}
ttc.ttc_attr = toast_attr;
- toast_tuple_init(&ttc);
+ if (!toast_tuple_init(&ttc, flags))
+ return NULL;
/* ----------
* Compress and/or save external until data fits into target length
@@ -621,11 +623,15 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
* sliceoffset is the byte offset within the TOAST value from which to fetch.
* slicelength is the number of bytes to be fetched from the TOAST value.
* result is the varlena into which the results should be written.
+ * flags records more options, see toast_helper.h.
+ *
+ * Returns true on success, or false if flags uses TOAST_MISSING_OK and
+ * chunks are missing rather than raising an error.
*/
-void
+bool
heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
int32 sliceoffset, int32 slicelength,
- varlena *result)
+ varlena *result, uint32 flags)
{
Relation *toastidxs;
ScanKeyData toastkey[3];
@@ -779,13 +785,23 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
* Final checks that we successfully fetched the datum
*/
if (expectedchunk != (endchunk + 1))
+ {
+ if (flags & TOAST_MISSING_OK)
+ {
+ systable_endscan_ordered(toastscan);
+ toast_close_indexes(toastidxs, num_indexes, AccessShareLock);
+ return false;
+ }
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
errmsg_internal("missing chunk number %d for toast value %u in %s",
expectedchunk, valueid,
RelationGetRelationName(toastrel))));
+ }
/* End scan and close indexes. */
systable_endscan_ordered(toastscan);
toast_close_indexes(toastidxs, num_indexes, AccessShareLock);
+
+ return true;
}
diff --git a/src/backend/access/heap/rewriteheap.c b/src/backend/access/heap/rewriteheap.c
index 0ccd392c3dc0..811f1934db34 100644
--- a/src/backend/access/heap/rewriteheap.c
+++ b/src/backend/access/heap/rewriteheap.c
@@ -211,7 +211,8 @@ typedef struct RewriteMappingDataEntry
/* prototypes for internal functions */
-static void raw_heap_insert(RewriteState state, HeapTuple tup);
+static bool raw_heap_insert(RewriteState state, HeapTuple tup,
+ uint32 flags);
/* internal logical remapping prototypes */
static void logical_begin_heap_rewrite(RewriteState state);
@@ -309,7 +310,7 @@ end_heap_rewrite(RewriteState state)
while ((unresolved = hash_seq_search(&seq_status)) != NULL)
{
ItemPointerSetInvalid(&unresolved->tuple->t_data->t_ctid);
- raw_heap_insert(state, unresolved->tuple);
+ raw_heap_insert(state, unresolved->tuple, 0);
}
/* Write the last page, if any */
@@ -338,9 +339,10 @@ end_heap_rewrite(RewriteState state)
* old_tuple original tuple in the old heap
* new_tuple new, rewritten tuple to be inserted to new heap
*/
-void
+bool
rewrite_heap_tuple(RewriteState state,
- HeapTuple old_tuple, HeapTuple new_tuple)
+ HeapTuple old_tuple, HeapTuple new_tuple,
+ uint32 flags)
{
MemoryContext old_cxt;
ItemPointerData old_tid;
@@ -437,7 +439,7 @@ rewrite_heap_tuple(RewriteState state,
* tuple will be written.
*/
MemoryContextSwitchTo(old_cxt);
- return;
+ return true;
}
}
@@ -455,7 +457,13 @@ rewrite_heap_tuple(RewriteState state,
ItemPointerData new_tid;
/* Insert the tuple and find out where it's put in new_heap */
- raw_heap_insert(state, new_tuple);
+ if (!raw_heap_insert(state, new_tuple, flags))
+ {
+ if (free_new)
+ heap_freetuple(new_tuple);
+ MemoryContextSwitchTo(old_cxt);
+ return false;
+ }
new_tid = new_tuple->t_self;
logical_rewrite_heap_tuple(state, old_tid, new_tuple);
@@ -532,6 +540,7 @@ rewrite_heap_tuple(RewriteState state,
}
MemoryContextSwitchTo(old_cxt);
+ return true;
}
/*
@@ -593,8 +602,8 @@ rewrite_heap_dead_tuple(RewriteState state, HeapTuple old_tuple)
* tuple is invalid on entry, it's replaced with the new TID as well (in
* the inserted data only, not in the caller's copy).
*/
-static void
-raw_heap_insert(RewriteState state, HeapTuple tup)
+static bool
+raw_heap_insert(RewriteState state, HeapTuple tup, uint32 flags)
{
Page page;
Size pageFreeSpace,
@@ -628,7 +637,9 @@ raw_heap_insert(RewriteState state, HeapTuple tup)
options |= HEAP_INSERT_NO_LOGICAL;
heaptup = heap_toast_insert_or_update(state->rs_new_rel, tup, NULL,
- options);
+ options, flags);
+ if (heaptup == NULL)
+ return false;
}
else
heaptup = tup;
@@ -702,6 +713,8 @@ raw_heap_insert(RewriteState state, HeapTuple tup)
/* If heaptup is a private copy, release it. */
if (heaptup != tup)
heap_freetuple(heaptup);
+
+ return true;
}
/* ------------------------------------------------------------------------
diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c
index 2f2022d99510..50ce051a6876 100644
--- a/src/backend/access/table/toast_helper.c
+++ b/src/backend/access/table/toast_helper.c
@@ -36,9 +36,13 @@
* toast_flags is just a single uint8, but toast_attr is a caller-provided
* array with a length equal to tupleDesc->natts. The caller need not
* perform any initialization of the array before calling this function.
+ *
+ * If flags uses TOAST_MISSING_OK and an external TOAST value cannot be
+ * fetched, returns false instead of raising an error. Returns true on
+ * success. See toast_helper.h.
*/
-void
-toast_tuple_init(ToastTupleContext *ttc)
+bool
+toast_tuple_init(ToastTupleContext *ttc, uint32 flags)
{
TupleDesc tupleDesc = ttc->ttc_rel->rd_att;
int numAttrs = tupleDesc->natts;
@@ -137,7 +141,14 @@ toast_tuple_init(ToastTupleContext *ttc)
if (VARATT_IS_EXTERNAL(new_value))
{
ttc->ttc_attr[i].tai_oldexternal = new_value;
- if (att->attstorage == TYPSTORAGE_PLAIN)
+ if ((flags & TOAST_MISSING_OK) != 0 &&
+ VARATT_IS_EXTERNAL_ONDISK(new_value))
+ {
+ new_value = detoast_external_attr_extended(new_value);
+ if (new_value == NULL)
+ return false;
+ }
+ else if (att->attstorage == TYPSTORAGE_PLAIN)
new_value = detoast_attr(new_value);
else
new_value = detoast_external_attr(new_value);
@@ -159,6 +170,8 @@ toast_tuple_init(ToastTupleContext *ttc)
ttc->ttc_attr[i].tai_colflags |= TOASTCOL_IGNORE;
}
}
+
+ return true;
}
/*
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index ad0d85f41897..7e583ee39a10 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -63,6 +63,7 @@ tests += {
't/052_checkpoint_segment_missing.pl',
't/053_standby_login_event_trigger.pl',
't/054_unlogged_sequence_promotion.pl',
+ 't/055_rewrite_stale_xmin.pl',
],
},
}
diff --git a/src/test/recovery/t/055_rewrite_stale_xmin.pl b/src/test/recovery/t/055_rewrite_stale_xmin.pl
new file mode 100644
index 000000000000..dad95c0cf967
--- /dev/null
+++ b/src/test/recovery/t/055_rewrite_stale_xmin.pl
@@ -0,0 +1,124 @@
+# Copyright (c) 2025-2026, PostgreSQL Global Development Group
+#
+# Test that heap rewrites (CLUSTER, VACUUM FULL, REPACK) and index builds
+# work in the case when the backend's own xmin is held back by a transaction
+# in another database.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+
+my $node = PostgreSQL::Test::Cluster->new('primary');
+$node->init;
+# No Autovacuum, no random interferences.
+$node->append_conf(
+ 'postgresql.conf', qq[
+autovacuum = off
+]);
+$node->start;
+
+# Create two databases.
+$node->safe_psql('postgres', 'CREATE DATABASE db_holder');
+$node->safe_psql('postgres', 'CREATE DATABASE db_target');
+
+# Set up a table with TOAST data. Returns the toast table name.
+sub setup_table
+{
+ my ($tblname, $extra_ddl) = @_;
+ $extra_ddl //= '';
+
+ $node->safe_psql(
+ 'db_target', qq{
+ DROP TABLE IF EXISTS $tblname;
+ CREATE TABLE $tblname (id int PRIMARY KEY, data text)
+ WITH (autovacuum_enabled = off);
+ ALTER TABLE $tblname ALTER COLUMN data SET STORAGE EXTERNAL;
+ INSERT INTO $tblname VALUES (1, repeat('x', 2500));
+ INSERT INTO $tblname VALUES (2, repeat('z', 2500));
+ $extra_ddl
+ });
+
+ my $toast = $node->safe_psql(
+ 'db_target', qq{
+ SELECT 'pg_toast.' || relname FROM pg_class
+ WHERE oid = (SELECT reltoastrelid FROM pg_class
+ WHERE relname = '$tblname');
+ });
+ chomp $toast;
+ return $toast;
+}
+
+# Create precondition state. Requires that $holder is already running with
+# a write XID.
+sub create_missing_toast_state
+{
+ my ($tblname, $toast_table) = @_;
+
+ # Hold xmin in same DB to prevent main tuple removal.
+ my $xmin_holder = $node->background_psql('db_target', on_error_stop => 1);
+ $xmin_holder->query_safe("BEGIN ISOLATION LEVEL REPEATABLE READ");
+ $xmin_holder->query_safe("SELECT 1");
+
+ # Delete the row with TOAST data.
+ $node->safe_psql('db_target', "DELETE FROM $tblname WHERE id = 1");
+
+ # VACUUM main table only. The xmin_holder keeps the dead tuple.
+ $node->safe_psql('db_target', "VACUUM (PROCESS_TOAST false) $tblname");
+
+ # Release same-DB xmin.
+ $xmin_holder->query_safe("COMMIT");
+ $xmin_holder->quit;
+
+ # Vacuum toast table: toast chunks now DEAD and removed.
+ $node->safe_psql('db_target', "VACUUM $toast_table");
+}
+
+# Open a transaction holding an XID before any deletes. This XID will hold
+# back GetSnapshotData() globally, on a different database than the one
+# running the various rewrite operations.
+my $holder = $node->background_psql('db_holder', on_error_stop => 1);
+$holder->query_safe("BEGIN");
+$holder->query_safe("CREATE TEMP TABLE xid_holder(x int)");
+
+# Test 1: CLUSTER
+my $toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+my ($ret, $stdout, $stderr) =
+ $node->psql('db_target', 'CLUSTER rewrite_test USING rewrite_test_pkey');
+is($ret, 0, "CLUSTER succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "CLUSTER: no missing-chunk error");
+
+# Test 2: VACUUM FULL
+$toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+($ret, $stdout, $stderr) =
+ $node->psql('db_target', 'VACUUM FULL rewrite_test');
+is($ret, 0, "VACUUM FULL succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "VACUUM FULL: no missing-chunk error");
+
+# Test 3: REPACK
+$toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+($ret, $stdout, $stderr) = $node->psql('db_target', 'REPACK rewrite_test');
+is($ret, 0, "REPACK succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "REPACK: no missing-chunk error");
+
+# Test 4: CREATE INDEX on toasted column
+$toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+($ret, $stdout, $stderr) = $node->psql('db_target',
+ 'CREATE INDEX rewrite_test_data_idx ON rewrite_test (data)');
+is($ret, 0, "CREATE INDEX succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "CREATE INDEX: no missing-chunk error");
+
+# Cleanup.
+$holder->query_safe("COMMIT");
+$holder->quit;
+
+$node->stop;
+done_testing();
diff --git a/contrib/dblink/.gitignore b/contrib/dblink/.gitignore
index 5dcb3ff97235..33bc8d8ffeb3 100644
--- a/contrib/dblink/.gitignore
+++ b/contrib/dblink/.gitignore
@@ -2,3 +2,5 @@
/log/
/results/
/tmp_check/
+/output_iso/
+/tmp_check_iso/
diff --git a/contrib/dblink/Makefile b/contrib/dblink/Makefile
index fde0b49ddbbd..a0ba41280813 100644
--- a/contrib/dblink/Makefile
+++ b/contrib/dblink/Makefile
@@ -13,6 +13,8 @@ PGFILEDESC = "dblink - connect to other PostgreSQL databases"
REGRESS = dblink
REGRESS_OPTS = --dlpath=$(top_builddir)/src/test/regress
+ISOLATION = rewrite_stale_xmin
+ISOLATION_OPTS = --load-extension=dblink
TAP_TESTS = 1
ifdef USE_PGXS
diff --git a/contrib/dblink/expected/rewrite_stale_xmin.out b/contrib/dblink/expected/rewrite_stale_xmin.out
new file mode 100644
index 000000000000..2e5a7940ea26
--- /dev/null
+++ b/contrib/dblink/expected/rewrite_stale_xmin.out
@@ -0,0 +1,237 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_cluster s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_cluster:
+ CLUSTER rewrite_test USING rewrite_test_pkey;
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_vacuum_full s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_vacuum_full:
+ VACUUM FULL rewrite_test;
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_repack s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_repack:
+ REPACK rewrite_test;
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_create_index s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_create_index:
+ CREATE INDEX rewrite_test_data_idx ON rewrite_test (data);
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
diff --git a/contrib/dblink/meson.build b/contrib/dblink/meson.build
index e2489f41229f..86fbd372b880 100644
--- a/contrib/dblink/meson.build
+++ b/contrib/dblink/meson.build
@@ -36,6 +36,11 @@ tests += {
],
'regress_args': ['--dlpath', meson.project_build_root() / 'src/test/regress'],
},
+ 'isolation': {
+ 'specs': [
+ 'rewrite_stale_xmin',
+ ],
+ },
'tap': {
'tests': [
't/001_auth_scram.pl',
diff --git a/contrib/dblink/specs/rewrite_stale_xmin.spec b/contrib/dblink/specs/rewrite_stale_xmin.spec
new file mode 100644
index 000000000000..fe88e789f854
--- /dev/null
+++ b/contrib/dblink/specs/rewrite_stale_xmin.spec
@@ -0,0 +1,137 @@
+# Test that heap rewrites work when the rewriting backend's own xmin is
+# held back by a transaction in another database.
+
+setup
+{
+ CREATE DATABASE regress_other_db;
+}
+
+setup
+{
+ CREATE EXTENSION IF NOT EXISTS dblink;
+
+ CREATE TABLE rewrite_test (id int PRIMARY KEY, data text)
+ WITH (autovacuum_enabled = off);
+ ALTER TABLE rewrite_test ALTER COLUMN data SET STORAGE EXTERNAL;
+ INSERT INTO rewrite_test VALUES (1, repeat('x', 2500));
+ INSERT INTO rewrite_test VALUES (2, repeat('y', 2500));
+
+ -- Rename the toast table to a known name so we can VACUUM it directly.
+ SET allow_system_table_mods TO true;
+ DO $$DECLARE r record;
+ BEGIN
+ SELECT INTO r reltoastrelid::regclass::text AS table_name
+ FROM pg_class WHERE oid = 'rewrite_test'::regclass;
+ EXECUTE 'ALTER TABLE ' || r.table_name || ' RENAME TO rewrite_test_toast;';
+ END$$;
+ RESET allow_system_table_mods;
+}
+
+teardown
+{
+ DROP DATABASE regress_other_db;
+}
+
+# Session s1: use dblink to hold a transaction in another database.
+# This must start before the DELETE so its XID is older, ensuring
+# GetSnapshotData() holds back any rewrites.
+session s1
+step s1_hold_xmin
+{
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+}
+step s1_release
+{
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+}
+
+# Session s2: holds xmin in the SAME database temporarily, so the
+# main-table vacuum preserves the dead tuple as RECENTLY_DEAD.
+session s2
+step s2_begin
+{
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+}
+step s2_commit { COMMIT; }
+
+# Session s3: performs DML, split vacuum, and the rewrite.
+session s3
+step s3_delete
+{
+ DELETE FROM rewrite_test WHERE id = 1;
+}
+step s3_vacuum_main_only
+{
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+}
+step s3_vacuum_toast
+{
+ VACUUM pg_toast.rewrite_test_toast;
+}
+step s3_cluster
+{
+ CLUSTER rewrite_test USING rewrite_test_pkey;
+}
+step s3_vacuum_full
+{
+ VACUUM FULL rewrite_test;
+}
+step s3_repack
+{
+ REPACK rewrite_test;
+}
+step s3_create_index
+{
+ CREATE INDEX rewrite_test_data_idx ON rewrite_test (data);
+}
+
+teardown { DROP TABLE IF EXISTS rewrite_test; }
+
+# CLUSTER
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_cluster
+ s1_release
+
+# VACUUM FULL
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_vacuum_full
+ s1_release
+
+# REPACK
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_repack
+ s1_release
+
+# CREATE INDEX
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_create_index
+ s1_release
--
2.55.0
Attachments:
[text/plain] v4-0001-Fix-missing-chunk-errors-during-heap-rewrites-and.patch (40.7K, ../../[email protected]/2-v4-0001-Fix-missing-chunk-errors-during-heap-rewrites-and.patch)
download | inline diff:
From 8d961035cdccfe9e76a859c5acd838e8f6969382 Mon Sep 17 00:00:00 2001
From: Michael Paquier <[email protected]>
Date: Thu, 9 Jul 2026 11:46:21 +0900
Subject: [PATCH v4] Fix "missing chunk" errors during heap rewrites and index
builds
Blah, blah.
This problem has been reported recently by Alexander for REPACK, but it
is a much older issue (tested down to 9.5 for curiosity's sake, but I
suspect that it is much older than that).
XXX: This patch includes both TAP and isolation tests. We should remove
one pattern at the end.
Reported-by: Alexander Lakhin <[email protected]>
Discussion: https://postgr.es/m/[email protected]
Discussion: https://postgr.es/m/[email protected]
---
src/include/access/detoast.h | 9 +
src/include/access/heaptoast.h | 18 +-
src/include/access/rewriteheap.h | 6 +-
src/include/access/tableam.h | 26 +-
src/include/access/toast_helper.h | 10 +-
src/backend/access/common/detoast.c | 42 +++-
src/backend/access/heap/heapam.c | 4 +-
src/backend/access/heap/heapam_handler.c | 148 ++++++++++-
src/backend/access/heap/heaptoast.c | 28 ++-
src/backend/access/heap/rewriteheap.c | 31 ++-
src/backend/access/table/toast_helper.c | 19 +-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/055_rewrite_stale_xmin.pl | 124 +++++++++
contrib/dblink/.gitignore | 2 +
contrib/dblink/Makefile | 2 +
.../dblink/expected/rewrite_stale_xmin.out | 237 ++++++++++++++++++
contrib/dblink/meson.build | 5 +
contrib/dblink/specs/rewrite_stale_xmin.spec | 137 ++++++++++
18 files changed, 794 insertions(+), 55 deletions(-)
create mode 100644 src/test/recovery/t/055_rewrite_stale_xmin.pl
create mode 100644 contrib/dblink/expected/rewrite_stale_xmin.out
create mode 100644 contrib/dblink/specs/rewrite_stale_xmin.spec
diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h
index fbd98181a3a1..c5eeed04b011 100644
--- a/src/include/access/detoast.h
+++ b/src/include/access/detoast.h
@@ -43,6 +43,15 @@ do { \
*/
extern varlena *detoast_external_attr(varlena *attr);
+/* ----------
+ * detoast_external_attr_extended() -
+ *
+ * Like detoast_external_attr, but returns NULL if toast chunks
+ * are missing (instead of raising an error).
+ * ----------
+ */
+extern varlena *detoast_external_attr_extended(varlena *attr);
+
/* ----------
* detoast_attr() -
*
diff --git a/src/include/access/heaptoast.h b/src/include/access/heaptoast.h
index 631cb1836b96..bc4087b3c95c 100644
--- a/src/include/access/heaptoast.h
+++ b/src/include/access/heaptoast.h
@@ -91,11 +91,15 @@
/* ----------
* heap_toast_insert_or_update -
*
- * Called by heap_insert() and heap_update().
+ * Called by heap_insert() and heap_update(). Returns NULL if the
+ * insert/update could not be completed.
+ *
+ * See toast_helper.h for the values of "flags".
* ----------
*/
extern HeapTuple heap_toast_insert_or_update(Relation rel, HeapTuple newtup,
- HeapTuple oldtup, uint32 options);
+ HeapTuple oldtup, uint32 options,
+ uint32 flags);
/* ----------
* heap_toast_delete -
@@ -139,11 +143,15 @@ extern HeapTuple toast_build_flattened_tuple(TupleDesc tupleDesc,
/* ----------
* heap_fetch_toast_slice
*
- * Fetch a slice from a toast value stored in a heap table.
+ * Fetch a slice from a toast value stored in a heap table. Returns
+ * false if the slice could not be fetched, otherwise true on success.
+ *
+ * See toast_helper.h for the values of "flags".
* ----------
*/
-extern void heap_fetch_toast_slice(Relation toastrel, Oid valueid,
+extern bool heap_fetch_toast_slice(Relation toastrel, Oid valueid,
int32 attrsize, int32 sliceoffset,
- int32 slicelength, varlena *result);
+ int32 slicelength, varlena *result,
+ uint32 flags);
#endif /* HEAPTOAST_H */
diff --git a/src/include/access/rewriteheap.h b/src/include/access/rewriteheap.h
index 6ccf7b45c04e..09110f6337a5 100644
--- a/src/include/access/rewriteheap.h
+++ b/src/include/access/rewriteheap.h
@@ -25,8 +25,10 @@ extern RewriteState begin_heap_rewrite(Relation old_heap, Relation new_heap,
TransactionId oldest_xmin, TransactionId freeze_xid,
MultiXactId cutoff_multi);
extern void end_heap_rewrite(RewriteState state);
-extern void rewrite_heap_tuple(RewriteState state, HeapTuple old_tuple,
- HeapTuple new_tuple);
+
+/* See toast_helper.h for the values of "flags" */
+extern bool rewrite_heap_tuple(RewriteState state, HeapTuple old_tuple,
+ HeapTuple new_tuple, uint32 flags);
extern bool rewrite_heap_dead_tuple(RewriteState state, HeapTuple old_tuple);
/*
diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h
index f2c36696bcad..707baf804e3a 100644
--- a/src/include/access/tableam.h
+++ b/src/include/access/tableam.h
@@ -782,12 +782,17 @@ typedef struct TableAmRoutine
* This callback is invoked when detoasting a value stored in a toast
* table implemented by this AM. See table_relation_fetch_toast_slice()
* for more details.
+ *
+ * If TOAST_MISSING_OK is set in flags, returns false when chunks are
+ * missing instead of raising an error. Returns true on success. See
+ * toast_helper.h for possible values of "flags".
*/
- void (*relation_fetch_toast_slice) (Relation toastrel, Oid valueid,
+ bool (*relation_fetch_toast_slice) (Relation toastrel, Oid valueid,
int32 attrsize,
int32 sliceoffset,
int32 slicelength,
- varlena *result);
+ varlena *result,
+ uint32 flags);
/* ------------------------------------------------------------------------
@@ -1981,16 +1986,21 @@ table_relation_toast_am(Relation rel)
*
* result is caller-allocated space into which the fetched bytes should be
* stored.
+ *
+ * flags: if TOAST_MISSING_OK is set, return false when toast chunks are
+ * missing instead of raising an error. Returns true on success. See
+ * also toast_helper.h.
*/
-static inline void
+static inline bool
table_relation_fetch_toast_slice(Relation toastrel, Oid valueid,
int32 attrsize, int32 sliceoffset,
- int32 slicelength, varlena *result)
+ int32 slicelength, varlena *result,
+ uint32 flags)
{
- toastrel->rd_tableam->relation_fetch_toast_slice(toastrel, valueid,
- attrsize,
- sliceoffset, slicelength,
- result);
+ return toastrel->rd_tableam->relation_fetch_toast_slice(toastrel, valueid,
+ attrsize,
+ sliceoffset, slicelength,
+ result, flags);
}
diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h
index 2ec92397f267..4abebe82798f 100644
--- a/src/include/access/toast_helper.h
+++ b/src/include/access/toast_helper.h
@@ -101,7 +101,15 @@ typedef struct
#define TOASTCOL_IGNORE 0x0010
#define TOASTCOL_INCOMPRESSIBLE 0x0020
-extern void toast_tuple_init(ToastTupleContext *ttc);
+/*
+ * Flags for TOAST fetch operations.
+ *
+ * TOAST_MISSING_OK, when set, means that a TOAST fetch tolerates missing
+ * chunks and returns a failure state instead of raising an error.
+ */
+#define TOAST_MISSING_OK (1 << 1)
+
+extern bool toast_tuple_init(ToastTupleContext *ttc, uint32 flags);
extern int toast_tuple_find_biggest_attribute(ToastTupleContext *ttc,
bool for_compression,
bool check_main);
diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c
index a6c1f3a734b2..3195d77db4b0 100644
--- a/src/backend/access/common/detoast.c
+++ b/src/backend/access/common/detoast.c
@@ -16,13 +16,14 @@
#include "access/detoast.h"
#include "access/table.h"
#include "access/tableam.h"
+#include "access/toast_helper.h"
#include "access/toast_internals.h"
#include "common/int.h"
#include "common/pg_lzcompress.h"
#include "utils/expandeddatum.h"
#include "utils/rel.h"
-static varlena *toast_fetch_datum(varlena *attr);
+static varlena *toast_fetch_datum(varlena *attr, uint32 flags);
static varlena *toast_fetch_datum_slice(varlena *attr,
int32 sliceoffset,
int32 slicelength);
@@ -51,7 +52,7 @@ detoast_external_attr(varlena *attr)
/*
* This is an external stored plain value
*/
- result = toast_fetch_datum(attr);
+ result = toast_fetch_datum(attr, 0);
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
@@ -102,6 +103,23 @@ detoast_external_attr(varlena *attr)
}
+/* ----------
+ * detoast_external_attr_extended -
+ *
+ * Like detoast_external_attr, but returns NULL if toast chunks are
+ * missing for external pointers.
+ * ----------
+ */
+varlena *
+detoast_external_attr_extended(varlena *attr)
+{
+ if (VARATT_IS_EXTERNAL_ONDISK(attr))
+ return toast_fetch_datum(attr, TOAST_MISSING_OK);
+ else
+ return detoast_external_attr(attr);
+}
+
+
/* ----------
* detoast_attr -
*
@@ -120,7 +138,7 @@ detoast_attr(varlena *attr)
/*
* This is an externally stored datum --- fetch it back from there
*/
- attr = toast_fetch_datum(attr);
+ attr = toast_fetch_datum(attr, 0);
/* If it's compressed, decompress it */
if (VARATT_IS_COMPRESSED(attr))
{
@@ -262,7 +280,7 @@ detoast_attr_slice(varlena *attr,
preslice = toast_fetch_datum_slice(attr, 0, max_size);
}
else
- preslice = toast_fetch_datum(attr);
+ preslice = toast_fetch_datum(attr, 0);
}
else if (VARATT_IS_EXTERNAL_INDIRECT(attr))
{
@@ -337,10 +355,12 @@ detoast_attr_slice(varlena *attr,
*
* Reconstruct an in memory Datum from the chunks saved
* in the toast relation
+ *
+ * If flags uses TOAST_MISSING_OK, returns NULL when chunks are missing.
* ----------
*/
static varlena *
-toast_fetch_datum(varlena *attr)
+toast_fetch_datum(varlena *attr, uint32 flags)
{
Relation toastrel;
varlena *result;
@@ -372,8 +392,14 @@ toast_fetch_datum(varlena *attr)
toastrel = table_open(toast_pointer.va_toastrelid, AccessShareLock);
/* Fetch all chunks */
- table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
- attrsize, 0, attrsize, result);
+ if (!table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
+ attrsize, 0, attrsize, result,
+ flags))
+ {
+ pfree(result);
+ table_close(toastrel, AccessShareLock);
+ return NULL;
+ }
/* Close toast table */
table_close(toastrel, AccessShareLock);
@@ -454,7 +480,7 @@ toast_fetch_datum_slice(varlena *attr, int32 sliceoffset,
/* Fetch all chunks */
table_relation_fetch_toast_slice(toastrel, toast_pointer.va_valueid,
attrsize, sliceoffset, slicelength,
- result);
+ result, 0);
/* Close toast table */
table_close(toastrel, AccessShareLock);
diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c
index abfd8e8970a6..c962393aab22 100644
--- a/src/backend/access/heap/heapam.c
+++ b/src/backend/access/heap/heapam.c
@@ -2236,7 +2236,7 @@ heap_prepare_insert(Relation relation, HeapTuple tup, TransactionId xid,
return tup;
}
else if (HeapTupleHasExternal(tup) || tup->t_len > TOAST_TUPLE_THRESHOLD)
- return heap_toast_insert_or_update(relation, tup, NULL, options);
+ return heap_toast_insert_or_update(relation, tup, NULL, options, 0);
else
return tup;
}
@@ -3871,7 +3871,7 @@ l2:
if (need_toast)
{
/* Note we always use WAL and FSM during updates */
- heaptup = heap_toast_insert_or_update(relation, newtup, &oldtup, 0);
+ heaptup = heap_toast_insert_or_update(relation, newtup, &oldtup, 0, 0);
newtupsize = MAXALIGN(heaptup->t_len);
}
else
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index bf87430cf017..e900ea0c0673 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -19,6 +19,7 @@
*/
#include "postgres.h"
+#include "access/detoast.h"
#include "access/genam.h"
#include "access/heapam.h"
#include "access/heaptoast.h"
@@ -26,6 +27,7 @@
#include "access/rewriteheap.h"
#include "access/syncscan.h"
#include "access/tableam.h"
+#include "access/toast_helper.h"
#include "access/tsmapi.h"
#include "access/visibilitymap.h"
#include "access/xact.h"
@@ -36,6 +38,7 @@
#include "commands/progress.h"
#include "executor/executor.h"
#include "miscadmin.h"
+#include "optimizer/optimizer.h"
#include "pgstat.h"
#include "storage/bufmgr.h"
#include "storage/bufpage.h"
@@ -48,9 +51,11 @@
#include "utils/rel.h"
#include "utils/tuplesort.h"
-static void reform_and_rewrite_tuple(HeapTuple tuple,
+/* See toast_helper.h for values of "flags" */
+static bool reform_and_rewrite_tuple(HeapTuple tuple,
Relation OldHeap, Relation NewHeap,
- Datum *values, bool *isnull, RewriteState rwstate);
+ Datum *values, bool *isnull, RewriteState rwstate,
+ uint32 flags);
static void heap_insert_for_repack(HeapTuple tuple, Relation OldHeap,
Relation NewHeap, Datum *values, bool *isnull,
BulkInsertState bistate);
@@ -711,6 +716,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
HeapTuple tuple;
Buffer buf;
bool isdead;
+ bool recently_dead = false;
CHECK_FOR_INTERRUPTS();
@@ -796,6 +802,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
break;
case HEAPTUPLE_RECENTLY_DEAD:
*tups_recently_dead += 1;
+ recently_dead = true;
pg_fallthrough;
case HEAPTUPLE_LIVE:
/* Live or recently dead, must copy it */
@@ -829,6 +836,7 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
RelationGetRelationName(OldHeap));
/* treat as recently dead */
*tups_recently_dead += 1;
+ recently_dead = true;
isdead = false;
break;
default:
@@ -857,6 +865,45 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
*num_tuples += 1;
if (tuplesort != NULL)
{
+ /*
+ * For RECENTLY_DEAD tuples, verify TOAST data is still available
+ * before putting the tuple in the sort. If a concurrent VACUUM
+ * already reclaimed it, skip the tuple.
+ */
+ if (recently_dead)
+ {
+ bool skip = false;
+
+ heap_deform_tuple(tuple, oldTupDesc, values, isnull);
+ for (int i = 0; i < oldTupDesc->natts; i++)
+ {
+ Form_pg_attribute att = TupleDescAttr(oldTupDesc, i);
+ varlena *d;
+
+ if (att->attlen != -1 || att->attisdropped)
+ continue;
+ if (isnull[i])
+ continue;
+ if (!VARATT_IS_EXTERNAL_ONDISK(DatumGetPointer(values[i])))
+ continue;
+ d = detoast_external_attr_extended(
+ (varlena *) DatumGetPointer(values[i]));
+ if (d == NULL)
+ {
+ skip = true;
+ break;
+ }
+ pfree(d);
+ }
+ if (skip)
+ {
+ *num_tuples -= 1;
+ *tups_vacuumed += 1;
+ *tups_recently_dead -= 1;
+ continue;
+ }
+ }
+
tuplesort_putheaptuple(tuplesort, tuple);
/*
@@ -875,8 +922,22 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
int64 ct_val[2];
if (!concurrent)
- reform_and_rewrite_tuple(tuple, OldHeap, NewHeap,
- values, isnull, rwstate);
+ {
+ if (!reform_and_rewrite_tuple(tuple, OldHeap, NewHeap,
+ values, isnull, rwstate,
+ recently_dead ? TOAST_MISSING_OK : 0))
+ {
+ /*
+ * Missing TOAST chunks for a recently-dead tuple. Treat
+ * it as dead.
+ */
+ *tups_vacuumed += 1;
+ *num_tuples -= 1;
+ if (recently_dead)
+ *tups_recently_dead -= 1;
+ continue;
+ }
+ }
else
heap_insert_for_repack(tuple, OldHeap, NewHeap,
values, isnull, bistate);
@@ -928,10 +989,12 @@ heapam_relation_copy_for_cluster(Relation OldHeap, Relation NewHeap,
n_tuples += 1;
if (!concurrent)
- reform_and_rewrite_tuple(tuple,
- OldHeap, NewHeap,
- values, isnull,
- rwstate);
+ {
+ (void) reform_and_rewrite_tuple(tuple,
+ OldHeap, NewHeap,
+ values, isnull,
+ rwstate, 0);
+ }
else
heap_insert_for_repack(tuple, OldHeap, NewHeap,
values, isnull, bistate);
@@ -1163,6 +1226,7 @@ heapam_index_build_range_scan(Relation heapRelation,
BlockNumber previous_blkno = InvalidBlockNumber;
BlockNumber root_blkno = InvalidBlockNumber;
OffsetNumber root_offsets[MaxHeapTuplesPerPage];
+ Bitmapset *index_attrs = NULL;
/*
* sanity checks
@@ -1259,6 +1323,21 @@ heapam_index_build_range_scan(Relation heapRelation,
!TransactionIdIsValid(OldestXmin));
Assert(snapshot == SnapshotAny || !anyvisible);
+ /*
+ * Extract the set of attributes an index may need to detoast, including
+ * expressions and predicates.
+ */
+ for (int i = 0; i < indexInfo->ii_NumIndexAttrs; i++)
+ {
+ int attnum = indexInfo->ii_IndexAttrNumbers[i];
+
+ if (attnum != 0)
+ index_attrs = bms_add_member(index_attrs,
+ attnum - FirstLowInvalidHeapAttributeNumber);
+ }
+ pull_varattnos((Node *) indexInfo->ii_Expressions, 1, &index_attrs);
+ pull_varattnos((Node *) indexInfo->ii_Predicate, 1, &index_attrs);
+
/* Publish number of blocks to scan */
if (progress)
{
@@ -1607,6 +1686,44 @@ heapam_index_build_range_scan(Relation heapRelation,
continue;
}
+ /* For RECENTLY_DEAD tuples, pre-detoast external TOAST values. */
+ if (!tupleIsAlive)
+ {
+ bool skip = false;
+ int attno;
+
+ slot_getallattrs(slot);
+ attno = -1;
+ while ((attno = bms_next_member(index_attrs, attno)) >= 0)
+ {
+ int attnum = attno + FirstLowInvalidHeapAttributeNumber;
+ Form_pg_attribute att;
+
+ if (attnum <= 0)
+ continue; /* system column */
+ att = TupleDescAttr(RelationGetDescr(heapRelation), attnum - 1);
+ if (att->attlen != -1 || att->attisdropped)
+ continue;
+ if (slot->tts_isnull[attnum - 1])
+ continue;
+ if (VARATT_IS_EXTERNAL_ONDISK(DatumGetPointer(slot->tts_values[attnum - 1])))
+ {
+ varlena *detoasted;
+
+ detoasted = detoast_external_attr_extended(
+ (varlena *) DatumGetPointer(slot->tts_values[attnum - 1]));
+ if (detoasted == NULL)
+ {
+ skip = true;
+ break;
+ }
+ slot->tts_values[attnum - 1] = PointerGetDatum(detoasted);
+ }
+ }
+ if (skip)
+ continue;
+ }
+
/*
* For the current heap tuple, extract all the attributes we use in
* this index, and note which are null. This also performs evaluation
@@ -2341,20 +2458,29 @@ heapam_scan_sample_next_tuple(TableScanDesc scan, SampleScanState *scanstate,
* SET WITHOUT OIDS.
*
* So, we must reconstruct the tuple from component Datums.
+ *
+ * Returns true on success, and false on failure if flags uses
+ * TOAST_MISSING_OK. See toast_helper.h.
*/
-static void
+static bool
reform_and_rewrite_tuple(HeapTuple tuple,
Relation OldHeap, Relation NewHeap,
- Datum *values, bool *isnull, RewriteState rwstate)
+ Datum *values, bool *isnull, RewriteState rwstate,
+ uint32 flags)
{
HeapTuple newtuple;
newtuple = reform_tuple(tuple, OldHeap, NewHeap, values, isnull);
/* The heap rewrite module does the rest */
- rewrite_heap_tuple(rwstate, tuple, newtuple);
+ if (!rewrite_heap_tuple(rwstate, tuple, newtuple, flags))
+ {
+ heap_freetuple(newtuple);
+ return false;
+ }
heap_freetuple(newtuple);
+ return true;
}
/*
diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c
index 03f885a25b07..ba5942a94953 100644
--- a/src/backend/access/heap/heaptoast.c
+++ b/src/backend/access/heap/heaptoast.c
@@ -84,9 +84,10 @@ heap_toast_delete(Relation rel, HeapTuple oldtup, bool is_speculative)
* newtup: the candidate new tuple to be inserted
* oldtup: the old row version for UPDATE, or NULL for INSERT
* options: options to be passed to heap_insert() for toast rows
+ * flags: additional control flags, see heaptoast.h.
* Result:
- * either newtup if no toasting is needed, or a palloc'd modified tuple
- * that is what should actually get stored
+ * either newtup if no toasting is needed, a palloc'd modified tuple
+ * that is what should actually get stored, or NULL.
*
* NOTE: neither newtup nor oldtup will be modified. This is a change
* from the pre-8.1 API of this routine.
@@ -94,7 +95,7 @@ heap_toast_delete(Relation rel, HeapTuple oldtup, bool is_speculative)
*/
HeapTuple
heap_toast_insert_or_update(Relation rel, HeapTuple newtup, HeapTuple oldtup,
- uint32 options)
+ uint32 options, uint32 flags)
{
HeapTuple result_tuple;
TupleDesc tupleDesc;
@@ -154,7 +155,8 @@ heap_toast_insert_or_update(Relation rel, HeapTuple newtup, HeapTuple oldtup,
ttc.ttc_oldisnull = toast_oldisnull;
}
ttc.ttc_attr = toast_attr;
- toast_tuple_init(&ttc);
+ if (!toast_tuple_init(&ttc, flags))
+ return NULL;
/* ----------
* Compress and/or save external until data fits into target length
@@ -621,11 +623,15 @@ toast_build_flattened_tuple(TupleDesc tupleDesc,
* sliceoffset is the byte offset within the TOAST value from which to fetch.
* slicelength is the number of bytes to be fetched from the TOAST value.
* result is the varlena into which the results should be written.
+ * flags records more options, see toast_helper.h.
+ *
+ * Returns true on success, or false if flags uses TOAST_MISSING_OK and
+ * chunks are missing rather than raising an error.
*/
-void
+bool
heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
int32 sliceoffset, int32 slicelength,
- varlena *result)
+ varlena *result, uint32 flags)
{
Relation *toastidxs;
ScanKeyData toastkey[3];
@@ -779,13 +785,23 @@ heap_fetch_toast_slice(Relation toastrel, Oid valueid, int32 attrsize,
* Final checks that we successfully fetched the datum
*/
if (expectedchunk != (endchunk + 1))
+ {
+ if (flags & TOAST_MISSING_OK)
+ {
+ systable_endscan_ordered(toastscan);
+ toast_close_indexes(toastidxs, num_indexes, AccessShareLock);
+ return false;
+ }
ereport(ERROR,
(errcode(ERRCODE_DATA_CORRUPTED),
errmsg_internal("missing chunk number %d for toast value %u in %s",
expectedchunk, valueid,
RelationGetRelationName(toastrel))));
+ }
/* End scan and close indexes. */
systable_endscan_ordered(toastscan);
toast_close_indexes(toastidxs, num_indexes, AccessShareLock);
+
+ return true;
}
diff --git a/src/backend/access/heap/rewriteheap.c b/src/backend/access/heap/rewriteheap.c
index 0ccd392c3dc0..811f1934db34 100644
--- a/src/backend/access/heap/rewriteheap.c
+++ b/src/backend/access/heap/rewriteheap.c
@@ -211,7 +211,8 @@ typedef struct RewriteMappingDataEntry
/* prototypes for internal functions */
-static void raw_heap_insert(RewriteState state, HeapTuple tup);
+static bool raw_heap_insert(RewriteState state, HeapTuple tup,
+ uint32 flags);
/* internal logical remapping prototypes */
static void logical_begin_heap_rewrite(RewriteState state);
@@ -309,7 +310,7 @@ end_heap_rewrite(RewriteState state)
while ((unresolved = hash_seq_search(&seq_status)) != NULL)
{
ItemPointerSetInvalid(&unresolved->tuple->t_data->t_ctid);
- raw_heap_insert(state, unresolved->tuple);
+ raw_heap_insert(state, unresolved->tuple, 0);
}
/* Write the last page, if any */
@@ -338,9 +339,10 @@ end_heap_rewrite(RewriteState state)
* old_tuple original tuple in the old heap
* new_tuple new, rewritten tuple to be inserted to new heap
*/
-void
+bool
rewrite_heap_tuple(RewriteState state,
- HeapTuple old_tuple, HeapTuple new_tuple)
+ HeapTuple old_tuple, HeapTuple new_tuple,
+ uint32 flags)
{
MemoryContext old_cxt;
ItemPointerData old_tid;
@@ -437,7 +439,7 @@ rewrite_heap_tuple(RewriteState state,
* tuple will be written.
*/
MemoryContextSwitchTo(old_cxt);
- return;
+ return true;
}
}
@@ -455,7 +457,13 @@ rewrite_heap_tuple(RewriteState state,
ItemPointerData new_tid;
/* Insert the tuple and find out where it's put in new_heap */
- raw_heap_insert(state, new_tuple);
+ if (!raw_heap_insert(state, new_tuple, flags))
+ {
+ if (free_new)
+ heap_freetuple(new_tuple);
+ MemoryContextSwitchTo(old_cxt);
+ return false;
+ }
new_tid = new_tuple->t_self;
logical_rewrite_heap_tuple(state, old_tid, new_tuple);
@@ -532,6 +540,7 @@ rewrite_heap_tuple(RewriteState state,
}
MemoryContextSwitchTo(old_cxt);
+ return true;
}
/*
@@ -593,8 +602,8 @@ rewrite_heap_dead_tuple(RewriteState state, HeapTuple old_tuple)
* tuple is invalid on entry, it's replaced with the new TID as well (in
* the inserted data only, not in the caller's copy).
*/
-static void
-raw_heap_insert(RewriteState state, HeapTuple tup)
+static bool
+raw_heap_insert(RewriteState state, HeapTuple tup, uint32 flags)
{
Page page;
Size pageFreeSpace,
@@ -628,7 +637,9 @@ raw_heap_insert(RewriteState state, HeapTuple tup)
options |= HEAP_INSERT_NO_LOGICAL;
heaptup = heap_toast_insert_or_update(state->rs_new_rel, tup, NULL,
- options);
+ options, flags);
+ if (heaptup == NULL)
+ return false;
}
else
heaptup = tup;
@@ -702,6 +713,8 @@ raw_heap_insert(RewriteState state, HeapTuple tup)
/* If heaptup is a private copy, release it. */
if (heaptup != tup)
heap_freetuple(heaptup);
+
+ return true;
}
/* ------------------------------------------------------------------------
diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c
index 2f2022d99510..50ce051a6876 100644
--- a/src/backend/access/table/toast_helper.c
+++ b/src/backend/access/table/toast_helper.c
@@ -36,9 +36,13 @@
* toast_flags is just a single uint8, but toast_attr is a caller-provided
* array with a length equal to tupleDesc->natts. The caller need not
* perform any initialization of the array before calling this function.
+ *
+ * If flags uses TOAST_MISSING_OK and an external TOAST value cannot be
+ * fetched, returns false instead of raising an error. Returns true on
+ * success. See toast_helper.h.
*/
-void
-toast_tuple_init(ToastTupleContext *ttc)
+bool
+toast_tuple_init(ToastTupleContext *ttc, uint32 flags)
{
TupleDesc tupleDesc = ttc->ttc_rel->rd_att;
int numAttrs = tupleDesc->natts;
@@ -137,7 +141,14 @@ toast_tuple_init(ToastTupleContext *ttc)
if (VARATT_IS_EXTERNAL(new_value))
{
ttc->ttc_attr[i].tai_oldexternal = new_value;
- if (att->attstorage == TYPSTORAGE_PLAIN)
+ if ((flags & TOAST_MISSING_OK) != 0 &&
+ VARATT_IS_EXTERNAL_ONDISK(new_value))
+ {
+ new_value = detoast_external_attr_extended(new_value);
+ if (new_value == NULL)
+ return false;
+ }
+ else if (att->attstorage == TYPSTORAGE_PLAIN)
new_value = detoast_attr(new_value);
else
new_value = detoast_external_attr(new_value);
@@ -159,6 +170,8 @@ toast_tuple_init(ToastTupleContext *ttc)
ttc->ttc_attr[i].tai_colflags |= TOASTCOL_IGNORE;
}
}
+
+ return true;
}
/*
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index ad0d85f41897..7e583ee39a10 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -63,6 +63,7 @@ tests += {
't/052_checkpoint_segment_missing.pl',
't/053_standby_login_event_trigger.pl',
't/054_unlogged_sequence_promotion.pl',
+ 't/055_rewrite_stale_xmin.pl',
],
},
}
diff --git a/src/test/recovery/t/055_rewrite_stale_xmin.pl b/src/test/recovery/t/055_rewrite_stale_xmin.pl
new file mode 100644
index 000000000000..dad95c0cf967
--- /dev/null
+++ b/src/test/recovery/t/055_rewrite_stale_xmin.pl
@@ -0,0 +1,124 @@
+# Copyright (c) 2025-2026, PostgreSQL Global Development Group
+#
+# Test that heap rewrites (CLUSTER, VACUUM FULL, REPACK) and index builds
+# work in the case when the backend's own xmin is held back by a transaction
+# in another database.
+
+use strict;
+use warnings;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+
+my $node = PostgreSQL::Test::Cluster->new('primary');
+$node->init;
+# No Autovacuum, no random interferences.
+$node->append_conf(
+ 'postgresql.conf', qq[
+autovacuum = off
+]);
+$node->start;
+
+# Create two databases.
+$node->safe_psql('postgres', 'CREATE DATABASE db_holder');
+$node->safe_psql('postgres', 'CREATE DATABASE db_target');
+
+# Set up a table with TOAST data. Returns the toast table name.
+sub setup_table
+{
+ my ($tblname, $extra_ddl) = @_;
+ $extra_ddl //= '';
+
+ $node->safe_psql(
+ 'db_target', qq{
+ DROP TABLE IF EXISTS $tblname;
+ CREATE TABLE $tblname (id int PRIMARY KEY, data text)
+ WITH (autovacuum_enabled = off);
+ ALTER TABLE $tblname ALTER COLUMN data SET STORAGE EXTERNAL;
+ INSERT INTO $tblname VALUES (1, repeat('x', 2500));
+ INSERT INTO $tblname VALUES (2, repeat('z', 2500));
+ $extra_ddl
+ });
+
+ my $toast = $node->safe_psql(
+ 'db_target', qq{
+ SELECT 'pg_toast.' || relname FROM pg_class
+ WHERE oid = (SELECT reltoastrelid FROM pg_class
+ WHERE relname = '$tblname');
+ });
+ chomp $toast;
+ return $toast;
+}
+
+# Create precondition state. Requires that $holder is already running with
+# a write XID.
+sub create_missing_toast_state
+{
+ my ($tblname, $toast_table) = @_;
+
+ # Hold xmin in same DB to prevent main tuple removal.
+ my $xmin_holder = $node->background_psql('db_target', on_error_stop => 1);
+ $xmin_holder->query_safe("BEGIN ISOLATION LEVEL REPEATABLE READ");
+ $xmin_holder->query_safe("SELECT 1");
+
+ # Delete the row with TOAST data.
+ $node->safe_psql('db_target', "DELETE FROM $tblname WHERE id = 1");
+
+ # VACUUM main table only. The xmin_holder keeps the dead tuple.
+ $node->safe_psql('db_target', "VACUUM (PROCESS_TOAST false) $tblname");
+
+ # Release same-DB xmin.
+ $xmin_holder->query_safe("COMMIT");
+ $xmin_holder->quit;
+
+ # Vacuum toast table: toast chunks now DEAD and removed.
+ $node->safe_psql('db_target', "VACUUM $toast_table");
+}
+
+# Open a transaction holding an XID before any deletes. This XID will hold
+# back GetSnapshotData() globally, on a different database than the one
+# running the various rewrite operations.
+my $holder = $node->background_psql('db_holder', on_error_stop => 1);
+$holder->query_safe("BEGIN");
+$holder->query_safe("CREATE TEMP TABLE xid_holder(x int)");
+
+# Test 1: CLUSTER
+my $toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+my ($ret, $stdout, $stderr) =
+ $node->psql('db_target', 'CLUSTER rewrite_test USING rewrite_test_pkey');
+is($ret, 0, "CLUSTER succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "CLUSTER: no missing-chunk error");
+
+# Test 2: VACUUM FULL
+$toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+($ret, $stdout, $stderr) =
+ $node->psql('db_target', 'VACUUM FULL rewrite_test');
+is($ret, 0, "VACUUM FULL succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "VACUUM FULL: no missing-chunk error");
+
+# Test 3: REPACK
+$toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+($ret, $stdout, $stderr) = $node->psql('db_target', 'REPACK rewrite_test');
+is($ret, 0, "REPACK succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "REPACK: no missing-chunk error");
+
+# Test 4: CREATE INDEX on toasted column
+$toast = setup_table('rewrite_test');
+create_missing_toast_state('rewrite_test', $toast);
+
+($ret, $stdout, $stderr) = $node->psql('db_target',
+ 'CREATE INDEX rewrite_test_data_idx ON rewrite_test (data)');
+is($ret, 0, "CREATE INDEX succeeds without missing-chunk error");
+unlike($stderr, qr/missing chunk/, "CREATE INDEX: no missing-chunk error");
+
+# Cleanup.
+$holder->query_safe("COMMIT");
+$holder->quit;
+
+$node->stop;
+done_testing();
diff --git a/contrib/dblink/.gitignore b/contrib/dblink/.gitignore
index 5dcb3ff97235..33bc8d8ffeb3 100644
--- a/contrib/dblink/.gitignore
+++ b/contrib/dblink/.gitignore
@@ -2,3 +2,5 @@
/log/
/results/
/tmp_check/
+/output_iso/
+/tmp_check_iso/
diff --git a/contrib/dblink/Makefile b/contrib/dblink/Makefile
index fde0b49ddbbd..a0ba41280813 100644
--- a/contrib/dblink/Makefile
+++ b/contrib/dblink/Makefile
@@ -13,6 +13,8 @@ PGFILEDESC = "dblink - connect to other PostgreSQL databases"
REGRESS = dblink
REGRESS_OPTS = --dlpath=$(top_builddir)/src/test/regress
+ISOLATION = rewrite_stale_xmin
+ISOLATION_OPTS = --load-extension=dblink
TAP_TESTS = 1
ifdef USE_PGXS
diff --git a/contrib/dblink/expected/rewrite_stale_xmin.out b/contrib/dblink/expected/rewrite_stale_xmin.out
new file mode 100644
index 000000000000..2e5a7940ea26
--- /dev/null
+++ b/contrib/dblink/expected/rewrite_stale_xmin.out
@@ -0,0 +1,237 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_cluster s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_cluster:
+ CLUSTER rewrite_test USING rewrite_test_pkey;
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_vacuum_full s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_vacuum_full:
+ VACUUM FULL rewrite_test;
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_repack s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_repack:
+ REPACK rewrite_test;
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
+
+starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only s2_commit s3_vacuum_toast s3_create_index s1_release
+step s1_hold_xmin:
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+
+dblink_connect
+--------------
+OK
+(1 row)
+
+dblink_exec
+-----------
+BEGIN
+(1 row)
+
+dblink_exec
+------------
+CREATE TABLE
+(1 row)
+
+step s2_begin:
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+
+?column?
+--------
+ 1
+(1 row)
+
+step s3_delete:
+ DELETE FROM rewrite_test WHERE id = 1;
+
+step s3_vacuum_main_only:
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+
+step s2_commit: COMMIT;
+step s3_vacuum_toast:
+ VACUUM pg_toast.rewrite_test_toast;
+
+step s3_create_index:
+ CREATE INDEX rewrite_test_data_idx ON rewrite_test (data);
+
+step s1_release:
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+
+dblink_exec
+-----------
+COMMIT
+(1 row)
+
+dblink_disconnect
+-----------------
+OK
+(1 row)
+
diff --git a/contrib/dblink/meson.build b/contrib/dblink/meson.build
index e2489f41229f..86fbd372b880 100644
--- a/contrib/dblink/meson.build
+++ b/contrib/dblink/meson.build
@@ -36,6 +36,11 @@ tests += {
],
'regress_args': ['--dlpath', meson.project_build_root() / 'src/test/regress'],
},
+ 'isolation': {
+ 'specs': [
+ 'rewrite_stale_xmin',
+ ],
+ },
'tap': {
'tests': [
't/001_auth_scram.pl',
diff --git a/contrib/dblink/specs/rewrite_stale_xmin.spec b/contrib/dblink/specs/rewrite_stale_xmin.spec
new file mode 100644
index 000000000000..fe88e789f854
--- /dev/null
+++ b/contrib/dblink/specs/rewrite_stale_xmin.spec
@@ -0,0 +1,137 @@
+# Test that heap rewrites work when the rewriting backend's own xmin is
+# held back by a transaction in another database.
+
+setup
+{
+ CREATE DATABASE regress_other_db;
+}
+
+setup
+{
+ CREATE EXTENSION IF NOT EXISTS dblink;
+
+ CREATE TABLE rewrite_test (id int PRIMARY KEY, data text)
+ WITH (autovacuum_enabled = off);
+ ALTER TABLE rewrite_test ALTER COLUMN data SET STORAGE EXTERNAL;
+ INSERT INTO rewrite_test VALUES (1, repeat('x', 2500));
+ INSERT INTO rewrite_test VALUES (2, repeat('y', 2500));
+
+ -- Rename the toast table to a known name so we can VACUUM it directly.
+ SET allow_system_table_mods TO true;
+ DO $$DECLARE r record;
+ BEGIN
+ SELECT INTO r reltoastrelid::regclass::text AS table_name
+ FROM pg_class WHERE oid = 'rewrite_test'::regclass;
+ EXECUTE 'ALTER TABLE ' || r.table_name || ' RENAME TO rewrite_test_toast;';
+ END$$;
+ RESET allow_system_table_mods;
+}
+
+teardown
+{
+ DROP DATABASE regress_other_db;
+}
+
+# Session s1: use dblink to hold a transaction in another database.
+# This must start before the DELETE so its XID is older, ensuring
+# GetSnapshotData() holds back any rewrites.
+session s1
+step s1_hold_xmin
+{
+ SELECT dblink_connect('holder',
+ 'dbname=regress_other_db port=' || current_setting('port'));
+ SELECT dblink_exec('holder', 'BEGIN');
+ SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)');
+}
+step s1_release
+{
+ SELECT dblink_exec('holder', 'COMMIT');
+ SELECT dblink_disconnect('holder');
+}
+
+# Session s2: holds xmin in the SAME database temporarily, so the
+# main-table vacuum preserves the dead tuple as RECENTLY_DEAD.
+session s2
+step s2_begin
+{
+ BEGIN ISOLATION LEVEL REPEATABLE READ;
+ SELECT 1;
+}
+step s2_commit { COMMIT; }
+
+# Session s3: performs DML, split vacuum, and the rewrite.
+session s3
+step s3_delete
+{
+ DELETE FROM rewrite_test WHERE id = 1;
+}
+step s3_vacuum_main_only
+{
+ VACUUM (PROCESS_TOAST false) rewrite_test;
+}
+step s3_vacuum_toast
+{
+ VACUUM pg_toast.rewrite_test_toast;
+}
+step s3_cluster
+{
+ CLUSTER rewrite_test USING rewrite_test_pkey;
+}
+step s3_vacuum_full
+{
+ VACUUM FULL rewrite_test;
+}
+step s3_repack
+{
+ REPACK rewrite_test;
+}
+step s3_create_index
+{
+ CREATE INDEX rewrite_test_data_idx ON rewrite_test (data);
+}
+
+teardown { DROP TABLE IF EXISTS rewrite_test; }
+
+# CLUSTER
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_cluster
+ s1_release
+
+# VACUUM FULL
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_vacuum_full
+ s1_release
+
+# REPACK
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_repack
+ s1_release
+
+# CREATE INDEX
+permutation
+ s1_hold_xmin
+ s2_begin
+ s3_delete
+ s3_vacuum_main_only
+ s2_commit
+ s3_vacuum_toast
+ s3_create_index
+ s1_release
--
2.55.0
[application/pgp-signature] signature.asc (833B, ../../[email protected]/3-signature.asc)
download
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: BUG #19519: REPACK can fail due to missing chunk for toast value
2026-06-14 07:00 BUG #19519: REPACK can fail due to missing chunk for toast value PG Bug reporting form <[email protected]>
2026-06-17 17:03 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-03 06:45 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Imran Zaheer <[email protected]>
2026-07-06 06:36 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 18:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
2026-07-06 23:18 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 23:51 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
2026-07-07 01:42 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-07 05:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-07 17:10 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-07 22:15 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-08 01:10 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-08 12:00 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Heikki Linnakangas <[email protected]>
@ 2026-07-09 05:17 ` Konstantin Knizhnik <[email protected]>
2026-07-09 05:43 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2 siblings, 1 reply; 21+ messages in thread
From: Konstantin Knizhnik @ 2026-07-09 05:17 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; Michael Paquier <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; +Cc: Matthias van de Meent <[email protected]>; Imran Zaheer <[email protected]>; Alexander Lakhin <[email protected]>; PostgreSQL mailing lists <[email protected]>
On 08/07/2026 3:00 PM, Heikki Linnakangas wrote:
> On 08/07/2026 04:10, Michael Paquier wrote:
>> On Wed, Jul 08, 2026 at 07:15:16AM +0900, Michael Paquier wrote:
>>> One thing that I still don't like much in the patch as written is my
>>> use of a missing_ok argument, which feels super confusing as it
>>> applies only to the underlying external TOAST, if the relation has
>>> any. I'd be tempted to rewrite this portion of the patch with a
>>> uint32 flags. Even if we assign one value for now (say MISSING_TOAST,
>>> MISSING_TOAST_OK or whatever), it would allow more flexibility on ABI
>>> grounds if we want more like states in the future across this portion
>>> of the stack. Again, I strongly doubt that we will be able to
>>> backpatch any of that.
>
> Yeah, boolean arguments in general can be confusing. Especially
> arguments like "missing_ok" where it's hard to remember which behavior
> is true and which is false. An IDE that shows the name of the argument
> helps, but having the caller say "MISSING_TOAST_OK" rather than "true"
> is much clearer.
>
> For backpatching, I think we could smuggle the flag in a global
> variable, to avoid changing the signature of the
> relation_fetch_toast_slice() callback. See attached patch 0002 on top
> of your 0001 patch (which is also attached for completeness). It
> doesn't fix the problem for hypothetical 3rd party tableam
> implementations that implement their own relation_fetch_toast_slice()
> callback. But do such extensions even exist? If yes, they could be
> fixed too by also checking the global variable.
>
> I thought about adding an extra "extended" callback next to
> relation_fetch_toast_slice(), but I don't see a way to add functions
> to TableAmRoutine in an ABI-compatible way. For the future, we might
> want to store sizeof(TableAmRoutine) in the struct itself, so that we
> could add fields to it in minor versions without breaking the API, in
> case we need something like this again.
Changing `table_relation_fetch_toast_slice` signature breaks
compatibility with some extensions, for example duckdb.
I wonder if we can change only `bool (*relation_fetch_toast_slice)(...,
bool missing_ok)` callback signature, but preserve signature of
`table_relation_fetch_toast_slice`, adding one more function
`table_relation_try_fetch_toast_slice`:
static inline void
table_relation_fetch_toast_slice(Relation toastrel, Oid valueid,
int32 attrsize,
int32 sliceoffset,
int32
slicelength, varlena *result,
bool missing_ok)
{
(void)toastrel->rd_tableam->relation_fetch_toast_slice(toastrel,
valueid,
attrsize,
sliceoffset, slicelength,
result, false);
}
static inline bool
table_relation_try_fetch_toast_slice(Relation toastrel, Oid valueid,
int32 attrsize,
int32 sliceoffset,
int32
slicelength, varlena *result,
bool missing_ok)
{
return toastrel->rd_tableam->relation_fetch_toast_slice(toastrel,
valueid,
attrsize,
sliceoffset, slicelength,
result, missing_ok);
}
>
>> A couple of extra notes while I do not forget about that stuff.. The
>> patch may be better split into two if we go with this approach, as the
>> index build and rewrite paths require different solutions:
>> - One for the rewrite path. It makes little sense to do any kind of
>> aggressive early detoasting because it may be wasteful due to the
>> tuple rewrites that update their data with only copies by reference
>> (main relation tuple is rewritten, reuses the same external TOAST
>> tuple). For workloads where UPDATEs do not touch the TOASTed
>> attributes, that would be a waste.
>
> I didn't understand this part. Do you mean when rebuilding the indexes
> after rewriting the heap? I didn't think we support storing toasted
> external datums in an index at all.
>
>> - One for the index build path, which is actually too aggressive with
>> its early detoasting, now that I think about it. There should be no
>> need to perform a detoast for anything else than the attributes that
>> are used in the index definition or the attributes that are used in
>> index expressions. So as written this patch would lead to a
>> regression.
> Yeah, although I wouldn't be too worried about the performance here.
> The penalty would be when building an index on values that are large
> enough to be toasted, with lots of RECENTLY_DEAD tuples. That doesn't
> seem like a very common case.
>
> - Heikki
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: BUG #19519: REPACK can fail due to missing chunk for toast value
2026-06-14 07:00 BUG #19519: REPACK can fail due to missing chunk for toast value PG Bug reporting form <[email protected]>
2026-06-17 17:03 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-03 06:45 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Imran Zaheer <[email protected]>
2026-07-06 06:36 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 18:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
2026-07-06 23:18 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 23:51 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
2026-07-07 01:42 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-07 05:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-07 17:10 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-07 22:15 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-08 01:10 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-08 12:00 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Heikki Linnakangas <[email protected]>
2026-07-09 05:17 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Konstantin Knizhnik <[email protected]>
@ 2026-07-09 05:43 ` Michael Paquier <[email protected]>
2026-07-09 10:08 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Konstantin Knizhnik <[email protected]>
0 siblings, 1 reply; 21+ messages in thread
From: Michael Paquier @ 2026-07-09 05:43 UTC (permalink / raw)
To: Konstantin Knizhnik <[email protected]>; +Cc: Heikki Linnakangas <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Imran Zaheer <[email protected]>; Alexander Lakhin <[email protected]>; PostgreSQL mailing lists <[email protected]>
On Thu, Jul 09, 2026 at 08:17:06AM +0300, Konstantin Knizhnik wrote:
> Changing `table_relation_fetch_toast_slice` signature breaks compatibility
> with some extensions, for example duckdb.
> I wonder if we can change only `bool (*relation_fetch_toast_slice)(...,
> bool missing_ok)` callback signature, but preserve signature of
> `table_relation_fetch_toast_slice`, adding one more function
> `table_relation_try_fetch_toast_slice`:
Thanks for the input.
Adding more functions implies to change the size of TableAmRoutine,
which may be bad. For now I'd be tempted to just focus on how to fix
the problem on HEAD in a way we are happy with. We could sort out an
optional back-branch version as a second step.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 21+ messages in thread
* Re: BUG #19519: REPACK can fail due to missing chunk for toast value
2026-06-14 07:00 BUG #19519: REPACK can fail due to missing chunk for toast value PG Bug reporting form <[email protected]>
2026-06-17 17:03 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-03 06:45 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Imran Zaheer <[email protected]>
2026-07-06 06:36 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 18:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
2026-07-06 23:18 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-06 23:51 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Matthias van de Meent <[email protected]>
2026-07-07 01:42 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-07 05:21 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-07 17:10 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Srinath Reddy Sadipiralla <[email protected]>
2026-07-07 22:15 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-08 01:10 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
2026-07-08 12:00 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Heikki Linnakangas <[email protected]>
2026-07-09 05:17 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Konstantin Knizhnik <[email protected]>
2026-07-09 05:43 ` Re: BUG #19519: REPACK can fail due to missing chunk for toast value Michael Paquier <[email protected]>
@ 2026-07-09 10:08 ` Konstantin Knizhnik <[email protected]>
0 siblings, 0 replies; 21+ messages in thread
From: Konstantin Knizhnik @ 2026-07-09 10:08 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Heikki Linnakangas <[email protected]>; Srinath Reddy Sadipiralla <[email protected]>; Matthias van de Meent <[email protected]>; Imran Zaheer <[email protected]>; Alexander Lakhin <[email protected]>; PostgreSQL mailing lists <[email protected]>
On 09/07/2026 8:43 AM, Michael Paquier wrote:
> On Thu, Jul 09, 2026 at 08:17:06AM +0300, Konstantin Knizhnik wrote:
>> Changing `table_relation_fetch_toast_slice` signature breaks compatibility
>> with some extensions, for example duckdb.
>> I wonder if we can change only `bool (*relation_fetch_toast_slice)(...,
>> bool missing_ok)` callback signature, but preserve signature of
>> `table_relation_fetch_toast_slice`, adding one more function
>> `table_relation_try_fetch_toast_slice`:
> Thanks for the input.
>
> Adding more functions implies to change the size of TableAmRoutine,
> which may be bad. For now I'd be tempted to just focus on how to fix
> the problem on HEAD in a way we are happy with. We could sort out an
> optional back-branch version as a second step.
> --
> Michael
Sorry, I do not suggest to extend TableAmRoutine - it will contain
single `relation_fetch_toast_slice` entry with new signature.
I suggest to add only new wrapper function
`table_relation_try_fetch_toast_slice`, preserving original
`table_relation_fetch_toast_slice`. It will preserve compatibility with
extensions (there are not using TableAmRoutine directly) and do not
extend TableAmRoutine.
^ permalink raw reply [nested|flat] 21+ messages in thread
end of thread, other threads:[~2026-07-09 10:08 UTC | newest]
Thread overview: 21+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-06-14 07:00 BUG #19519: REPACK can fail due to missing chunk for toast value PG Bug reporting form <[email protected]>
2026-06-17 17:03 ` Srinath Reddy Sadipiralla <[email protected]>
2026-07-03 06:45 ` Imran Zaheer <[email protected]>
2026-07-06 06:36 ` Michael Paquier <[email protected]>
2026-07-06 13:38 ` Konstantin Knizhnik <[email protected]>
2026-07-06 18:00 ` Alexander Lakhin <[email protected]>
2026-07-06 18:21 ` Matthias van de Meent <[email protected]>
2026-07-06 23:18 ` Michael Paquier <[email protected]>
2026-07-06 23:51 ` Matthias van de Meent <[email protected]>
2026-07-07 01:42 ` Michael Paquier <[email protected]>
2026-07-07 05:21 ` Michael Paquier <[email protected]>
2026-07-07 17:10 ` Srinath Reddy Sadipiralla <[email protected]>
2026-07-07 22:15 ` Michael Paquier <[email protected]>
2026-07-08 01:10 ` Michael Paquier <[email protected]>
2026-07-08 12:00 ` Heikki Linnakangas <[email protected]>
2026-07-08 15:00 ` Imran Zaheer <[email protected]>
2026-07-08 22:49 ` Michael Paquier <[email protected]>
2026-07-09 02:47 ` Michael Paquier <[email protected]>
2026-07-09 05:17 ` Konstantin Knizhnik <[email protected]>
2026-07-09 05:43 ` Michael Paquier <[email protected]>
2026-07-09 10:08 ` Konstantin Knizhnik <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox