Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1wgcwm-006Ido-2Z for pgsql-bugs@arkaria.postgresql.org; Mon, 06 Jul 2026 06:36:53 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.96) (envelope-from ) id 1wgcwl-00FPG1-1T for pgsql-bugs@arkaria.postgresql.org; Mon, 06 Jul 2026 06:36:51 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1wgcwk-00FPFt-2D for pgsql-bugs@lists.postgresql.org; Mon, 06 Jul 2026 06:36:51 +0000 Received: from fout-b6-smtp.messagingengine.com ([202.12.124.149]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1wgcwh-00000001lhX-1FUr for pgsql-bugs@lists.postgresql.org; Mon, 06 Jul 2026 06:36:49 +0000 Received: from phl-compute-05.internal (phl-compute-05.internal [10.202.2.45]) by mailfout.stl.internal (Postfix) with ESMTP id 6B62A1D0006D; Mon, 6 Jul 2026 02:36:46 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-05.internal (MEProxy); Mon, 06 Jul 2026 02:36:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=paquier.xyz; h= cc:cc:content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1783319806; x=1783406206; bh=HDNiG1YDeM RX0JVIuoGtH8ZtJiFGjoiWsobhGV/oJqs=; b=WT8gSEtwO0DyMAJyC7FekZOcCV naUC5+FyYmElMyd1GdOhFde2W0o8f6F1Zm7E9pF8lDmzpUAr7ANUvUL32ndcG3h5 hy9+EkkA+zdIP65lwNhVQ5fmwF7ympS6331JQBK+ceg2Mad7A2xj4qD65wcon+Wd kUItGYAagGXqbgFeLg2bZqbA12LnEWBilX1nmTu4czeuYREgcDjRoNm+zycXwBIh z4IlXGCYAq7W4d2zDyL0zEk9bjUDRYs2DFkBSRqAabwRh5QnU2vYXiRSsxw0bXYC AELqHu1a6Dj00SeafODqbznVwAP2nGaWcPLwDcRArC/mvJvUhcgEhj9Bj1Tw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1783319806; x=1783406206; bh=HDNiG1YDeMRX0JVIuoGtH8ZtJiFGjoiWsob hGV/oJqs=; b=G9PcD7cishBRI0pw/ftG+BN4le9wLHcYM1OcWOgTmvjlTC7u22q G1rHVUZLsfSpWGZ6GCryVOrmqrjDEUcWduEzPe0v70h5DUzhu8QtpyVXZUugZQqw ekzZ74Oafx5nxPTV56TVp4N6uIpxPdA4YbcrVWSoV19eVw/dhBJIIAhMRc602XzA ZnHUIcWMy/waGtOdVoMK9eZIP6g6trAuDETpnRoOe+EdxPQt3W6DAVMjymj9r1ud 4NIrPiQIASxb6RlbLg8ANFsTQwN2vfJh2kSwilqLTLrU9Ub7996wmgKNHtPbjYTL hKkOilIZxkJZ68hyVuzKu1rATPjfCPVQQAQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTGLGPowgYme58Gswy/9Q7/0loMzPAx171qh5y9agXYfGIC/nFg4FTON9NkuUd1NgK 3TXeZ/J23y8d88TzlVEVsGX5fLKWx2taX/NmterrK+A1ppw3NQQV2woKoYtJB62Krm5OIo a9Z+iT8MuX503QP8pPI/3U6qmUjvYQFbvJKm0a7dp7VIsGRhOq3iFGFify5MGCNXOAprs7 qfAuVi/MmlRvmErbHxTf1EaYXvssQG1Z1jxxciqpPkHN+kLKEeJ7bkvKxv1UVU6ZIUz3tu tNNeQJARg+gsimlSmaX44sLnSUfaEf1ChVSlX6GX3f88ijB+TjhZON7AjzVTdTM/OBYTcL 3hOk47sX6I3squMyasXqXX3AM05o1f+3P518BP/2HQReEBVRhorXwFgStFIFkd77HqWR/v iG2BbzBBQ3WyfAlgp8JcCt4UMw2EyVJdbB01Lb27q3paHaMh0Br+lBt5pVWtFH1EGRDbvw cXnr6Ncs2BvSnRtVHdOnbuVduFwPvsr120faG2niYzDTDVlkfdDsuvlRBjAfMWniOpbT2z rv+HIh76TWL6vAsSkl9f/Zjnj2VgW0KRG0+k3SHPsQtZbHlC/luCYTOw4PMHGHGhBZKUTM CLtLskA+nILsrv62XZk2oCkyi3rFkIGx3RUrYzCCTWeX9gGb/QunofZOIqbA X-ME-Proxy: Feedback-ID: i0fe9450f:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 6 Jul 2026 02:36:44 -0400 (EDT) Date: Mon, 6 Jul 2026 15:36:39 +0900 From: Michael Paquier To: Imran Zaheer Cc: Srinath Reddy Sadipiralla , exclusion@gmail.com, pgsql-bugs@lists.postgresql.org Subject: Re: BUG #19519: REPACK can fail due to missing chunk for toast value Message-ID: References: <19519-fe02d8ff679d834d@postgresql.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="M7OpH/Q9yPhJPxKO" Content-Disposition: inline In-Reply-To: List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --M7OpH/Q9yPhJPxKO Content-Type: multipart/mixed; boundary="lnLgQFmtffG/DLE8" Content-Disposition: inline --lnLgQFmtffG/DLE8 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jul 03, 2026 at 11:45:42AM +0500, Imran Zaheer wrote: > Can we safely use the same fix in the index build path too? Can we use > GetOldestNonRemovableTransactionIdforRewrite or something similar > here? Normal serial index builds use SnapShotAny and concurrent index > builds use MVCC, but the bug only exists in the serial index build > path. Yeah, I think that we should be able to safely reuse your ForRewrite() API for the index build case (SnapshotAny part) as well to make the horizon computations more conservative, exclusing one own XID as we do in the lazy VACUUM case. > Other than that, after applying your patch, the bug was not > reproducible with either this repro or the other report's repro [1] in > the rewrite path. However, the create index bug is still there. You > can use the following repro as mentioned in the thread [1]. The assymetry between the global xmin computation and the per-database horizon is what's killing us here. All these patterns are complicated enough that they warrant some tests, even if it is necessary to have multiple databases to trigger the buggy horizon computations. TAP tests would be an obvious choice for that. Now I have a set of tricks in my sleeves to make an isolation test fully deterministic: - dblink() that opens a transaction to a different database than the one of the isolation regression database. - Rename of a TOAST table using allow_system_table_mods, for a VACUUM. The second one is one of the dirtiest tricks I've done in the past for a REINDEX CONCURRENTLY test with TOAST. Quite useful. If the backend side change is reverted, the tests fail. The attached includes both test suites for all four cases reported (REPACK, VACUUM, CLUSTER, non-MVCC index builds). We'll need to remove one of them, just keeping both posted as I am not sure if the isolation tests would be entirely stable in the buildfarm.. I still need to dive more into the code, but for now this bit stands out: vacuum_get_cutoffs(OldHeap, ¶ms, &cutoffs); =20 + /* + * vacuum_get_cutoffs() folds our own backend's xmin into OldestXmin. = For + * a rewrite that is too conservative: the snapshot we hold exists onl= y to + * evaluate index expressions against other relations, not to read + * OldHeap's historical rows. If our xmin is held back by a transacti= on + * that cannot even see OldHeap (e.g. one in another database), we wou= ld + * preserve a recently-dead tuple whose TOAST chunks a concurrent or p= rior + * lazy vacuum was free to remove, and then fail with "missing chunk" = while + * copying it. Recompute OldestXmin ignoring our own backend so it ma= tches + * the horizon lazy vacuum uses. This can only move OldestXmin forwar= d, so + * the freeze cutoffs derived above remain valid. + */ + cutoffs.OldestXmin =3D GetOldestNonRemovableTransactionIdForRewrite(Ol= dHeap); In copy_table_data() (for the data copy with repack, cluster, vacuum full), I think that this is incorrect. For one, this breaks the FreezeLimit which should always be older than OldestXmin, and this patch enforces a new recomputation of OldestXmin ignoring most of the internals of vacuum_get_cutoffs(). That's brittle, to say the least if we change the way the vacuum cutoffs are calculated in the future. Thoughts and comments from others are welcome for now. My day is almost out, at least I got all these scenarios working some tests. -- Michael --lnLgQFmtffG/DLE8 Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=v2-0001-Fix-missing-chunk-errors-during-heap-rewrites-and.patch Content-Transfer-Encoding: quoted-printable =46rom 36fdec282e5c48866bf915f767502ccc3c6bc0b2 Mon Sep 17 00:00:00 2001 =46rom: Michael Paquier Date: Mon, 6 Jul 2026 15:27:54 +0900 Subject: [PATCH v2] Fix "missing chunk" errors during heap rewrites and ind= ex builds When performing a heap rewrite (CLUSTER, VACUUM FULL, REPACK) or a non-concurrent index build, the backend holds an MVCC snapshot whose xmin can be held back by a transaction in another database, since GetSnapshotData() considers XIDs from all databases. However, ComputeXidHorizons() only considers same-database backends for data_oldest_nonremovable. This asymmetry means a concurrent VACUUM can compute a more aggressive horizon and remove TOAST chunks for a tuple that the rewrite/index-build backend still considers as recently dead, leading to "missing chunk number 0 for toast value" errors when attempting to copy or index a tuple. This issue is fixed by introducing a ForRewrite() equivalent of GetOldestNonRemovableTransactionId(). It calls ComputeXidHorizons() with a new option called "excludeMyself", mirroring how PROC_IN_VACUUM excludes a lazy VACUUM own's XID to make rewrites more conservative with the computation of the XID horizon. This problem has been reported recently by Alexander for REPACK, but it is a much older issue (tested down to 9.5 for curiosity's sake, but I suspect that it is much older than that). Srinath has authored the C portion of the patch, while I got some tests working for the four problematic cases reported. XXX: This patch includes both TAP and isolation tests. We should remove one pattern at the end. Reported-by: Alexander Lakhin Author: Srinath Reddy Sadipiralla Co-authored-by: Michael Paquier Reviewed-by: Imran Zaheer Discussion: https://postgr.es/m/18351-f6e06364b3a2e669@postgresql.org Discussion: https://postgr.es/m/19519-fe02d8ff679d834d@postgresql.org Backpatch-through: 14 --- src/include/storage/procarray.h | 1 + src/backend/access/heap/heapam_handler.c | 2 +- src/backend/commands/repack.c | 15 ++ src/backend/storage/ipc/procarray.c | 78 +++++- src/test/recovery/meson.build | 1 + src/test/recovery/t/055_rewrite_stale_xmin.pl | 124 +++++++++ contrib/dblink/.gitignore | 2 + contrib/dblink/Makefile | 2 + .../dblink/expected/rewrite_stale_xmin.out | 237 ++++++++++++++++++ contrib/dblink/meson.build | 5 + contrib/dblink/specs/rewrite_stale_xmin.spec | 137 ++++++++++ 11 files changed, 596 insertions(+), 8 deletions(-) create mode 100644 src/test/recovery/t/055_rewrite_stale_xmin.pl create mode 100644 contrib/dblink/expected/rewrite_stale_xmin.out create mode 100644 contrib/dblink/specs/rewrite_stale_xmin.spec diff --git a/src/include/storage/procarray.h b/src/include/storage/procarra= y.h index d718a5b542f0..ccfc31cdb5d1 100644 --- a/src/include/storage/procarray.h +++ b/src/include/storage/procarray.h @@ -51,6 +51,7 @@ extern RunningTransactions GetRunningTransactionData(void= ); =20 extern bool TransactionIdIsInProgress(TransactionId xid); extern TransactionId GetOldestNonRemovableTransactionId(Relation rel); +extern TransactionId GetOldestNonRemovableTransactionIdForRewrite(Relation= rel); extern TransactionId GetOldestTransactionIdConsideredRunning(void); extern TransactionId GetOldestActiveTransactionId(bool inCommitOnly, bool allDbs); diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/= heap/heapam_handler.c index 2268cc277bce..fb1540784236 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -1213,7 +1213,7 @@ heapam_index_build_range_scan(Relation heapRelation, =20 /* okay to ignore lazy VACUUMs here */ if (!IsBootstrapProcessingMode() && !indexInfo->ii_Concurrent) - OldestXmin =3D GetOldestNonRemovableTransactionId(heapRelation); + OldestXmin =3D GetOldestNonRemovableTransactionIdForRewrite(heapRelation= ); =20 if (!scan) { diff --git a/src/backend/commands/repack.c b/src/backend/commands/repack.c index faa07d1a118b..1585d67f0b34 100644 --- a/src/backend/commands/repack.c +++ b/src/backend/commands/repack.c @@ -71,6 +71,7 @@ #include "storage/lmgr.h" #include "storage/predicate.h" #include "storage/proc.h" +#include "storage/procarray.h" #include "utils/acl.h" #include "utils/fmgroids.h" #include "utils/guc.h" @@ -1390,6 +1391,20 @@ copy_table_data(Relation NewHeap, Relation OldHeap, = Relation OldIndex, memset(¶ms, 0, sizeof(VacuumParams)); vacuum_get_cutoffs(OldHeap, ¶ms, &cutoffs); =20 + /* + * vacuum_get_cutoffs() folds our own backend's xmin into OldestXmin. For + * a rewrite that is too conservative: the snapshot we hold exists only to + * evaluate index expressions against other relations, not to read + * OldHeap's historical rows. If our xmin is held back by a transaction + * that cannot even see OldHeap (e.g. one in another database), we would + * preserve a recently-dead tuple whose TOAST chunks a concurrent or prior + * lazy vacuum was free to remove, and then fail with "missing chunk" + * while copying it. Recompute OldestXmin ignoring our own backend so it + * matches the horizon lazy vacuum uses. This can only move OldestXmin + * forward, so the freeze cutoffs derived above remain valid. + */ + cutoffs.OldestXmin =3D GetOldestNonRemovableTransactionIdForRewrite(OldHe= ap); + /* * FreezeXid will become the table's new relfrozenxid, and that mustn't go * backwards, so take the max. diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/= procarray.c index 60336b318036..a11c01eb9d42 100644 --- a/src/backend/storage/ipc/procarray.c +++ b/src/backend/storage/ipc/procarray.c @@ -1671,7 +1671,7 @@ TransactionIdIsInProgress(TransactionId xid) * code doesn't expect (breaking HOT). */ static void -ComputeXidHorizons(ComputeXidHorizonsResult *h) +ComputeXidHorizons(ComputeXidHorizonsResult *h, bool excludeMyself) { ProcArrayStruct *arrayP =3D procArray; TransactionId kaxmin; @@ -1770,6 +1770,22 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h) if (statusFlags & (PROC_IN_VACUUM | PROC_IN_LOGICAL_DECODING)) continue; =20 + /* + * Optionally ignore our own backend's xmin. A relation rewrite holds + * an MVCC snapshot only so that index expressions can be evaluated + * against other relations; it never reads the historical rows of the + * relation being rewritten through that snapshot. Letting our own + * xmin hold back that relation's removal horizon makes the rewrite + * more conservative than the lazy vacuum that may already have + * removed a recently-dead tuple's TOAST chunks, which would then fail + * with "missing chunk" while copying the tuple. This mirrors how + * PROC_IN_VACUUM excludes a lazy vacuum's own xmin. + * + * See also GetOldestNonRemovableTransactionIdForRewrite(). + */ + if (excludeMyself && proc =3D=3D MyProc) + continue; + /* shared tables need to take backends in all databases into account */ h->shared_oldest_nonremovable =3D TransactionIdOlder(h->shared_oldest_nonremovable, xmin); @@ -1898,8 +1914,15 @@ ComputeXidHorizons(ComputeXidHorizonsResult *h) TransactionIdPrecedesOrEquals(h->oldest_considered_running, h->slot_catalog_xmin)); =20 - /* update approximate horizons with the computed horizons */ - GlobalVisUpdateApply(h); + /* + * Update approximate horizons with the computed horizons. Skip this when + * excluding our own xmin: the resulting horizons are more aggressive than + * what is globally safe and are only valid for the caller's own rewrite + * of a single relation, so they must not be published to the shared + * GlobalVisState. + */ + if (!excludeMyself) + GlobalVisUpdateApply(h); } =20 /* @@ -1945,7 +1968,48 @@ GetOldestNonRemovableTransactionId(Relation rel) { ComputeXidHorizonsResult horizons; =20 - ComputeXidHorizons(&horizons); + ComputeXidHorizons(&horizons, false); + + switch (GlobalVisHorizonKindForRel(rel)) + { + case VISHORIZON_SHARED: + return horizons.shared_oldest_nonremovable; + case VISHORIZON_CATALOG: + return horizons.catalog_oldest_nonremovable; + case VISHORIZON_DATA: + return horizons.data_oldest_nonremovable; + case VISHORIZON_TEMP: + return horizons.temp_oldest_nonremovable; + } + + /* just to prevent compiler warnings */ + return InvalidTransactionId; +} + +/* + * GetOldestNonRemovableTransactionIdForRewrite -- variant of + * GetOldestNonRemovableTransactionId() that ignores the calling backend's + * own xmin. + * + * A rewrite must preserve recently-dead tuples of the relation being + * rewritten so that other backends' snapshots still see them afterwards. + * + * It is not, however, a reader of that relation's + * historical rows itself: the MVCC snapshot it holds exists only to evalu= ate + * index expressions against *other* relations. Excluding our own xmin yi= elds + * the same horizon a lazy vacuum would use (PROC_IN_VACUUM does the equiv= alent + * for lazy vacuum). That matters for TOAST relations to prevent the remo= val of + * recently-dead tuple's TOAST chunks. + * + * The caller must hold a lock on rel strong enough that the set of backen= ds + * that could read its rows cannot change underneath us. + */ +TransactionId +GetOldestNonRemovableTransactionIdForRewrite(Relation rel) +{ + ComputeXidHorizonsResult horizons; + + ComputeXidHorizons(&horizons, true); =20 switch (GlobalVisHorizonKindForRel(rel)) { @@ -1974,7 +2038,7 @@ GetOldestTransactionIdConsideredRunning(void) { ComputeXidHorizonsResult horizons; =20 - ComputeXidHorizons(&horizons); + ComputeXidHorizons(&horizons, false); =20 return horizons.oldest_considered_running; } @@ -1987,7 +2051,7 @@ GetReplicationHorizons(TransactionId *xmin, Transacti= onId *catalog_xmin) { ComputeXidHorizonsResult horizons; =20 - ComputeXidHorizons(&horizons); + ComputeXidHorizons(&horizons, false); =20 /* * Don't want to use shared_oldest_nonremovable here, as that contains the @@ -4214,7 +4278,7 @@ GlobalVisUpdate(void) ComputeXidHorizonsResult horizons; =20 /* updates the horizons as a side-effect */ - ComputeXidHorizons(&horizons); + ComputeXidHorizons(&horizons, false); } =20 /* diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build index ad0d85f41897..7e583ee39a10 100644 --- a/src/test/recovery/meson.build +++ b/src/test/recovery/meson.build @@ -63,6 +63,7 @@ tests +=3D { 't/052_checkpoint_segment_missing.pl', 't/053_standby_login_event_trigger.pl', 't/054_unlogged_sequence_promotion.pl', + 't/055_rewrite_stale_xmin.pl', ], }, } diff --git a/src/test/recovery/t/055_rewrite_stale_xmin.pl b/src/test/recov= ery/t/055_rewrite_stale_xmin.pl new file mode 100644 index 000000000000..dad95c0cf967 --- /dev/null +++ b/src/test/recovery/t/055_rewrite_stale_xmin.pl @@ -0,0 +1,124 @@ +# Copyright (c) 2025-2026, PostgreSQL Global Development Group +# +# Test that heap rewrites (CLUSTER, VACUUM FULL, REPACK) and index builds +# work in the case when the backend's own xmin is held back by a transacti= on +# in another database. + +use strict; +use warnings; +use PostgreSQL::Test::Cluster; +use Test::More; + +my $node =3D PostgreSQL::Test::Cluster->new('primary'); +$node->init; +# No Autovacuum, no random interferences. +$node->append_conf( + 'postgresql.conf', qq[ +autovacuum =3D off +]); +$node->start; + +# Create two databases. +$node->safe_psql('postgres', 'CREATE DATABASE db_holder'); +$node->safe_psql('postgres', 'CREATE DATABASE db_target'); + +# Set up a table with TOAST data. Returns the toast table name. +sub setup_table +{ + my ($tblname, $extra_ddl) =3D @_; + $extra_ddl //=3D ''; + + $node->safe_psql( + 'db_target', qq{ + DROP TABLE IF EXISTS $tblname; + CREATE TABLE $tblname (id int PRIMARY KEY, data text) + WITH (autovacuum_enabled =3D off); + ALTER TABLE $tblname ALTER COLUMN data SET STORAGE EXTERNAL; + INSERT INTO $tblname VALUES (1, repeat('x', 2500)); + INSERT INTO $tblname VALUES (2, repeat('z', 2500)); + $extra_ddl + }); + + my $toast =3D $node->safe_psql( + 'db_target', qq{ + SELECT 'pg_toast.' || relname FROM pg_class + WHERE oid =3D (SELECT reltoastrelid FROM pg_class + WHERE relname =3D '$tblname'); + }); + chomp $toast; + return $toast; +} + +# Create precondition state. Requires that $holder is already running with +# a write XID. +sub create_missing_toast_state +{ + my ($tblname, $toast_table) =3D @_; + + # Hold xmin in same DB to prevent main tuple removal. + my $xmin_holder =3D $node->background_psql('db_target', on_error_stop =3D= > 1); + $xmin_holder->query_safe("BEGIN ISOLATION LEVEL REPEATABLE READ"); + $xmin_holder->query_safe("SELECT 1"); + + # Delete the row with TOAST data. + $node->safe_psql('db_target', "DELETE FROM $tblname WHERE id =3D 1"); + + # VACUUM main table only. The xmin_holder keeps the dead tuple. + $node->safe_psql('db_target', "VACUUM (PROCESS_TOAST false) $tblname"); + + # Release same-DB xmin. + $xmin_holder->query_safe("COMMIT"); + $xmin_holder->quit; + + # Vacuum toast table: toast chunks now DEAD and removed. + $node->safe_psql('db_target', "VACUUM $toast_table"); +} + +# Open a transaction holding an XID before any deletes. This XID will hold +# back GetSnapshotData() globally, on a different database than the one +# running the various rewrite operations. +my $holder =3D $node->background_psql('db_holder', on_error_stop =3D> 1); +$holder->query_safe("BEGIN"); +$holder->query_safe("CREATE TEMP TABLE xid_holder(x int)"); + +# Test 1: CLUSTER +my $toast =3D setup_table('rewrite_test'); +create_missing_toast_state('rewrite_test', $toast); + +my ($ret, $stdout, $stderr) =3D + $node->psql('db_target', 'CLUSTER rewrite_test USING rewrite_test_pkey'); +is($ret, 0, "CLUSTER succeeds without missing-chunk error"); +unlike($stderr, qr/missing chunk/, "CLUSTER: no missing-chunk error"); + +# Test 2: VACUUM FULL +$toast =3D setup_table('rewrite_test'); +create_missing_toast_state('rewrite_test', $toast); + +($ret, $stdout, $stderr) =3D + $node->psql('db_target', 'VACUUM FULL rewrite_test'); +is($ret, 0, "VACUUM FULL succeeds without missing-chunk error"); +unlike($stderr, qr/missing chunk/, "VACUUM FULL: no missing-chunk error"); + +# Test 3: REPACK +$toast =3D setup_table('rewrite_test'); +create_missing_toast_state('rewrite_test', $toast); + +($ret, $stdout, $stderr) =3D $node->psql('db_target', 'REPACK rewrite_test= '); +is($ret, 0, "REPACK succeeds without missing-chunk error"); +unlike($stderr, qr/missing chunk/, "REPACK: no missing-chunk error"); + +# Test 4: CREATE INDEX on toasted column +$toast =3D setup_table('rewrite_test'); +create_missing_toast_state('rewrite_test', $toast); + +($ret, $stdout, $stderr) =3D $node->psql('db_target', + 'CREATE INDEX rewrite_test_data_idx ON rewrite_test (data)'); +is($ret, 0, "CREATE INDEX succeeds without missing-chunk error"); +unlike($stderr, qr/missing chunk/, "CREATE INDEX: no missing-chunk error"); + +# Cleanup. +$holder->query_safe("COMMIT"); +$holder->quit; + +$node->stop; +done_testing(); diff --git a/contrib/dblink/.gitignore b/contrib/dblink/.gitignore index 5dcb3ff97235..33bc8d8ffeb3 100644 --- a/contrib/dblink/.gitignore +++ b/contrib/dblink/.gitignore @@ -2,3 +2,5 @@ /log/ /results/ /tmp_check/ +/output_iso/ +/tmp_check_iso/ diff --git a/contrib/dblink/Makefile b/contrib/dblink/Makefile index fde0b49ddbbd..a0ba41280813 100644 --- a/contrib/dblink/Makefile +++ b/contrib/dblink/Makefile @@ -13,6 +13,8 @@ PGFILEDESC =3D "dblink - connect to other PostgreSQL data= bases" =20 REGRESS =3D dblink REGRESS_OPTS =3D --dlpath=3D$(top_builddir)/src/test/regress +ISOLATION =3D rewrite_stale_xmin +ISOLATION_OPTS =3D --load-extension=3Ddblink TAP_TESTS =3D 1 =20 ifdef USE_PGXS diff --git a/contrib/dblink/expected/rewrite_stale_xmin.out b/contrib/dblin= k/expected/rewrite_stale_xmin.out new file mode 100644 index 000000000000..2e5a7940ea26 --- /dev/null +++ b/contrib/dblink/expected/rewrite_stale_xmin.out @@ -0,0 +1,237 @@ +Parsed test spec with 3 sessions + +starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only = s2_commit s3_vacuum_toast s3_cluster s1_release +step s1_hold_xmin:=20 + SELECT dblink_connect('holder', + 'dbname=3Dregress_other_db port=3D' || current_setting('port')); + SELECT dblink_exec('holder', 'BEGIN'); + SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)'); + +dblink_connect +-------------- +OK =20 +(1 row) + +dblink_exec +----------- +BEGIN =20 +(1 row) + +dblink_exec=20 +------------ +CREATE TABLE +(1 row) + +step s2_begin:=20 + BEGIN ISOLATION LEVEL REPEATABLE READ; + SELECT 1; + +?column? +-------- + 1 +(1 row) + +step s3_delete:=20 + DELETE FROM rewrite_test WHERE id =3D 1; + +step s3_vacuum_main_only:=20 + VACUUM (PROCESS_TOAST false) rewrite_test; + +step s2_commit: COMMIT; +step s3_vacuum_toast:=20 + VACUUM pg_toast.rewrite_test_toast; + +step s3_cluster:=20 + CLUSTER rewrite_test USING rewrite_test_pkey; + +step s1_release:=20 + SELECT dblink_exec('holder', 'COMMIT'); + SELECT dblink_disconnect('holder'); + +dblink_exec +----------- +COMMIT =20 +(1 row) + +dblink_disconnect +----------------- +OK =20 +(1 row) + + +starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only = s2_commit s3_vacuum_toast s3_vacuum_full s1_release +step s1_hold_xmin:=20 + SELECT dblink_connect('holder', + 'dbname=3Dregress_other_db port=3D' || current_setting('port')); + SELECT dblink_exec('holder', 'BEGIN'); + SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)'); + +dblink_connect +-------------- +OK =20 +(1 row) + +dblink_exec +----------- +BEGIN =20 +(1 row) + +dblink_exec=20 +------------ +CREATE TABLE +(1 row) + +step s2_begin:=20 + BEGIN ISOLATION LEVEL REPEATABLE READ; + SELECT 1; + +?column? +-------- + 1 +(1 row) + +step s3_delete:=20 + DELETE FROM rewrite_test WHERE id =3D 1; + +step s3_vacuum_main_only:=20 + VACUUM (PROCESS_TOAST false) rewrite_test; + +step s2_commit: COMMIT; +step s3_vacuum_toast:=20 + VACUUM pg_toast.rewrite_test_toast; + +step s3_vacuum_full:=20 + VACUUM FULL rewrite_test; + +step s1_release:=20 + SELECT dblink_exec('holder', 'COMMIT'); + SELECT dblink_disconnect('holder'); + +dblink_exec +----------- +COMMIT =20 +(1 row) + +dblink_disconnect +----------------- +OK =20 +(1 row) + + +starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only = s2_commit s3_vacuum_toast s3_repack s1_release +step s1_hold_xmin:=20 + SELECT dblink_connect('holder', + 'dbname=3Dregress_other_db port=3D' || current_setting('port')); + SELECT dblink_exec('holder', 'BEGIN'); + SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)'); + +dblink_connect +-------------- +OK =20 +(1 row) + +dblink_exec +----------- +BEGIN =20 +(1 row) + +dblink_exec=20 +------------ +CREATE TABLE +(1 row) + +step s2_begin:=20 + BEGIN ISOLATION LEVEL REPEATABLE READ; + SELECT 1; + +?column? +-------- + 1 +(1 row) + +step s3_delete:=20 + DELETE FROM rewrite_test WHERE id =3D 1; + +step s3_vacuum_main_only:=20 + VACUUM (PROCESS_TOAST false) rewrite_test; + +step s2_commit: COMMIT; +step s3_vacuum_toast:=20 + VACUUM pg_toast.rewrite_test_toast; + +step s3_repack:=20 + REPACK rewrite_test; + +step s1_release:=20 + SELECT dblink_exec('holder', 'COMMIT'); + SELECT dblink_disconnect('holder'); + +dblink_exec +----------- +COMMIT =20 +(1 row) + +dblink_disconnect +----------------- +OK =20 +(1 row) + + +starting permutation: s1_hold_xmin s2_begin s3_delete s3_vacuum_main_only = s2_commit s3_vacuum_toast s3_create_index s1_release +step s1_hold_xmin:=20 + SELECT dblink_connect('holder', + 'dbname=3Dregress_other_db port=3D' || current_setting('port')); + SELECT dblink_exec('holder', 'BEGIN'); + SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)'); + +dblink_connect +-------------- +OK =20 +(1 row) + +dblink_exec +----------- +BEGIN =20 +(1 row) + +dblink_exec=20 +------------ +CREATE TABLE +(1 row) + +step s2_begin:=20 + BEGIN ISOLATION LEVEL REPEATABLE READ; + SELECT 1; + +?column? +-------- + 1 +(1 row) + +step s3_delete:=20 + DELETE FROM rewrite_test WHERE id =3D 1; + +step s3_vacuum_main_only:=20 + VACUUM (PROCESS_TOAST false) rewrite_test; + +step s2_commit: COMMIT; +step s3_vacuum_toast:=20 + VACUUM pg_toast.rewrite_test_toast; + +step s3_create_index:=20 + CREATE INDEX rewrite_test_data_idx ON rewrite_test (data); + +step s1_release:=20 + SELECT dblink_exec('holder', 'COMMIT'); + SELECT dblink_disconnect('holder'); + +dblink_exec +----------- +COMMIT =20 +(1 row) + +dblink_disconnect +----------------- +OK =20 +(1 row) + diff --git a/contrib/dblink/meson.build b/contrib/dblink/meson.build index e2489f41229f..86fbd372b880 100644 --- a/contrib/dblink/meson.build +++ b/contrib/dblink/meson.build @@ -36,6 +36,11 @@ tests +=3D { ], 'regress_args': ['--dlpath', meson.project_build_root() / 'src/test/re= gress'], }, + 'isolation': { + 'specs': [ + 'rewrite_stale_xmin', + ], + }, 'tap': { 'tests': [ 't/001_auth_scram.pl', diff --git a/contrib/dblink/specs/rewrite_stale_xmin.spec b/contrib/dblink/= specs/rewrite_stale_xmin.spec new file mode 100644 index 000000000000..fe88e789f854 --- /dev/null +++ b/contrib/dblink/specs/rewrite_stale_xmin.spec @@ -0,0 +1,137 @@ +# Test that heap rewrites work when the rewriting backend's own xmin is +# held back by a transaction in another database. + +setup +{ + CREATE DATABASE regress_other_db; +} + +setup +{ + CREATE EXTENSION IF NOT EXISTS dblink; + + CREATE TABLE rewrite_test (id int PRIMARY KEY, data text) + WITH (autovacuum_enabled =3D off); + ALTER TABLE rewrite_test ALTER COLUMN data SET STORAGE EXTERNAL; + INSERT INTO rewrite_test VALUES (1, repeat('x', 2500)); + INSERT INTO rewrite_test VALUES (2, repeat('y', 2500)); + + -- Rename the toast table to a known name so we can VACUUM it directly. + SET allow_system_table_mods TO true; + DO $$DECLARE r record; + BEGIN + SELECT INTO r reltoastrelid::regclass::text AS table_name + FROM pg_class WHERE oid =3D 'rewrite_test'::regclass; + EXECUTE 'ALTER TABLE ' || r.table_name || ' RENAME TO rewrite_test= _toast;'; + END$$; + RESET allow_system_table_mods; +} + +teardown +{ + DROP DATABASE regress_other_db; +} + +# Session s1: use dblink to hold a transaction in another database. +# This must start before the DELETE so its XID is older, ensuring +# GetSnapshotData() holds back any rewrites. +session s1 +step s1_hold_xmin +{ + SELECT dblink_connect('holder', + 'dbname=3Dregress_other_db port=3D' || current_setting('port')); + SELECT dblink_exec('holder', 'BEGIN'); + SELECT dblink_exec('holder', 'CREATE TEMP TABLE xid_holder(x int)'); +} +step s1_release +{ + SELECT dblink_exec('holder', 'COMMIT'); + SELECT dblink_disconnect('holder'); +} + +# Session s2: holds xmin in the SAME database temporarily, so the +# main-table vacuum preserves the dead tuple as RECENTLY_DEAD. +session s2 +step s2_begin +{ + BEGIN ISOLATION LEVEL REPEATABLE READ; + SELECT 1; +} +step s2_commit { COMMIT; } + +# Session s3: performs DML, split vacuum, and the rewrite. +session s3 +step s3_delete +{ + DELETE FROM rewrite_test WHERE id =3D 1; +} +step s3_vacuum_main_only +{ + VACUUM (PROCESS_TOAST false) rewrite_test; +} +step s3_vacuum_toast +{ + VACUUM pg_toast.rewrite_test_toast; +} +step s3_cluster +{ + CLUSTER rewrite_test USING rewrite_test_pkey; +} +step s3_vacuum_full +{ + VACUUM FULL rewrite_test; +} +step s3_repack +{ + REPACK rewrite_test; +} +step s3_create_index +{ + CREATE INDEX rewrite_test_data_idx ON rewrite_test (data); +} + +teardown { DROP TABLE IF EXISTS rewrite_test; } + +# CLUSTER +permutation + s1_hold_xmin + s2_begin + s3_delete + s3_vacuum_main_only + s2_commit + s3_vacuum_toast + s3_cluster + s1_release + +# VACUUM FULL +permutation + s1_hold_xmin + s2_begin + s3_delete + s3_vacuum_main_only + s2_commit + s3_vacuum_toast + s3_vacuum_full + s1_release + +# REPACK +permutation + s1_hold_xmin + s2_begin + s3_delete + s3_vacuum_main_only + s2_commit + s3_vacuum_toast + s3_repack + s1_release + +# CREATE INDEX +permutation + s1_hold_xmin + s2_begin + s3_delete + s3_vacuum_main_only + s2_commit + s3_vacuum_toast + s3_create_index + s1_release --=20 2.55.0 --lnLgQFmtffG/DLE8-- --M7OpH/Q9yPhJPxKO Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAmpLTPcACgkQnvQgOdby QH3+shAAnuNhf4y1YRLNtcHAHlVV29skODX25reUOUNANQjX4moe3rg+OeNKxtpC yHXJE4D3bSXpQZKdEjg6zczIWkmQB6MO0QzkMbIk5Of7nB4Y1v4tOZfa1oo0TZ6r XOMCdwC4Ec47Rx6r0NpuDBusYf4CBIebjqWeCQ1xfeeKMkD+S/tzKiVfBxW05RYQ BP4x5Efp1YFbYcD0AQkR8hmlNf5ecJJxeABISgGAuOxygmRxcP6PmwfyIfiUQK6a XWGeAE0aT1Ct43Nb0ZIAtuTmvrJaGN0zzX9j0ELHLxTdjihHObhjQeSGU76lReC9 Jz0IfS77WGH6SRUYIfdGSbKBZEnHzND4ckegQ9OeMxuDxhWJJ/CMl80/LESk6288 jVlNwvIEJDm1g+THmyOwOJqjcw5GrakuGZYux/AprQWQYz06TXo2Qcl28nRH2JP6 acpR+YdeF3IHijBp5ubN4qPQ4ZgBh36gmVbEBSnIgmmt40B4ulMbLJOWAOceozGD NmW9zLQULFxOlaQysTKnBncYs7BGooTv2hB9S43cT5l5iWsh2yduyHwRa0skKCK2 d9DOplnlk0eOhVNtSzIFUo/mVTl0hZRq2wjWVbpAt6E4wkYFUXbY5XE5Oky1/MJ7 KOk6aGpUkIV4fLjflSHFpeTmrFIUFwNCdNIrc/jEA+4KpkQxMAQ= =mCyl -----END PGP SIGNATURE----- --M7OpH/Q9yPhJPxKO--