public inbox for [email protected]
help / color / mirror / Atom feedFrom: SATYANARAYANA NARLAPURAM <[email protected]>
To: Tom Lane <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Cc: Álvaro Herrera <[email protected]>
Cc: Antonin Houska <[email protected]>
Subject: Re: [Bug]Vacuum full silently NULL out fast default columns
Date: Mon, 4 May 2026 07:26:34 -0700
Message-ID: <CAHg+QDeteXPbPFsThTGTL=qp4dgiP1AH1kKktifcksDbiaRROw@mail.gmail.com> (raw)
In-Reply-To: <[email protected]>
References: <CAHg+QDeoccU5CudrJpmSKZfKZ1gRMNY=5BxSC=JpHgkonzgcOw@mail.gmail.com>
<[email protected]>
Hi,
On Mon, May 4, 2026 at 6:40 AM Tom Lane <[email protected]> wrote:
> SATYANARAYANA NARLAPURAM <[email protected]> writes:
> > VACUUM FULL silently turns columns added via ALTER TABLE ... ADD COLUMN
> ...
> > DEFAULT <const> into NULL
> > on all pre-existing rows. The issue exists for other operations like
> > CLUSTER, REPACK.
>
> That is a seriously awful bug. Fortunately it is not in any shipping
> release. A quick bisect run agrees that it broke here:
>
> 28d534e2ae0ac888b5460f977a10cd9bb017ef98 is the first bad commit
> commit 28d534e2ae0ac888b5460f977a10cd9bb017ef98 (HEAD)
> Author: Álvaro Herrera <[email protected]>
> Date: Mon Apr 6 21:55:08 2026 +0200
>
> Add CONCURRENTLY option to REPACK
>
> > Patch attached. Added a regression test in fast_default.sql covering
> > VACUUM FULL, CLUSTER, and REPACK on a table with fast-default columns
> > including a NOT NULL CHECK column.
>
> I don't know if this is the best code fix (I don't like putting extra
> checks into a loop condition like this). But I agree we need some
> more tests covering this area.
>
Thanks for reviewing! Attached v2 patch. Agreed, tried to optimize LOC in
V1. Before the change
loop was not breaking early, I fixed that as well in V2.
Thanks,
Satya
Attachments:
[application/octet-stream] v2-0001-VACUUM-FULL-silently-NULL-out-fast-default-columns.patch (4.7K, 3-v2-0001-VACUUM-FULL-silently-NULL-out-fast-default-columns.patch)
download | inline diff:
From 0d1b3254184cda790bc880d264295a6b5b93a151 Mon Sep 17 00:00:00 2001
From: Satya Narlapuram <[email protected]>
Date: Mon, 4 May 2026 08:52:50 +0000
Subject: [PATCH] VACUUM FULL silently NULL out fast-default columns
Commit 28d534e2 introduced reform_tuple() in heapam_handler.c with a
fast path that returns the source tuple verbatim when no dropped
columns require fixing up. The check ignores tuples that are short
due to attmissingval (the fast-default mechanism).
After the rewrite, finish_heap_swap() calls RelationClearMissing(),
clearing the catalog metadata that was the only source of those
values. Short tuples then read as NULL (or the type's zero value if
NOT NULL is in effect, also bypassing CHECK constraints). Affects
VACUUM FULL, CLUSTER, REPACK, and REPACK (CONCURRENTLY).
Force reform when the source tuple is shorter than the new tuple
descriptor so heap_deform_tuple() materializes the missing values
before heap_form_tuple() rebuilds a full-width tuple.
---
src/backend/access/heap/heapam_handler.c | 24 ++++++++++++---
src/test/regress/expected/fast_default.out | 36 ++++++++++++++++++++++
src/test/regress/sql/fast_default.sql | 16 ++++++++++
3 files changed, 72 insertions(+), 4 deletions(-)
diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c
index 20d3b46e..d7c07e08 100644
--- a/src/backend/access/heap/heapam_handler.c
+++ b/src/backend/access/heap/heapam_handler.c
@@ -2411,12 +2411,28 @@ reform_tuple(HeapTuple tuple, Relation OldHeap, Relation NewHeap,
TupleDesc newTupDesc = RelationGetDescr(NewHeap);
bool needs_reform = false;
+ /*
+ * If the tuple is short due to missing-value (fast-default) attributes,
+ * we must materialize them now: finish_heap_swap() will subsequently call
+ * RelationClearMissing() on the new relation, dropping the catalog
+ * metadata that is the only source of those values. Without reforming,
+ * those columns would silently become NULL after the rewrite.
+ */
+ if (HeapTupleHeaderGetNatts(tuple->t_data) < newTupDesc->natts)
+ needs_reform = true;
+
/* Skip work if the tuple doesn't need any attributes changed */
- for (int i = 0; i < newTupDesc->natts; i++)
+ if (!needs_reform)
{
- if (TupleDescCompactAttr(newTupDesc, i)->attisdropped &&
- !heap_attisnull(tuple, i + 1, newTupDesc))
- needs_reform = true;
+ for (int i = 0; i < newTupDesc->natts; i++)
+ {
+ if (TupleDescCompactAttr(newTupDesc, i)->attisdropped &&
+ !heap_attisnull(tuple, i + 1, newTupDesc))
+ {
+ needs_reform = true;
+ break;
+ }
+ }
}
if (!needs_reform)
return heap_copytuple(tuple);
diff --git a/src/test/regress/expected/fast_default.out b/src/test/regress/expected/fast_default.out
index bd142ed4..5813f1c6 100644
--- a/src/test/regress/expected/fast_default.out
+++ b/src/test/regress/expected/fast_default.out
@@ -969,6 +969,42 @@ SELECT count(*)
0
(1 row)
+-- Verify that table-rewriting maintenance commands preserve attmissingval
+-- columns.
+CREATE TABLE t (id int PRIMARY KEY);
+INSERT INTO t SELECT generate_series(1, 3);
+ALTER TABLE t ADD COLUMN a int DEFAULT 42;
+ALTER TABLE t ADD COLUMN b int NOT NULL DEFAULT 7 CHECK (b > 0);
+VACUUM FULL t;
+SELECT * FROM t ORDER BY id;
+ id | a | b
+----+----+---
+ 1 | 42 | 7
+ 2 | 42 | 7
+ 3 | 42 | 7
+(3 rows)
+
+ALTER TABLE t ADD COLUMN c text DEFAULT 'hello';
+CLUSTER t USING t_pkey;
+SELECT * FROM t ORDER BY id;
+ id | a | b | c
+----+----+---+-------
+ 1 | 42 | 7 | hello
+ 2 | 42 | 7 | hello
+ 3 | 42 | 7 | hello
+(3 rows)
+
+ALTER TABLE t ADD COLUMN d int DEFAULT 99;
+REPACK t;
+SELECT * FROM t ORDER BY id;
+ id | a | b | c | d
+----+----+---+-------+----
+ 1 | 42 | 7 | hello | 99
+ 2 | 42 | 7 | hello | 99
+ 3 | 42 | 7 | hello | 99
+(3 rows)
+
+DROP TABLE t;
-- cleanup
DROP FOREIGN TABLE ft1;
DROP SERVER s0;
diff --git a/src/test/regress/sql/fast_default.sql b/src/test/regress/sql/fast_default.sql
index 8b31d317..e5d9a3d2 100644
--- a/src/test/regress/sql/fast_default.sql
+++ b/src/test/regress/sql/fast_default.sql
@@ -653,6 +653,22 @@ SELECT count(*)
WHERE attrelid = 'ft1'::regclass AND
(attmissingval IS NOT NULL OR atthasmissing);
+-- Verify that table-rewriting maintenance commands preserve attmissingval
+-- columns.
+CREATE TABLE t (id int PRIMARY KEY);
+INSERT INTO t SELECT generate_series(1, 3);
+ALTER TABLE t ADD COLUMN a int DEFAULT 42;
+ALTER TABLE t ADD COLUMN b int NOT NULL DEFAULT 7 CHECK (b > 0);
+VACUUM FULL t;
+SELECT * FROM t ORDER BY id;
+ALTER TABLE t ADD COLUMN c text DEFAULT 'hello';
+CLUSTER t USING t_pkey;
+SELECT * FROM t ORDER BY id;
+ALTER TABLE t ADD COLUMN d int DEFAULT 99;
+REPACK t;
+SELECT * FROM t ORDER BY id;
+DROP TABLE t;
+
-- cleanup
DROP FOREIGN TABLE ft1;
DROP SERVER s0;
--
2.43.0
view thread (6+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: [Bug]Vacuum full silently NULL out fast default columns
In-Reply-To: <CAHg+QDeteXPbPFsThTGTL=qp4dgiP1AH1kKktifcksDbiaRROw@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox