public inbox for [email protected]
help / color / mirror / Atom feedFix tuple deformation with virtual generated NOT NULL columns
22+ messages / 5 participants
[nested] [flat]
* Fix tuple deformation with virtual generated NOT NULL columns
@ 2026-06-04 05:57 Chao Li <[email protected]>
2026-06-04 09:32 ` Re: Fix tuple deformation with virtual generated NOT NULL columns =?utf-8?B?Y2NhNTUwNw==?= <[email protected]>
2026-06-06 04:50 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
0 siblings, 2 replies; 22+ messages in thread
From: Chao Li @ 2026-06-04 05:57 UTC (permalink / raw)
To: Postgres hackers <[email protected]>; +Cc: David Rowley <[email protected]>
Hi,
While testing "Optimize tuple deformation”, I found a bug:
```
evantest=# create table t (a int not null,
evantest(# g int generated always as (a+1) virtual not null,
evantest(# b int not null);
CREATE TABLE
evantest=# insert into t (a, b) values (10, 20);
INSERT 0 1
evantest=# select a, g, b from t;
a | g | b
----+----+---
10 | 11 | 0
(1 row)
```
Here, b was inserted as 20, but select only returned 0.
I think the problem is in finding the first non-guaranteed attribute where virtual generated attributes are not considered:
```
for (int i = 0; i < tupdesc->natts; i++)
{
CompactAttribute *cattr = TupleDescCompactAttr(tupdesc, i);
/*
* Find the highest attnum which is guaranteed to exist in all tuples
* in the table. We currently only pay attention to byval attributes
* to allow additional optimizations during tuple deformation.
*/
if (firstNonGuaranteedAttr == tupdesc->natts &&
(cattr->attnullability != ATTNULLABLE_VALID || !cattr->attbyval ||
cattr->atthasmissing || cattr->attisdropped || cattr->attlen <= 0))
firstNonGuaranteedAttr = i;
```
To fix this, we should consider virtual generated attributes as non-guaranteed. The tricky part is that cattr->attgenerated is only a boolean and cannot distinguish virtual generated from stored. So we have to further check TupleDescAttr(tupdesc, i)->attgenerated. In the patch, I changed the check as follows:
```
if (firstNonGuaranteedAttr == tupdesc->natts &&
(cattr->attnullability != ATTNULLABLE_VALID || !cattr->attbyval ||
cattr->atthasmissing || cattr->attisdropped || cattr->attlen <= 0 ||
(cattr->attgenerated &&
TupleDescAttr(tupdesc, i)->attgenerated == ATTRIBUTE_GENERATED_VIRTUAL)))
firstNonGuaranteedAttr = i;
```
This way, we only check TupleDescAttr(tupdesc, i)->attgenerated when needed.
See the attached patch for details. I also added a regression test case to cover this fix. With the fix, select now returns correct values:
```
evantest=# select a, g, b from t;
a | g | b
----+----+----
10 | 11 | 20
(1 row)
```
Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/
Attachments:
[application/octet-stream] v1-0001-Fix-tuple-deformation-with-virtual-generated-NOT-.patch (4.0K, 2-v1-0001-Fix-tuple-deformation-with-virtual-generated-NOT-.patch)
download | inline diff:
From e462b3468489d2c42d6d836b677a06bd699194ba Mon Sep 17 00:00:00 2001
From: "Chao Li (Evan)" <[email protected]>
Date: Thu, 4 Jun 2026 13:51:24 +0800
Subject: [PATCH v1] Fix tuple deformation with virtual generated NOT NULL
columns
TupleDescFinalize() computes firstNonGuaranteedAttr for the slot
deformation fast path. Virtual generated columns can have valid NOT NULL
constraints, but they are not physically stored in heap tuples. Treating
such columns as part of the guaranteed physical prefix can make tuple
deformation advance the data offset as if the virtual column were stored,
causing following attributes to be read from the wrong location.
Exclude virtual generated columns from the guaranteed prefix while still
allowing stored generated columns to use the optimization.
Add a regression test with a virtual generated NOT NULL column followed by
another fixed-width NOT NULL column, which previously exposed the wrong
offset calculation.
Author: Chao Li <[email protected]>
Reviewed-by:
Discussion: https://postgr.es/m/
---
src/backend/access/common/tupdesc.c | 8 ++++++--
src/test/regress/expected/generated_virtual.out | 9 +++++++++
src/test/regress/sql/generated_virtual.sql | 5 +++++
3 files changed, 20 insertions(+), 2 deletions(-)
diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c
index 196472c05d0..4aee876a055 100644
--- a/src/backend/access/common/tupdesc.c
+++ b/src/backend/access/common/tupdesc.c
@@ -521,11 +521,15 @@ TupleDescFinalize(TupleDesc tupdesc)
/*
* Find the highest attnum which is guaranteed to exist in all tuples
* in the table. We currently only pay attention to byval attributes
- * to allow additional optimizations during tuple deformation.
+ * to allow additional optimizations during tuple deformation. Virtual
+ * generated columns are excluded, since they are computed at read
+ * time and are not physically stored in tuples.
*/
if (firstNonGuaranteedAttr == tupdesc->natts &&
(cattr->attnullability != ATTNULLABLE_VALID || !cattr->attbyval ||
- cattr->atthasmissing || cattr->attisdropped || cattr->attlen <= 0))
+ cattr->atthasmissing || cattr->attisdropped || cattr->attlen <= 0 ||
+ (cattr->attgenerated &&
+ TupleDescAttr(tupdesc, i)->attgenerated == ATTRIBUTE_GENERATED_VIRTUAL)))
firstNonGuaranteedAttr = i;
if (cattr->attlen <= 0)
diff --git a/src/test/regress/expected/generated_virtual.out b/src/test/regress/expected/generated_virtual.out
index 24d5dbf46ca..7a5788146f5 100644
--- a/src/test/regress/expected/generated_virtual.out
+++ b/src/test/regress/expected/generated_virtual.out
@@ -727,6 +727,15 @@ ERROR: null value in column "b" of relation "gtest21b" violates not-null constr
DETAIL: Failing row contains (null, virtual).
ALTER TABLE gtest21b ALTER COLUMN b DROP NOT NULL;
INSERT INTO gtest21b (a) VALUES (0); -- ok now
+-- virtual generated columns are not physically stored, even when not null
+CREATE TABLE gtest21c (a int NOT NULL, b int GENERATED ALWAYS AS (a * 2) VIRTUAL NOT NULL, c int NOT NULL);
+INSERT INTO gtest21c (a, c) VALUES (10, 42);
+SELECT a, b, c FROM gtest21c;
+ a | b | c
+----+----+----
+ 10 | 20 | 42
+(1 row)
+
-- not-null constraint with partitioned table
CREATE TABLE gtestnn_parent (
f1 int,
diff --git a/src/test/regress/sql/generated_virtual.sql b/src/test/regress/sql/generated_virtual.sql
index 9c2bb6590b3..126ae3ecda9 100644
--- a/src/test/regress/sql/generated_virtual.sql
+++ b/src/test/regress/sql/generated_virtual.sql
@@ -374,6 +374,11 @@ INSERT INTO gtest21b (a) VALUES (NULL); -- error
ALTER TABLE gtest21b ALTER COLUMN b DROP NOT NULL;
INSERT INTO gtest21b (a) VALUES (0); -- ok now
+-- virtual generated columns are not physically stored, even when not null
+CREATE TABLE gtest21c (a int NOT NULL, b int GENERATED ALWAYS AS (a * 2) VIRTUAL NOT NULL, c int NOT NULL);
+INSERT INTO gtest21c (a, c) VALUES (10, 42);
+SELECT a, b, c FROM gtest21c;
+
-- not-null constraint with partitioned table
CREATE TABLE gtestnn_parent (
f1 int,
--
2.50.1 (Apple Git-155)
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Fix tuple deformation with virtual generated NOT NULL columns
2026-06-04 05:57 Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
@ 2026-06-04 09:32 ` =?utf-8?B?Y2NhNTUwNw==?= <[email protected]>
2026-06-04 10:30 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
1 sibling, 1 reply; 22+ messages in thread
From: =?utf-8?B?Y2NhNTUwNw==?= @ 2026-06-04 09:32 UTC (permalink / raw)
To: =?utf-8?B?Q2hhbyBMaQ==?= <[email protected]>; =?utf-8?B?UG9zdGdyZXMgaGFja2Vycw==?= <[email protected]>; +Cc: =?utf-8?B?RGF2aWQgUm93bGV5?= <[email protected]>
> While testing "Optimize tuple deformation”, I found a bug:
> ```
> evantest=# create table t (a int not null,
> evantest(# g int generated always as (a+1) virtual not null,
> evantest(# b int not null);
> CREATE TABLE
> evantest=# insert into t (a, b) values (10, 20);
> INSERT 0 1
> evantest=# select a, g, b from t;
> a | g | b
> ----+----+---
> 10 | 11 | 0
> (1 row)
> ```
Nice catch! I can reproduce this bug on master. Some comments about the fix:
I find that a virtual generated column is stored as a null in heap tuple, so I think
we should stop setting 'attcacheoff' when we see a virtual generated column in
TupleDescFinalize(), or we will set wrong 'attcacheoff' value. But it seems that
we don't use these wrong value because we can only use 'attcacheoff' up until
the first NULL.
--
Regards,
ChangAo Chen
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Fix tuple deformation with virtual generated NOT NULL columns
2026-06-04 05:57 Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-04 09:32 ` Re: Fix tuple deformation with virtual generated NOT NULL columns =?utf-8?B?Y2NhNTUwNw==?= <[email protected]>
@ 2026-06-04 10:30 ` Chao Li <[email protected]>
2026-06-05 08:52 ` Re: Fix tuple deformation with virtual generated NOT NULL columns =?utf-8?B?Y2NhNTUwNw==?= <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: Chao Li @ 2026-06-04 10:30 UTC (permalink / raw)
To: cca5507 <[email protected]>; +Cc: Postgres hackers <[email protected]>; David Rowley <[email protected]>
> On Jun 4, 2026, at 17:32, cca5507 <[email protected]> wrote:
>
>> While testing "Optimize tuple deformation”, I found a bug:
>> ```
>> evantest=# create table t (a int not null,
>> evantest(# g int generated always as (a+1) virtual not null,
>> evantest(# b int not null);
>> CREATE TABLE
>> evantest=# insert into t (a, b) values (10, 20);
>> INSERT 0 1
>> evantest=# select a, g, b from t;
>> a | g | b
>> ----+----+---
>> 10 | 11 | 0
>> (1 row)
>> ```
>
> Nice catch! I can reproduce this bug on master. Some comments about the fix:
>
> I find that a virtual generated column is stored as a null in heap tuple, so I think
> we should stop setting 'attcacheoff' when we see a virtual generated column in
> TupleDescFinalize(), or we will set wrong 'attcacheoff' value. But it seems that
> we don't use these wrong value because we can only use 'attcacheoff' up until
> the first NULL.
>
> --
> Regards,
> ChangAo Chen
Hi ChangAo,
Thanks for your review. Please see v2 that addressed your comment.
Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/
Attachments:
[application/octet-stream] v2-0001-Fix-tuple-deformation-with-virtual-generated-NOT-.patch (4.4K, 2-v2-0001-Fix-tuple-deformation-with-virtual-generated-NOT-.patch)
download | inline diff:
From e8798ed747c6dc9e2930af5dce10de262d1e929a Mon Sep 17 00:00:00 2001
From: "Chao Li (Evan)" <[email protected]>
Date: Thu, 4 Jun 2026 13:51:24 +0800
Subject: [PATCH v2] Fix tuple deformation with virtual generated NOT NULL
columns
TupleDescFinalize() computes firstNonGuaranteedAttr for the slot
deformation fast path. Virtual generated columns can have valid NOT NULL
constraints, but they are not physically stored in heap tuples. Treating
such columns as part of the guaranteed physical prefix can make tuple
deformation advance the data offset as if the virtual column were stored,
causing following attributes to be read from the wrong location.
Exclude virtual generated columns from the guaranteed prefix while still
allowing stored generated columns to use the optimization.
Add a regression test with a virtual generated NOT NULL column followed by
another fixed-width NOT NULL column, which previously exposed the wrong
offset calculation.
Author: Chao Li <[email protected]>
Reviewed-by: ChangAo Chen <[email protected]>
Discussion: https://postgr.es/m/[email protected]
---
src/backend/access/common/tupdesc.c | 11 ++++++++---
src/test/regress/expected/generated_virtual.out | 9 +++++++++
src/test/regress/sql/generated_virtual.sql | 5 +++++
3 files changed, 22 insertions(+), 3 deletions(-)
diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c
index 196472c05d0..25267f7a9e5 100644
--- a/src/backend/access/common/tupdesc.c
+++ b/src/backend/access/common/tupdesc.c
@@ -517,18 +517,23 @@ TupleDescFinalize(TupleDesc tupdesc)
for (int i = 0; i < tupdesc->natts; i++)
{
CompactAttribute *cattr = TupleDescCompactAttr(tupdesc, i);
+ bool isVirtualGenerated = cattr->attgenerated &&
+ TupleDescAttr(tupdesc, i)->attgenerated == ATTRIBUTE_GENERATED_VIRTUAL;
/*
* Find the highest attnum which is guaranteed to exist in all tuples
* in the table. We currently only pay attention to byval attributes
- * to allow additional optimizations during tuple deformation.
+ * to allow additional optimizations during tuple deformation. Virtual
+ * generated columns are excluded, since they are computed at read
+ * time and are not physically stored in tuples.
*/
if (firstNonGuaranteedAttr == tupdesc->natts &&
(cattr->attnullability != ATTNULLABLE_VALID || !cattr->attbyval ||
- cattr->atthasmissing || cattr->attisdropped || cattr->attlen <= 0))
+ cattr->atthasmissing || cattr->attisdropped || cattr->attlen <= 0 ||
+ isVirtualGenerated))
firstNonGuaranteedAttr = i;
- if (cattr->attlen <= 0)
+ if (isVirtualGenerated || cattr->attlen <= 0)
break;
off = att_nominal_alignby(off, cattr->attalignby);
diff --git a/src/test/regress/expected/generated_virtual.out b/src/test/regress/expected/generated_virtual.out
index 24d5dbf46ca..7a5788146f5 100644
--- a/src/test/regress/expected/generated_virtual.out
+++ b/src/test/regress/expected/generated_virtual.out
@@ -727,6 +727,15 @@ ERROR: null value in column "b" of relation "gtest21b" violates not-null constr
DETAIL: Failing row contains (null, virtual).
ALTER TABLE gtest21b ALTER COLUMN b DROP NOT NULL;
INSERT INTO gtest21b (a) VALUES (0); -- ok now
+-- virtual generated columns are not physically stored, even when not null
+CREATE TABLE gtest21c (a int NOT NULL, b int GENERATED ALWAYS AS (a * 2) VIRTUAL NOT NULL, c int NOT NULL);
+INSERT INTO gtest21c (a, c) VALUES (10, 42);
+SELECT a, b, c FROM gtest21c;
+ a | b | c
+----+----+----
+ 10 | 20 | 42
+(1 row)
+
-- not-null constraint with partitioned table
CREATE TABLE gtestnn_parent (
f1 int,
diff --git a/src/test/regress/sql/generated_virtual.sql b/src/test/regress/sql/generated_virtual.sql
index 9c2bb6590b3..126ae3ecda9 100644
--- a/src/test/regress/sql/generated_virtual.sql
+++ b/src/test/regress/sql/generated_virtual.sql
@@ -374,6 +374,11 @@ INSERT INTO gtest21b (a) VALUES (NULL); -- error
ALTER TABLE gtest21b ALTER COLUMN b DROP NOT NULL;
INSERT INTO gtest21b (a) VALUES (0); -- ok now
+-- virtual generated columns are not physically stored, even when not null
+CREATE TABLE gtest21c (a int NOT NULL, b int GENERATED ALWAYS AS (a * 2) VIRTUAL NOT NULL, c int NOT NULL);
+INSERT INTO gtest21c (a, c) VALUES (10, 42);
+SELECT a, b, c FROM gtest21c;
+
-- not-null constraint with partitioned table
CREATE TABLE gtestnn_parent (
f1 int,
--
2.50.1 (Apple Git-155)
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Fix tuple deformation with virtual generated NOT NULL columns
2026-06-04 05:57 Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-04 09:32 ` Re: Fix tuple deformation with virtual generated NOT NULL columns =?utf-8?B?Y2NhNTUwNw==?= <[email protected]>
2026-06-04 10:30 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
@ 2026-06-05 08:52 ` =?utf-8?B?Y2NhNTUwNw==?= <[email protected]>
0 siblings, 0 replies; 22+ messages in thread
From: =?utf-8?B?Y2NhNTUwNw==?= @ 2026-06-05 08:52 UTC (permalink / raw)
To: =?utf-8?B?Q2hhbyBMaQ==?= <[email protected]>; +Cc: =?utf-8?B?UG9zdGdyZXMgaGFja2Vycw==?= <[email protected]>; =?utf-8?B?RGF2aWQgUm93bGV5?= <[email protected]>
> Thanks for your review. Please see v2 that addressed your comment.
The v2 patch LGTM.
--
Regards,
ChangAo Chen
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Fix tuple deformation with virtual generated NOT NULL columns
2026-06-04 05:57 Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
@ 2026-06-06 04:50 ` David Rowley <[email protected]>
2026-06-07 01:43 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-08 13:01 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Andres Freund <[email protected]>
1 sibling, 2 replies; 22+ messages in thread
From: David Rowley @ 2026-06-06 04:50 UTC (permalink / raw)
To: Chao Li <[email protected]>; +Cc: Postgres hackers <[email protected]>
On Thu, 4 Jun 2026 at 17:57, Chao Li <[email protected]> wrote:
> While testing "Optimize tuple deformation”, I found a bug:
> I think the problem is in finding the first non-guaranteed attribute where virtual generated attributes are not considered:
Thanks for the report and fix. I pushed a slightly adjusted version.
Form_pg_attribute.attgenerated is '\0' for non-generated columns, so
there's no point in checking cattr->attgenerated as well as that.
I also added an Assert() to help catch any other reason that the
guaranteed column gets set incorrectly again in TupleDescFinalize().
David
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Fix tuple deformation with virtual generated NOT NULL columns
2026-06-04 05:57 Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-06 04:50 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
@ 2026-06-07 01:43 ` Chao Li <[email protected]>
1 sibling, 0 replies; 22+ messages in thread
From: Chao Li @ 2026-06-07 01:43 UTC (permalink / raw)
To: David Rowley <[email protected]>; +Cc: Postgres hackers <[email protected]>
> On Jun 6, 2026, at 12:50, David Rowley <[email protected]> wrote:
>
> On Thu, 4 Jun 2026 at 17:57, Chao Li <[email protected]> wrote:
>> While testing "Optimize tuple deformation”, I found a bug:
>
>> I think the problem is in finding the first non-guaranteed attribute where virtual generated attributes are not considered:
>
> Thanks for the report and fix. I pushed a slightly adjusted version.
Thanks for pushing.
> Form_pg_attribute.attgenerated is '\0' for non-generated columns, so
> there's no point in checking cattr->attgenerated as well as that.
>
I was trying to avoid unconditionally fetching TupleDescAttr(tupdesc, i), since cattr->attgenerated is already available and most columns won't be virtual generated columns. But I was probably overthinking the performance angle here. TupleDescFinalize() runs only when finalizing a tuple descriptor, not during per-tuple deformation. I agree your adjusted version is cleaner.
Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Fix tuple deformation with virtual generated NOT NULL columns
2026-06-04 05:57 Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-06 04:50 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
@ 2026-06-08 13:01 ` Andres Freund <[email protected]>
2026-06-09 02:19 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
1 sibling, 1 reply; 22+ messages in thread
From: Andres Freund @ 2026-06-08 13:01 UTC (permalink / raw)
To: David Rowley <[email protected]>; +Cc: Chao Li <[email protected]>; Postgres hackers <[email protected]>
Hi,
On 2026-06-06 16:50:29 +1200, David Rowley wrote:
> On Thu, 4 Jun 2026 at 17:57, Chao Li <[email protected]> wrote:
> > While testing "Optimize tuple deformation”, I found a bug:
>
> > I think the problem is in finding the first non-guaranteed attribute where virtual generated attributes are not considered:
>
> Thanks for the report and fix. I pushed a slightly adjusted version.
> Form_pg_attribute.attgenerated is '\0' for non-generated columns, so
> there's no point in checking cattr->attgenerated as well as that.
>
> I also added an Assert() to help catch any other reason that the
> guaranteed column gets set incorrectly again in TupleDescFinalize().
Seems like a test for some of this would be good too?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Fix tuple deformation with virtual generated NOT NULL columns
2026-06-04 05:57 Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-06 04:50 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-08 13:01 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Andres Freund <[email protected]>
@ 2026-06-09 02:19 ` Chao Li <[email protected]>
2026-06-12 04:26 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: Chao Li @ 2026-06-09 02:19 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: David Rowley <[email protected]>; Postgres hackers <[email protected]>
> On Jun 8, 2026, at 21:01, Andres Freund <[email protected]> wrote:
>
> Hi,
>
> On 2026-06-06 16:50:29 +1200, David Rowley wrote:
>> On Thu, 4 Jun 2026 at 17:57, Chao Li <[email protected]> wrote:
>>> While testing "Optimize tuple deformation”, I found a bug:
>>
>>> I think the problem is in finding the first non-guaranteed attribute where virtual generated attributes are not considered:
>>
>> Thanks for the report and fix. I pushed a slightly adjusted version.
>> Form_pg_attribute.attgenerated is '\0' for non-generated columns, so
>> there's no point in checking cattr->attgenerated as well as that.
>>
>> I also added an Assert() to help catch any other reason that the
>> guaranteed column gets set incorrectly again in TupleDescFinalize().
>
> Seems like a test for some of this would be good too?
>
> Greetings,
>
> Andres Freund
Okay, I tried to add a test matching my repro. With this test in place, I reverted the fix and ran make check; it failed by hitting the Assert David added:
```
TRAP: failed Assert("first_null_attr(tup->t_bits, natts) >= firstNullAttr"), File: "execTuples.c", Line: 1083, PID: 65804
0 postgres 0x0000000104e883b8 ExceptionalCondition + 216
1 postgres 0x00000001048e5ebc slot_deform_heap_tuple + 456
2 postgres 0x00000001048e2a18 tts_buffer_heap_getsomeattrs + 112
3 postgres 0x00000001048c5044 slot_getsomeattrs + 68
4 postgres 0x00000001048b97a4 ExecInterpExpr + 416
5 postgres 0x00000001048b8ea0 ExecInterpExprStillValid + 76
6 postgres 0x000000010492ec68 ExecEvalExprNoReturn + 44
7 postgres 0x000000010492ec28 ExecEvalExprNoReturnSwitchContext + 48
8 postgres 0x000000010492eb20 ExecProject + 72
9 postgres 0x000000010492e680 ExecScanExtended + 288
10 postgres 0x000000010492d7cc ExecSeqScanWithProject + 220
11 postgres 0x00000001048db508 ExecProcNodeFirst + 92
12 postgres 0x00000001048d1eb8 ExecProcNode + 60
```
See the attached patch for details.
Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/
Attachments:
[application/octet-stream] v1-0001-Add-regression-test-for-virtual-generated-column-.patch (2.6K, 2-v1-0001-Add-regression-test-for-virtual-generated-column-.patch)
download | inline diff:
From 944e72ad211a9ac694e7fdeb9361f9fe3e2a01d4 Mon Sep 17 00:00:00 2001
From: "Chao Li (Evan)" <[email protected]>
Date: Tue, 9 Jun 2026 09:58:44 +0800
Subject: [PATCH v1] Add regression test for virtual generated column
deformation
Add coverage for a virtual generated NOT NULL column followed by a
physically stored NOT NULL column. This exercises the tuple deformation
case fixed by 89eafad297a, where TupleDescFinalize() could incorrectly
treat a virtual generated column as part of the guaranteed physical column
prefix and compute cached offsets past it.
Without that fix, deforming the following column could read from the wrong
tuple offset.
Suggested-by: Andres Freund <[email protected]>
Author: Chao Li <[email protected]>
Discussion: https://postgr.es/m/A4BC563C-0CA3-4EF3-952A-EA41F9E5BF1E%40gmail.com
---
src/test/regress/expected/generated_virtual.out | 9 +++++++++
src/test/regress/sql/generated_virtual.sql | 5 +++++
2 files changed, 14 insertions(+)
diff --git a/src/test/regress/expected/generated_virtual.out b/src/test/regress/expected/generated_virtual.out
index 24d5dbf46ca..7a5788146f5 100644
--- a/src/test/regress/expected/generated_virtual.out
+++ b/src/test/regress/expected/generated_virtual.out
@@ -727,6 +727,15 @@ ERROR: null value in column "b" of relation "gtest21b" violates not-null constr
DETAIL: Failing row contains (null, virtual).
ALTER TABLE gtest21b ALTER COLUMN b DROP NOT NULL;
INSERT INTO gtest21b (a) VALUES (0); -- ok now
+-- virtual generated columns are not physically stored, even when not null
+CREATE TABLE gtest21c (a int NOT NULL, b int GENERATED ALWAYS AS (a * 2) VIRTUAL NOT NULL, c int NOT NULL);
+INSERT INTO gtest21c (a, c) VALUES (10, 42);
+SELECT a, b, c FROM gtest21c;
+ a | b | c
+----+----+----
+ 10 | 20 | 42
+(1 row)
+
-- not-null constraint with partitioned table
CREATE TABLE gtestnn_parent (
f1 int,
diff --git a/src/test/regress/sql/generated_virtual.sql b/src/test/regress/sql/generated_virtual.sql
index 9c2bb6590b3..126ae3ecda9 100644
--- a/src/test/regress/sql/generated_virtual.sql
+++ b/src/test/regress/sql/generated_virtual.sql
@@ -374,6 +374,11 @@ INSERT INTO gtest21b (a) VALUES (NULL); -- error
ALTER TABLE gtest21b ALTER COLUMN b DROP NOT NULL;
INSERT INTO gtest21b (a) VALUES (0); -- ok now
+-- virtual generated columns are not physically stored, even when not null
+CREATE TABLE gtest21c (a int NOT NULL, b int GENERATED ALWAYS AS (a * 2) VIRTUAL NOT NULL, c int NOT NULL);
+INSERT INTO gtest21c (a, c) VALUES (10, 42);
+SELECT a, b, c FROM gtest21c;
+
-- not-null constraint with partitioned table
CREATE TABLE gtestnn_parent (
f1 int,
--
2.50.1 (Apple Git-155)
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Fix tuple deformation with virtual generated NOT NULL columns
2026-06-04 05:57 Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-06 04:50 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-08 13:01 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Andres Freund <[email protected]>
2026-06-09 02:19 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
@ 2026-06-12 04:26 ` David Rowley <[email protected]>
2026-06-17 05:04 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: David Rowley @ 2026-06-12 04:26 UTC (permalink / raw)
To: Chao Li <[email protected]>; Peter Eisentraut <[email protected]>; +Cc: Andres Freund <[email protected]>; Postgres hackers <[email protected]>
On Tue, 9 Jun 2026 at 14:20, Chao Li <[email protected]> wrote:
> > On Jun 8, 2026, at 21:01, Andres Freund <[email protected]> wrote:
> > Seems like a test for some of this would be good too?
> Okay, I tried to add a test matching my repro. With this test in place, I reverted the fix and ran make check; it failed by hitting the Assert David added:
> ```
> TRAP: failed Assert("first_null_attr(tup->t_bits, natts) >= firstNullAttr"), File: "execTuples.c", Line: 1083, PID: 65804
I was on the fence about that test as I felt it was mostly
highlighting that there was a bug once with that particular case, and
didn't feel like it did much to prevent future omissions from
TupleDescFinalize(). I felt the Assert would help highlight future
stuff.
However, I don't object to adding the test if others feel it's
worthwhile, but on looking at the patch, there are a couple of things
that stand out.
1. Nothing has been done for the comment at the top of the file that
says "-- keep these tests aligned with generated_stored.sql". It
looks like there have been quite a few commits already which have
neglected this. Maybe that means we should do away with the comment
rather than try to align the two.
2. I can't quite figure out the pattern in these tests for dropping vs
not dropping the tables at the end of the test. Many tests do DROP
TABLE and a large number of others don't bother. What's meant to be
happening here?
I've added Peter as I think it was his intention to keep these tests
aligned with generated_stored.sql.
David
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Fix tuple deformation with virtual generated NOT NULL columns
2026-06-04 05:57 Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-06 04:50 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-08 13:01 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Andres Freund <[email protected]>
2026-06-09 02:19 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-12 04:26 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
@ 2026-06-17 05:04 ` David Rowley <[email protected]>
2026-06-17 06:11 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Tom Lane <[email protected]>
2026-06-17 22:23 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Tom Lane <[email protected]>
0 siblings, 2 replies; 22+ messages in thread
From: David Rowley @ 2026-06-17 05:04 UTC (permalink / raw)
To: Chao Li <[email protected]>; Peter Eisentraut <[email protected]>; +Cc: Andres Freund <[email protected]>; Postgres hackers <[email protected]>
On Fri, 12 Jun 2026 at 16:26, David Rowley <[email protected]> wrote:
> However, I don't object to adding the test if others feel it's
> worthwhile, but on looking at the patch, there are a couple of things
> that stand out.
>
> 1. Nothing has been done for the comment at the top of the file that
> says "-- keep these tests aligned with generated_stored.sql". It
> looks like there have been quite a few commits already which have
> neglected this. Maybe that means we should do away with the comment
> rather than try to align the two.
I pushed the patch to add the new test after modifying it to also
include the change in the generated_stored test, but with it commented
out. This is the method that 83ea6c540 introduced.
Since a few commits have already put these files out of sync, maybe we
should consider having a single test file that we include with \i
after doing something like:
\set generated_type STORED
\set test_generated_stored true
\set test_generated_virtual false
\i generated_generic.sql
and have generated_generic.sql use \if test_generated_(stored|virtual)
as and when required.
> 2. I can't quite figure out the pattern in these tests for dropping vs
> not dropping the tables at the end of the test. Many tests do DROP
> TABLE and a large number of others don't bother. What's meant to be
> happening here?
I added the DROP TABLE too, as I didn't see any reason mentioned
anywhere that they should be kept.
David
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Fix tuple deformation with virtual generated NOT NULL columns
2026-06-04 05:57 Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-06 04:50 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-08 13:01 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Andres Freund <[email protected]>
2026-06-09 02:19 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-12 04:26 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 05:04 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
@ 2026-06-17 06:11 ` Tom Lane <[email protected]>
1 sibling, 0 replies; 22+ messages in thread
From: Tom Lane @ 2026-06-17 06:11 UTC (permalink / raw)
To: David Rowley <[email protected]>; +Cc: Chao Li <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Postgres hackers <[email protected]>
David Rowley <[email protected]> writes:
>> 2. I can't quite figure out the pattern in these tests for dropping vs
>> not dropping the tables at the end of the test. Many tests do DROP
>> TABLE and a large number of others don't bother. What's meant to be
>> happening here?
> I added the DROP TABLE too, as I didn't see any reason mentioned
> anywhere that they should be kept.
A very rough rule of thumb is that we leave tables around if they
might be interesting for the pg_upgrade tests (which try to upgrade
the ending state of the core regression tests).
regards, tom lane
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Fix tuple deformation with virtual generated NOT NULL columns
2026-06-04 05:57 Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-06 04:50 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-08 13:01 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Andres Freund <[email protected]>
2026-06-09 02:19 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-12 04:26 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 05:04 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
@ 2026-06-17 22:23 ` Tom Lane <[email protected]>
2026-06-17 22:45 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Tom Lane <[email protected]>
1 sibling, 1 reply; 22+ messages in thread
From: Tom Lane @ 2026-06-17 22:23 UTC (permalink / raw)
To: David Rowley <[email protected]>; +Cc: Chao Li <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Postgres hackers <[email protected]>
David Rowley <[email protected]> writes:
> I pushed the patch to add the new test after modifying it to also
> include the change in the generated_stored test, but with it commented
> out. This is the method that 83ea6c540 introduced.
The buildfarm is showing clear evidence of a bug here somewhere:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=bushmaster&dt=2026-06-17%2021%3A53%3A38
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=taipan&dt=2026-06-17%2018%3A50%3A42
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=canebrake&dt=2026-06-17%2018%3A02%3A59
All three failures look like
diff -U3 /home/bf/bf-build/taipan/HEAD/pgsql/src/test/regress/expected/generated_virtual.out /home/bf/bf-build/taipan/HEAD/pgsql.build/src/test/regress/results/generated_virtual.out
--- /home/bf/bf-build/taipan/HEAD/pgsql/src/test/regress/expected/generated_virtual.out 2026-06-17 07:00:34.298450176 +0200
+++ /home/bf/bf-build/taipan/HEAD/pgsql.build/src/test/regress/results/generated_virtual.out 2026-06-17 20:51:26.084674716 +0200
@@ -731,9 +731,9 @@
CREATE TABLE gtest21c (a int NOT NULL, b int GENERATED ALWAYS AS (a * 2) VIRTUAL NOT NULL, c int NOT NULL);
INSERT INTO gtest21c (a, c) VALUES (10, 42);
SELECT a, b, c FROM gtest21c;
- a | b | c
-----+----+----
- 10 | 20 | 42
+ a | b | c
+----+----+---
+ 10 | 20 | 0
(1 row)
DROP TABLE gtest21c;
Speculating wildly, I'm wondering about an uninitialized variable that
happens to usually have the right value.
regards, tom lane
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Fix tuple deformation with virtual generated NOT NULL columns
2026-06-04 05:57 Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-06 04:50 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-08 13:01 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Andres Freund <[email protected]>
2026-06-09 02:19 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-12 04:26 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 05:04 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 22:23 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Tom Lane <[email protected]>
@ 2026-06-17 22:45 ` Tom Lane <[email protected]>
2026-06-17 23:06 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: Tom Lane @ 2026-06-17 22:45 UTC (permalink / raw)
To: David Rowley <[email protected]>; +Cc: Chao Li <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Postgres hackers <[email protected]>
I wrote:
> Speculating wildly, I'm wondering about an uninitialized variable that
> happens to usually have the right value.
Nope: valgrind finds nothing, and neither does debug_discard_caches.
What does reproduce it, with seeming 100% reliability, is
set jit = 1;
set jit_above_cost = 0;
set jit_optimize_above_cost = 1000;
CREATE TABLE gtest21c (a int NOT NULL, b int GENERATED ALWAYS AS (a * 2) VIRTUAL NOT NULL, c int NOT NULL);
INSERT INTO gtest21c (a, c) VALUES (10, 42);
table gtest21c;
I conclude that something in the JIT code for tuple formation
is unaware of virtual generated columns.
regards, tom lane
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Fix tuple deformation with virtual generated NOT NULL columns
2026-06-04 05:57 Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-06 04:50 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-08 13:01 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Andres Freund <[email protected]>
2026-06-09 02:19 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-12 04:26 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 05:04 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 22:23 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Tom Lane <[email protected]>
2026-06-17 22:45 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Tom Lane <[email protected]>
@ 2026-06-17 23:06 ` David Rowley <[email protected]>
2026-06-17 23:37 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: David Rowley @ 2026-06-17 23:06 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Chao Li <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Postgres hackers <[email protected]>
On Thu, 18 Jun 2026 at 10:45, Tom Lane <[email protected]> wrote:
> set jit = 1;
> set jit_above_cost = 0;
> set jit_optimize_above_cost = 1000;
> CREATE TABLE gtest21c (a int NOT NULL, b int GENERATED ALWAYS AS (a * 2) VIRTUAL NOT NULL, c int NOT NULL);
> INSERT INTO gtest21c (a, c) VALUES (10, 42);
> table gtest21c;
>
> I conclude that something in the JIT code for tuple formation
> is unaware of virtual generated columns.
Just starting to look now, but I suspect that this code in
llvmjit_deform.c needs to be updated now that we have virtual
generated columns.
/*
* If the column is declared NOT NULL then it must be present in every
* tuple, unless there's a "missing" entry that could provide a
* non-NULL value for it. That in turn guarantees that the NULL bitmap
* - if there are any NULLable columns - is at least long enough to
* cover columns up to attnum.
*
* Be paranoid and also check !attisdropped, even though the
* combination of attisdropped && attnotnull combination shouldn't
* exist.
*/
if (att->attnullability == ATTNULLABLE_VALID &&
!att->atthasmissing &&
!att->attisdropped)
guaranteed_column_number = attnum;
David
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Fix tuple deformation with virtual generated NOT NULL columns
2026-06-04 05:57 Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-06 04:50 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-08 13:01 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Andres Freund <[email protected]>
2026-06-09 02:19 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-12 04:26 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 05:04 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 22:23 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Tom Lane <[email protected]>
2026-06-17 22:45 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Tom Lane <[email protected]>
2026-06-17 23:06 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
@ 2026-06-17 23:37 ` David Rowley <[email protected]>
2026-06-18 02:35 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: David Rowley @ 2026-06-17 23:37 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Chao Li <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Postgres hackers <[email protected]>
On Thu, 18 Jun 2026 at 11:06, David Rowley <[email protected]> wrote:
> Just starting to look now, but I suspect that this code in
> llvmjit_deform.c needs to be updated now that we have virtual
> generated columns.
I've not fully processed all this code yet. I've still not worked out
why the loop that sets guaranteed_column_number doesn't break when it
finds something non-guaranteed.
Here's a draft patch I was experimenting with. It seems to fix the
issue. I need to spend more time to check it's correct.
David
diff --git a/src/backend/jit/llvm/llvmjit_deform.c b/src/backend/jit/llvm/llvmjit_deform.c
index 12521e3e46a..62a7e1c37fb 100644
--- a/src/backend/jit/llvm/llvmjit_deform.c
+++ b/src/backend/jit/llvm/llvmjit_deform.c
@@ -109,22 +109,27 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
*/
for (attnum = 0; attnum < desc->natts; attnum++)
{
- CompactAttribute *att = TupleDescCompactAttr(desc, attnum);
+ CompactAttribute *catt = TupleDescCompactAttr(desc, attnum);
+ Form_pg_attribute attr = TupleDescAttr(desc, attnum);
/*
* If the column is declared NOT NULL then it must be present in every
* tuple, unless there's a "missing" entry that could provide a
- * non-NULL value for it. That in turn guarantees that the NULL bitmap
- * - if there are any NULLable columns - is at least long enough to
- * cover columns up to attnum.
+ * non-NULL value for it or the column is a virtual generated column.
+ * That in turn guarantees that the NULL bitmap - if there are any
+ * NULLable columns - is at least long enough to cover columns up to
+ * attnum.
*
* Be paranoid and also check !attisdropped, even though the
* combination of attisdropped && attnotnull combination shouldn't
* exist.
*/
- if (att->attnullability == ATTNULLABLE_VALID &&
- !att->atthasmissing &&
- !att->attisdropped)
+ if (attr->attgenerated == ATTRIBUTE_GENERATED_VIRTUAL)
+ break;
+
+ if (catt->attnullability == ATTNULLABLE_VALID &&
+ !catt->atthasmissing &&
+ !catt->attisdropped)
guaranteed_column_number = attnum;
}
@@ -392,6 +397,8 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
for (attnum = 0; attnum < natts; attnum++)
{
CompactAttribute *att = TupleDescCompactAttr(desc, attnum);
+ Form_pg_attribute attr = TupleDescAttr(desc, attnum);
+
LLVMValueRef v_incby;
int alignto = att->attalignby;
LLVMValueRef l_attno = l_int16_const(lc, attnum);
@@ -436,7 +443,8 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
* into account, because if they're present the heaptuple's natts
* would have indicated that a slot_getmissingattrs() is needed.
*/
- if (att->attnullability != ATTNULLABLE_VALID)
+ if (att->attnullability != ATTNULLABLE_VALID ||
+ attr->attgenerated == ATTRIBUTE_GENERATED_VIRTUAL)
{
LLVMBasicBlockRef b_ifnotnull;
LLVMBasicBlockRef b_ifnull;
@@ -614,6 +622,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
known_alignment += att->attlen;
}
else if (att->attnullability == ATTNULLABLE_VALID &&
+ attr->attgenerated != ATTRIBUTE_GENERATED_VIRTUAL &&
(att->attlen % alignto) == 0)
{
/*
Attachments:
[text/plain] fix_jit_deform_for_virtual_generated_cols.patch (2.6K, 2-fix_jit_deform_for_virtual_generated_cols.patch)
download | inline diff:
diff --git a/src/backend/jit/llvm/llvmjit_deform.c b/src/backend/jit/llvm/llvmjit_deform.c
index 12521e3e46a..62a7e1c37fb 100644
--- a/src/backend/jit/llvm/llvmjit_deform.c
+++ b/src/backend/jit/llvm/llvmjit_deform.c
@@ -109,22 +109,27 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
*/
for (attnum = 0; attnum < desc->natts; attnum++)
{
- CompactAttribute *att = TupleDescCompactAttr(desc, attnum);
+ CompactAttribute *catt = TupleDescCompactAttr(desc, attnum);
+ Form_pg_attribute attr = TupleDescAttr(desc, attnum);
/*
* If the column is declared NOT NULL then it must be present in every
* tuple, unless there's a "missing" entry that could provide a
- * non-NULL value for it. That in turn guarantees that the NULL bitmap
- * - if there are any NULLable columns - is at least long enough to
- * cover columns up to attnum.
+ * non-NULL value for it or the column is a virtual generated column.
+ * That in turn guarantees that the NULL bitmap - if there are any
+ * NULLable columns - is at least long enough to cover columns up to
+ * attnum.
*
* Be paranoid and also check !attisdropped, even though the
* combination of attisdropped && attnotnull combination shouldn't
* exist.
*/
- if (att->attnullability == ATTNULLABLE_VALID &&
- !att->atthasmissing &&
- !att->attisdropped)
+ if (attr->attgenerated == ATTRIBUTE_GENERATED_VIRTUAL)
+ break;
+
+ if (catt->attnullability == ATTNULLABLE_VALID &&
+ !catt->atthasmissing &&
+ !catt->attisdropped)
guaranteed_column_number = attnum;
}
@@ -392,6 +397,8 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
for (attnum = 0; attnum < natts; attnum++)
{
CompactAttribute *att = TupleDescCompactAttr(desc, attnum);
+ Form_pg_attribute attr = TupleDescAttr(desc, attnum);
+
LLVMValueRef v_incby;
int alignto = att->attalignby;
LLVMValueRef l_attno = l_int16_const(lc, attnum);
@@ -436,7 +443,8 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
* into account, because if they're present the heaptuple's natts
* would have indicated that a slot_getmissingattrs() is needed.
*/
- if (att->attnullability != ATTNULLABLE_VALID)
+ if (att->attnullability != ATTNULLABLE_VALID ||
+ attr->attgenerated == ATTRIBUTE_GENERATED_VIRTUAL)
{
LLVMBasicBlockRef b_ifnotnull;
LLVMBasicBlockRef b_ifnull;
@@ -614,6 +622,7 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
known_alignment += att->attlen;
}
else if (att->attnullability == ATTNULLABLE_VALID &&
+ attr->attgenerated != ATTRIBUTE_GENERATED_VIRTUAL &&
(att->attlen % alignto) == 0)
{
/*
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Fix tuple deformation with virtual generated NOT NULL columns
2026-06-04 05:57 Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-06 04:50 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-08 13:01 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Andres Freund <[email protected]>
2026-06-09 02:19 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-12 04:26 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 05:04 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 22:23 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Tom Lane <[email protected]>
2026-06-17 22:45 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Tom Lane <[email protected]>
2026-06-17 23:06 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 23:37 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
@ 2026-06-18 02:35 ` Chao Li <[email protected]>
2026-06-18 05:18 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: Chao Li @ 2026-06-18 02:35 UTC (permalink / raw)
To: David Rowley <[email protected]>; +Cc: Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Postgres hackers <[email protected]>
> On Jun 18, 2026, at 07:37, David Rowley <[email protected]> wrote:
>
> On Thu, 18 Jun 2026 at 11:06, David Rowley <[email protected]> wrote:
>> Just starting to look now, but I suspect that this code in
>> llvmjit_deform.c needs to be updated now that we have virtual
>> generated columns.
>
> I've not fully processed all this code yet. I've still not worked out
> why the loop that sets guaranteed_column_number doesn't break when it
> finds something non-guaranteed.
>
> Here's a draft patch I was experimenting with. It seems to fix the
> issue. I need to spend more time to check it's correct.
>
> David
> <fix_jit_deform_for_virtual_generated_cols.patch>
I was not aware of the JIT code before. I think the good thing is that the new test uncovered this JIT bug.
I tested the fix, and it seems to work. While tracing the code, I wondered about this part:
```
- if (att->attnullability == ATTNULLABLE_VALID &&
- !att->atthasmissing &&
- !att->attisdropped)
+ if (attr->attgenerated == ATTRIBUTE_GENERATED_VIRTUAL)
+ break;
+
+ if (catt->attnullability == ATTNULLABLE_VALID &&
+ !catt->atthasmissing &&
+ !catt->attisdropped)
guaranteed_column_number = attnum;
```
When computing guaranteed_column_number, I think we can just skip the virtual generated column rather than break. Using the test from Tom’s email:
```
CREATE TABLE gtest21c (a int NOT NULL, b int GENERATED ALWAYS AS (a * 2) VIRTUAL NOT NULL, c int NOT NULL);
```
Here, “c" is a real NOT NULL column, so we can skip “b" and let “c" set guaranteed_column_number. This matters for the later check:
```
/*
* Check if it is guaranteed that all the desired attributes are available
* in the tuple (but still possibly NULL), by dint of either the last
* to-be-deformed column being NOT NULL, or subsequent ones not accessed
* here being NOT NULL. If that's not guaranteed the tuple headers natt's
* has to be checked, and missing attributes potentially have to be
* fetched (using slot_getmissingattrs().
*/
if ((natts - 1) <= guaranteed_column_number)
{
```
If we break at “b", then this check goes to the else branch and invokes the maxatt check plus slot_getmissingattrs(), even though the later NOT NULL “c" proves that the tuple has enough attributes.
Otherwise, the fix looks good to me.
Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Fix tuple deformation with virtual generated NOT NULL columns
2026-06-04 05:57 Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-06 04:50 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-08 13:01 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Andres Freund <[email protected]>
2026-06-09 02:19 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-12 04:26 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 05:04 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 22:23 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Tom Lane <[email protected]>
2026-06-17 22:45 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Tom Lane <[email protected]>
2026-06-17 23:06 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 23:37 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-18 02:35 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
@ 2026-06-18 05:18 ` David Rowley <[email protected]>
2026-06-18 05:55 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: David Rowley @ 2026-06-18 05:18 UTC (permalink / raw)
To: Chao Li <[email protected]>; +Cc: Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Postgres hackers <[email protected]>
On Thu, 18 Jun 2026 at 14:36, Chao Li <[email protected]> wrote:
> I tested the fix, and it seems to work. While tracing the code, I wondered about this part:
> ```
> - if (att->attnullability == ATTNULLABLE_VALID &&
> - !att->atthasmissing &&
> - !att->attisdropped)
> + if (attr->attgenerated == ATTRIBUTE_GENERATED_VIRTUAL)
> + break;
> +
> + if (catt->attnullability == ATTNULLABLE_VALID &&
> + !catt->atthasmissing &&
> + !catt->attisdropped)
> guaranteed_column_number = attnum;
> ```
>
> When computing guaranteed_column_number, I think we can just skip the virtual generated column rather than break. Using the test from Tom’s email:
Yeah, I was confused at first as I'd done a similar optimisation in
the non-JIT deform code, but there "guaranteed" means guaranteed to be
present in the tuple data, whereas with the JIT code it means
guaranteed in the tuple data or its NULL bitmap.
I've attached v2 which includes a test that exercises deforming with
tuples which have various natts counts. I propose to backpatch
1f7dfe8c8 to v18 before applying the attached to master and v18.
David
diff --git a/src/backend/jit/llvm/llvmjit_deform.c b/src/backend/jit/llvm/llvmjit_deform.c
index 12521e3e46a..3e4ecda7b00 100644
--- a/src/backend/jit/llvm/llvmjit_deform.c
+++ b/src/backend/jit/llvm/llvmjit_deform.c
@@ -104,27 +104,32 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
funcname = llvm_expand_funcname(context, "deform");
/*
- * Check which columns have to exist, so we don't have to check the row's
- * natts unnecessarily.
+ * Check which columns have to exist in all tuples, so we don't have to
+ * check the row's natts unnecessarily.
*/
for (attnum = 0; attnum < desc->natts; attnum++)
{
- CompactAttribute *att = TupleDescCompactAttr(desc, attnum);
+ CompactAttribute *catt = TupleDescCompactAttr(desc, attnum);
+ Form_pg_attribute attr = TupleDescAttr(desc, attnum);
/*
* If the column is declared NOT NULL then it must be present in every
* tuple, unless there's a "missing" entry that could provide a
* non-NULL value for it. That in turn guarantees that the NULL bitmap
* - if there are any NULLable columns - is at least long enough to
- * cover columns up to attnum.
+ * cover columns up to attnum. We treat virtual generated columns
+ * similar to atthasmissing columns, as these columns could either not
+ * be represented in the tuple or could have the column represented as
+ * a NULL in the null bitmap.
*
* Be paranoid and also check !attisdropped, even though the
* combination of attisdropped && attnotnull combination shouldn't
* exist.
*/
- if (att->attnullability == ATTNULLABLE_VALID &&
- !att->atthasmissing &&
- !att->attisdropped)
+ if (catt->attnullability == ATTNULLABLE_VALID &&
+ !catt->atthasmissing &&
+ !catt->attisdropped &&
+ attr->attgenerated != ATTRIBUTE_GENERATED_VIRTUAL)
guaranteed_column_number = attnum;
}
@@ -392,6 +397,8 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
for (attnum = 0; attnum < natts; attnum++)
{
CompactAttribute *att = TupleDescCompactAttr(desc, attnum);
+ Form_pg_attribute attr = TupleDescAttr(desc, attnum);
+
LLVMValueRef v_incby;
int alignto = att->attalignby;
LLVMValueRef l_attno = l_int16_const(lc, attnum);
@@ -435,8 +442,11 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
* Check for nulls if necessary. No need to take missing attributes
* into account, because if they're present the heaptuple's natts
* would have indicated that a slot_getmissingattrs() is needed.
+ * When present in the tuple, virtual generated columns are always
+ * stored as NULL, so we must always perform NULL checks for these.
*/
- if (att->attnullability != ATTNULLABLE_VALID)
+ if (att->attnullability != ATTNULLABLE_VALID ||
+ attr->attgenerated == ATTRIBUTE_GENERATED_VIRTUAL)
{
LLVMBasicBlockRef b_ifnotnull;
LLVMBasicBlockRef b_ifnull;
@@ -614,12 +624,14 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
known_alignment += att->attlen;
}
else if (att->attnullability == ATTNULLABLE_VALID &&
+ attr->attgenerated != ATTRIBUTE_GENERATED_VIRTUAL &&
(att->attlen % alignto) == 0)
{
/*
- * After a NOT NULL fixed-width column with a length that is a
- * multiple of its alignment requirement, we know the following
- * column is aligned to at least the current column's alignment.
+ * After a NOT NULL (and not virtual generated) fixed-width column
+ * with a length that is a multiple of its alignment requirement,
+ * we know the following column is aligned to at least the current
+ * column's alignment.
*/
Assert(att->attlen > 0);
known_alignment = alignto;
diff --git a/src/test/regress/expected/generated_stored.out b/src/test/regress/expected/generated_stored.out
index f87a756b231..a7fe8033fa3 100644
--- a/src/test/regress/expected/generated_stored.out
+++ b/src/test/regress/expected/generated_stored.out
@@ -726,6 +726,20 @@ INSERT INTO gtest21b (a) VALUES (0); -- ok now
--INSERT INTO gtest21c (a, c) VALUES (10, 42);
--SELECT a, b, c FROM gtest21c;
--DROP TABLE gtest21c;
+-- try adding a virtual generated column to an existing table with tuples,
+-- then try adding an atthasmissing column before adding a normal nullable
+-- column.
+--CREATE TABLE gtest21d (a int NOT NULL);
+--INSERT INTO gtest21d (a) VALUES(10);
+--ALTER TABLE gtest21d ADD COLUMN b INT GENERATED ALWAYS AS (a * 10) VIRTUAL NOT NULL;
+--SELECT * FROM gtest21c ORDER BY a;
+--INSERT INTO gtest21d (a) VALUES(20);
+--ALTER TABLE gtest21d ADD COLUMN c INT NOT NULL DEFAULT 1234;
+--SELECT * FROM gtest21c ORDER BY a;
+--ALTER TABLE gtest21d ADD COLUMN d INT;
+--INSERT INTO gtest21d (a, c, d) VALUES(30, 12345, 100);
+--SELECT * FROM gtest21c ORDER BY a;
+--DROP TABLE gtest21d;
-- not-null constraint with partitioned table
CREATE TABLE gtestnn_parent (
f1 int,
diff --git a/src/test/regress/expected/generated_virtual.out b/src/test/regress/expected/generated_virtual.out
index b8d5def44db..01ee29fee10 100644
--- a/src/test/regress/expected/generated_virtual.out
+++ b/src/test/regress/expected/generated_virtual.out
@@ -737,6 +737,38 @@ SELECT a, b, c FROM gtest21c;
(1 row)
DROP TABLE gtest21c;
+-- try adding a virtual generated column to an existing table with tuples,
+-- then try adding an atthasmissing column before adding a normal nullable
+-- column.
+CREATE TABLE gtest21d (a int NOT NULL);
+INSERT INTO gtest21d (a) VALUES(10);
+ALTER TABLE gtest21d ADD COLUMN b INT GENERATED ALWAYS AS (a * 10) VIRTUAL NOT NULL;
+SELECT * FROM gtest21d ORDER BY a;
+ a | b
+----+-----
+ 10 | 100
+(1 row)
+
+INSERT INTO gtest21d (a) VALUES(20);
+ALTER TABLE gtest21d ADD COLUMN c INT NOT NULL DEFAULT 1234;
+SELECT * FROM gtest21d ORDER BY a;
+ a | b | c
+----+-----+------
+ 10 | 100 | 1234
+ 20 | 200 | 1234
+(2 rows)
+
+ALTER TABLE gtest21d ADD COLUMN d INT;
+INSERT INTO gtest21d (a, c, d) VALUES(30, 12345, 100);
+SELECT * FROM gtest21d ORDER BY a;
+ a | b | c | d
+----+-----+-------+-----
+ 10 | 100 | 1234 |
+ 20 | 200 | 1234 |
+ 30 | 300 | 12345 | 100
+(3 rows)
+
+DROP TABLE gtest21d;
-- not-null constraint with partitioned table
CREATE TABLE gtestnn_parent (
f1 int,
diff --git a/src/test/regress/sql/generated_stored.sql b/src/test/regress/sql/generated_stored.sql
index 71b0ba6d8d7..6a4fddbfdf1 100644
--- a/src/test/regress/sql/generated_stored.sql
+++ b/src/test/regress/sql/generated_stored.sql
@@ -373,6 +373,20 @@ INSERT INTO gtest21b (a) VALUES (0); -- ok now
--INSERT INTO gtest21c (a, c) VALUES (10, 42);
--SELECT a, b, c FROM gtest21c;
--DROP TABLE gtest21c;
+-- try adding a virtual generated column to an existing table with tuples,
+-- then try adding an atthasmissing column before adding a normal nullable
+-- column.
+--CREATE TABLE gtest21d (a int NOT NULL);
+--INSERT INTO gtest21d (a) VALUES(10);
+--ALTER TABLE gtest21d ADD COLUMN b INT GENERATED ALWAYS AS (a * 10) VIRTUAL NOT NULL;
+--SELECT * FROM gtest21c ORDER BY a;
+--INSERT INTO gtest21d (a) VALUES(20);
+--ALTER TABLE gtest21d ADD COLUMN c INT NOT NULL DEFAULT 1234;
+--SELECT * FROM gtest21c ORDER BY a;
+--ALTER TABLE gtest21d ADD COLUMN d INT;
+--INSERT INTO gtest21d (a, c, d) VALUES(30, 12345, 100);
+--SELECT * FROM gtest21c ORDER BY a;
+--DROP TABLE gtest21d;
-- not-null constraint with partitioned table
CREATE TABLE gtestnn_parent (
f1 int,
diff --git a/src/test/regress/sql/generated_virtual.sql b/src/test/regress/sql/generated_virtual.sql
index 9e3dc99c71d..0cb14eb0e36 100644
--- a/src/test/regress/sql/generated_virtual.sql
+++ b/src/test/regress/sql/generated_virtual.sql
@@ -380,6 +380,21 @@ INSERT INTO gtest21c (a, c) VALUES (10, 42);
SELECT a, b, c FROM gtest21c;
DROP TABLE gtest21c;
+-- try adding a virtual generated column to an existing table with tuples,
+-- then try adding an atthasmissing column before adding a normal nullable
+-- column.
+CREATE TABLE gtest21d (a int NOT NULL);
+INSERT INTO gtest21d (a) VALUES(10);
+ALTER TABLE gtest21d ADD COLUMN b INT GENERATED ALWAYS AS (a * 10) VIRTUAL NOT NULL;
+SELECT * FROM gtest21d ORDER BY a;
+INSERT INTO gtest21d (a) VALUES(20);
+ALTER TABLE gtest21d ADD COLUMN c INT NOT NULL DEFAULT 1234;
+SELECT * FROM gtest21d ORDER BY a;
+ALTER TABLE gtest21d ADD COLUMN d INT;
+INSERT INTO gtest21d (a, c, d) VALUES(30, 12345, 100);
+SELECT * FROM gtest21d ORDER BY a;
+DROP TABLE gtest21d;
+
-- not-null constraint with partitioned table
CREATE TABLE gtestnn_parent (
f1 int,
Attachments:
[text/plain] fix_jit_deform_for_virtual_generated_cols_v2.patch (8.3K, 2-fix_jit_deform_for_virtual_generated_cols_v2.patch)
download | inline diff:
diff --git a/src/backend/jit/llvm/llvmjit_deform.c b/src/backend/jit/llvm/llvmjit_deform.c
index 12521e3e46a..3e4ecda7b00 100644
--- a/src/backend/jit/llvm/llvmjit_deform.c
+++ b/src/backend/jit/llvm/llvmjit_deform.c
@@ -104,27 +104,32 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
funcname = llvm_expand_funcname(context, "deform");
/*
- * Check which columns have to exist, so we don't have to check the row's
- * natts unnecessarily.
+ * Check which columns have to exist in all tuples, so we don't have to
+ * check the row's natts unnecessarily.
*/
for (attnum = 0; attnum < desc->natts; attnum++)
{
- CompactAttribute *att = TupleDescCompactAttr(desc, attnum);
+ CompactAttribute *catt = TupleDescCompactAttr(desc, attnum);
+ Form_pg_attribute attr = TupleDescAttr(desc, attnum);
/*
* If the column is declared NOT NULL then it must be present in every
* tuple, unless there's a "missing" entry that could provide a
* non-NULL value for it. That in turn guarantees that the NULL bitmap
* - if there are any NULLable columns - is at least long enough to
- * cover columns up to attnum.
+ * cover columns up to attnum. We treat virtual generated columns
+ * similar to atthasmissing columns, as these columns could either not
+ * be represented in the tuple or could have the column represented as
+ * a NULL in the null bitmap.
*
* Be paranoid and also check !attisdropped, even though the
* combination of attisdropped && attnotnull combination shouldn't
* exist.
*/
- if (att->attnullability == ATTNULLABLE_VALID &&
- !att->atthasmissing &&
- !att->attisdropped)
+ if (catt->attnullability == ATTNULLABLE_VALID &&
+ !catt->atthasmissing &&
+ !catt->attisdropped &&
+ attr->attgenerated != ATTRIBUTE_GENERATED_VIRTUAL)
guaranteed_column_number = attnum;
}
@@ -392,6 +397,8 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
for (attnum = 0; attnum < natts; attnum++)
{
CompactAttribute *att = TupleDescCompactAttr(desc, attnum);
+ Form_pg_attribute attr = TupleDescAttr(desc, attnum);
+
LLVMValueRef v_incby;
int alignto = att->attalignby;
LLVMValueRef l_attno = l_int16_const(lc, attnum);
@@ -435,8 +442,11 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
* Check for nulls if necessary. No need to take missing attributes
* into account, because if they're present the heaptuple's natts
* would have indicated that a slot_getmissingattrs() is needed.
+ * When present in the tuple, virtual generated columns are always
+ * stored as NULL, so we must always perform NULL checks for these.
*/
- if (att->attnullability != ATTNULLABLE_VALID)
+ if (att->attnullability != ATTNULLABLE_VALID ||
+ attr->attgenerated == ATTRIBUTE_GENERATED_VIRTUAL)
{
LLVMBasicBlockRef b_ifnotnull;
LLVMBasicBlockRef b_ifnull;
@@ -614,12 +624,14 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
known_alignment += att->attlen;
}
else if (att->attnullability == ATTNULLABLE_VALID &&
+ attr->attgenerated != ATTRIBUTE_GENERATED_VIRTUAL &&
(att->attlen % alignto) == 0)
{
/*
- * After a NOT NULL fixed-width column with a length that is a
- * multiple of its alignment requirement, we know the following
- * column is aligned to at least the current column's alignment.
+ * After a NOT NULL (and not virtual generated) fixed-width column
+ * with a length that is a multiple of its alignment requirement,
+ * we know the following column is aligned to at least the current
+ * column's alignment.
*/
Assert(att->attlen > 0);
known_alignment = alignto;
diff --git a/src/test/regress/expected/generated_stored.out b/src/test/regress/expected/generated_stored.out
index f87a756b231..a7fe8033fa3 100644
--- a/src/test/regress/expected/generated_stored.out
+++ b/src/test/regress/expected/generated_stored.out
@@ -726,6 +726,20 @@ INSERT INTO gtest21b (a) VALUES (0); -- ok now
--INSERT INTO gtest21c (a, c) VALUES (10, 42);
--SELECT a, b, c FROM gtest21c;
--DROP TABLE gtest21c;
+-- try adding a virtual generated column to an existing table with tuples,
+-- then try adding an atthasmissing column before adding a normal nullable
+-- column.
+--CREATE TABLE gtest21d (a int NOT NULL);
+--INSERT INTO gtest21d (a) VALUES(10);
+--ALTER TABLE gtest21d ADD COLUMN b INT GENERATED ALWAYS AS (a * 10) VIRTUAL NOT NULL;
+--SELECT * FROM gtest21c ORDER BY a;
+--INSERT INTO gtest21d (a) VALUES(20);
+--ALTER TABLE gtest21d ADD COLUMN c INT NOT NULL DEFAULT 1234;
+--SELECT * FROM gtest21c ORDER BY a;
+--ALTER TABLE gtest21d ADD COLUMN d INT;
+--INSERT INTO gtest21d (a, c, d) VALUES(30, 12345, 100);
+--SELECT * FROM gtest21c ORDER BY a;
+--DROP TABLE gtest21d;
-- not-null constraint with partitioned table
CREATE TABLE gtestnn_parent (
f1 int,
diff --git a/src/test/regress/expected/generated_virtual.out b/src/test/regress/expected/generated_virtual.out
index b8d5def44db..01ee29fee10 100644
--- a/src/test/regress/expected/generated_virtual.out
+++ b/src/test/regress/expected/generated_virtual.out
@@ -737,6 +737,38 @@ SELECT a, b, c FROM gtest21c;
(1 row)
DROP TABLE gtest21c;
+-- try adding a virtual generated column to an existing table with tuples,
+-- then try adding an atthasmissing column before adding a normal nullable
+-- column.
+CREATE TABLE gtest21d (a int NOT NULL);
+INSERT INTO gtest21d (a) VALUES(10);
+ALTER TABLE gtest21d ADD COLUMN b INT GENERATED ALWAYS AS (a * 10) VIRTUAL NOT NULL;
+SELECT * FROM gtest21d ORDER BY a;
+ a | b
+----+-----
+ 10 | 100
+(1 row)
+
+INSERT INTO gtest21d (a) VALUES(20);
+ALTER TABLE gtest21d ADD COLUMN c INT NOT NULL DEFAULT 1234;
+SELECT * FROM gtest21d ORDER BY a;
+ a | b | c
+----+-----+------
+ 10 | 100 | 1234
+ 20 | 200 | 1234
+(2 rows)
+
+ALTER TABLE gtest21d ADD COLUMN d INT;
+INSERT INTO gtest21d (a, c, d) VALUES(30, 12345, 100);
+SELECT * FROM gtest21d ORDER BY a;
+ a | b | c | d
+----+-----+-------+-----
+ 10 | 100 | 1234 |
+ 20 | 200 | 1234 |
+ 30 | 300 | 12345 | 100
+(3 rows)
+
+DROP TABLE gtest21d;
-- not-null constraint with partitioned table
CREATE TABLE gtestnn_parent (
f1 int,
diff --git a/src/test/regress/sql/generated_stored.sql b/src/test/regress/sql/generated_stored.sql
index 71b0ba6d8d7..6a4fddbfdf1 100644
--- a/src/test/regress/sql/generated_stored.sql
+++ b/src/test/regress/sql/generated_stored.sql
@@ -373,6 +373,20 @@ INSERT INTO gtest21b (a) VALUES (0); -- ok now
--INSERT INTO gtest21c (a, c) VALUES (10, 42);
--SELECT a, b, c FROM gtest21c;
--DROP TABLE gtest21c;
+-- try adding a virtual generated column to an existing table with tuples,
+-- then try adding an atthasmissing column before adding a normal nullable
+-- column.
+--CREATE TABLE gtest21d (a int NOT NULL);
+--INSERT INTO gtest21d (a) VALUES(10);
+--ALTER TABLE gtest21d ADD COLUMN b INT GENERATED ALWAYS AS (a * 10) VIRTUAL NOT NULL;
+--SELECT * FROM gtest21c ORDER BY a;
+--INSERT INTO gtest21d (a) VALUES(20);
+--ALTER TABLE gtest21d ADD COLUMN c INT NOT NULL DEFAULT 1234;
+--SELECT * FROM gtest21c ORDER BY a;
+--ALTER TABLE gtest21d ADD COLUMN d INT;
+--INSERT INTO gtest21d (a, c, d) VALUES(30, 12345, 100);
+--SELECT * FROM gtest21c ORDER BY a;
+--DROP TABLE gtest21d;
-- not-null constraint with partitioned table
CREATE TABLE gtestnn_parent (
f1 int,
diff --git a/src/test/regress/sql/generated_virtual.sql b/src/test/regress/sql/generated_virtual.sql
index 9e3dc99c71d..0cb14eb0e36 100644
--- a/src/test/regress/sql/generated_virtual.sql
+++ b/src/test/regress/sql/generated_virtual.sql
@@ -380,6 +380,21 @@ INSERT INTO gtest21c (a, c) VALUES (10, 42);
SELECT a, b, c FROM gtest21c;
DROP TABLE gtest21c;
+-- try adding a virtual generated column to an existing table with tuples,
+-- then try adding an atthasmissing column before adding a normal nullable
+-- column.
+CREATE TABLE gtest21d (a int NOT NULL);
+INSERT INTO gtest21d (a) VALUES(10);
+ALTER TABLE gtest21d ADD COLUMN b INT GENERATED ALWAYS AS (a * 10) VIRTUAL NOT NULL;
+SELECT * FROM gtest21d ORDER BY a;
+INSERT INTO gtest21d (a) VALUES(20);
+ALTER TABLE gtest21d ADD COLUMN c INT NOT NULL DEFAULT 1234;
+SELECT * FROM gtest21d ORDER BY a;
+ALTER TABLE gtest21d ADD COLUMN d INT;
+INSERT INTO gtest21d (a, c, d) VALUES(30, 12345, 100);
+SELECT * FROM gtest21d ORDER BY a;
+DROP TABLE gtest21d;
+
-- not-null constraint with partitioned table
CREATE TABLE gtestnn_parent (
f1 int,
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Fix tuple deformation with virtual generated NOT NULL columns
2026-06-04 05:57 Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-06 04:50 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-08 13:01 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Andres Freund <[email protected]>
2026-06-09 02:19 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-12 04:26 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 05:04 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 22:23 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Tom Lane <[email protected]>
2026-06-17 22:45 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Tom Lane <[email protected]>
2026-06-17 23:06 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 23:37 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-18 02:35 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-18 05:18 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
@ 2026-06-18 05:55 ` Chao Li <[email protected]>
2026-06-18 06:27 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: Chao Li @ 2026-06-18 05:55 UTC (permalink / raw)
To: David Rowley <[email protected]>; +Cc: Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Postgres hackers <[email protected]>
> On Jun 18, 2026, at 13:18, David Rowley <[email protected]> wrote:
>
> On Thu, 18 Jun 2026 at 14:36, Chao Li <[email protected]> wrote:
>> I tested the fix, and it seems to work. While tracing the code, I wondered about this part:
>> ```
>> - if (att->attnullability == ATTNULLABLE_VALID &&
>> - !att->atthasmissing &&
>> - !att->attisdropped)
>> + if (attr->attgenerated == ATTRIBUTE_GENERATED_VIRTUAL)
>> + break;
>> +
>> + if (catt->attnullability == ATTNULLABLE_VALID &&
>> + !catt->atthasmissing &&
>> + !catt->attisdropped)
>> guaranteed_column_number = attnum;
>> ```
>>
>> When computing guaranteed_column_number, I think we can just skip the virtual generated column rather than break. Using the test from Tom’s email:
>
> Yeah, I was confused at first as I'd done a similar optimisation in
> the non-JIT deform code, but there "guaranteed" means guaranteed to be
> present in the tuple data, whereas with the JIT code it means
> guaranteed in the tuple data or its NULL bitmap.
>
> I've attached v2 which includes a test that exercises deforming with
> tuples which have various natts counts. I propose to backpatch
> 1f7dfe8c8 to v18 before applying the attached to master and v18.
>
> David
> <fix_jit_deform_for_virtual_generated_cols_v2.patch>
This version looks good to me. Only a small comment:
```
+-- try adding a virtual generated column to an existing table with tuples,
+-- then try adding an atthasmissing column before adding a normal nullable
+-- column.
+--CREATE TABLE gtest21d (a int NOT NULL);
+--INSERT INTO gtest21d (a) VALUES(10);
+--ALTER TABLE gtest21d ADD COLUMN b INT GENERATED ALWAYS AS (a * 10) VIRTUAL NOT NULL;
+--SELECT * FROM gtest21c ORDER BY a;
+--INSERT INTO gtest21d (a) VALUES(20);
+--ALTER TABLE gtest21d ADD COLUMN c INT NOT NULL DEFAULT 1234;
+--SELECT * FROM gtest21c ORDER BY a;
+--ALTER TABLE gtest21d ADD COLUMN d INT;
+--INSERT INTO gtest21d (a, c, d) VALUES(30, 12345, 100);
+--SELECT * FROM gtest21c ORDER BY a;
+--DROP TABLE gtest21d;
```
I don’t know why you added these commented SQL statements, I guess you have your reason. The problem is, in the 3 SELECTs, gtest21c should probably be gtest21d.
Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Fix tuple deformation with virtual generated NOT NULL columns
2026-06-04 05:57 Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-06 04:50 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-08 13:01 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Andres Freund <[email protected]>
2026-06-09 02:19 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-12 04:26 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 05:04 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 22:23 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Tom Lane <[email protected]>
2026-06-17 22:45 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Tom Lane <[email protected]>
2026-06-17 23:06 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 23:37 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-18 02:35 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-18 05:18 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-18 05:55 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
@ 2026-06-18 06:27 ` David Rowley <[email protected]>
2026-06-18 06:46 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: David Rowley @ 2026-06-18 06:27 UTC (permalink / raw)
To: Chao Li <[email protected]>; +Cc: Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Postgres hackers <[email protected]>
On Thu, 18 Jun 2026 at 17:56, Chao Li <[email protected]> wrote:
> I don’t know why you added these commented SQL statements, I guess you have your reason. The problem is, in the 3 SELECTs, gtest21c should probably be gtest21d.
This relates to the "-- keep these tests aligned with
generated_stored.sql" at the top of the file.
I mentioned in [1];
> I pushed the patch to add the new test after modifying it to also
> include the change in the generated_stored test, but with it commented
> out. This is the method that 83ea6c540 introduced.
and this is meant to follow the same pattern.
I've attached a version with the commented table names fixed. Thanks
for looking.
David
[1] https://postgr.es/m/CAApHDvrJBXEhet4=Es_wHBKdv5PCV5OGCaJOSmJexeaFqfmUHA@mail.gmail.com
diff --git a/src/backend/jit/llvm/llvmjit_deform.c b/src/backend/jit/llvm/llvmjit_deform.c
index 12521e3e46a..3e4ecda7b00 100644
--- a/src/backend/jit/llvm/llvmjit_deform.c
+++ b/src/backend/jit/llvm/llvmjit_deform.c
@@ -104,27 +104,32 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
funcname = llvm_expand_funcname(context, "deform");
/*
- * Check which columns have to exist, so we don't have to check the row's
- * natts unnecessarily.
+ * Check which columns have to exist in all tuples, so we don't have to
+ * check the row's natts unnecessarily.
*/
for (attnum = 0; attnum < desc->natts; attnum++)
{
- CompactAttribute *att = TupleDescCompactAttr(desc, attnum);
+ CompactAttribute *catt = TupleDescCompactAttr(desc, attnum);
+ Form_pg_attribute attr = TupleDescAttr(desc, attnum);
/*
* If the column is declared NOT NULL then it must be present in every
* tuple, unless there's a "missing" entry that could provide a
* non-NULL value for it. That in turn guarantees that the NULL bitmap
* - if there are any NULLable columns - is at least long enough to
- * cover columns up to attnum.
+ * cover columns up to attnum. We treat virtual generated columns
+ * similar to atthasmissing columns, as these columns could either not
+ * be represented in the tuple or could have the column represented as
+ * a NULL in the null bitmap.
*
* Be paranoid and also check !attisdropped, even though the
* combination of attisdropped && attnotnull combination shouldn't
* exist.
*/
- if (att->attnullability == ATTNULLABLE_VALID &&
- !att->atthasmissing &&
- !att->attisdropped)
+ if (catt->attnullability == ATTNULLABLE_VALID &&
+ !catt->atthasmissing &&
+ !catt->attisdropped &&
+ attr->attgenerated != ATTRIBUTE_GENERATED_VIRTUAL)
guaranteed_column_number = attnum;
}
@@ -392,6 +397,8 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
for (attnum = 0; attnum < natts; attnum++)
{
CompactAttribute *att = TupleDescCompactAttr(desc, attnum);
+ Form_pg_attribute attr = TupleDescAttr(desc, attnum);
+
LLVMValueRef v_incby;
int alignto = att->attalignby;
LLVMValueRef l_attno = l_int16_const(lc, attnum);
@@ -435,8 +442,11 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
* Check for nulls if necessary. No need to take missing attributes
* into account, because if they're present the heaptuple's natts
* would have indicated that a slot_getmissingattrs() is needed.
+ * When present in the tuple, virtual generated columns are always
+ * stored as NULL, so we must always perform NULL checks for these.
*/
- if (att->attnullability != ATTNULLABLE_VALID)
+ if (att->attnullability != ATTNULLABLE_VALID ||
+ attr->attgenerated == ATTRIBUTE_GENERATED_VIRTUAL)
{
LLVMBasicBlockRef b_ifnotnull;
LLVMBasicBlockRef b_ifnull;
@@ -614,12 +624,14 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
known_alignment += att->attlen;
}
else if (att->attnullability == ATTNULLABLE_VALID &&
+ attr->attgenerated != ATTRIBUTE_GENERATED_VIRTUAL &&
(att->attlen % alignto) == 0)
{
/*
- * After a NOT NULL fixed-width column with a length that is a
- * multiple of its alignment requirement, we know the following
- * column is aligned to at least the current column's alignment.
+ * After a NOT NULL (and not virtual generated) fixed-width column
+ * with a length that is a multiple of its alignment requirement,
+ * we know the following column is aligned to at least the current
+ * column's alignment.
*/
Assert(att->attlen > 0);
known_alignment = alignto;
diff --git a/src/test/regress/expected/generated_stored.out b/src/test/regress/expected/generated_stored.out
index f87a756b231..e17ba2f4881 100644
--- a/src/test/regress/expected/generated_stored.out
+++ b/src/test/regress/expected/generated_stored.out
@@ -726,6 +726,20 @@ INSERT INTO gtest21b (a) VALUES (0); -- ok now
--INSERT INTO gtest21c (a, c) VALUES (10, 42);
--SELECT a, b, c FROM gtest21c;
--DROP TABLE gtest21c;
+-- try adding a virtual generated column to an existing table with tuples,
+-- then try adding an atthasmissing column before adding a normal nullable
+-- column.
+--CREATE TABLE gtest21d (a int NOT NULL);
+--INSERT INTO gtest21d (a) VALUES(10);
+--ALTER TABLE gtest21d ADD COLUMN b INT GENERATED ALWAYS AS (a * 10) VIRTUAL NOT NULL;
+--SELECT * FROM gtest21d ORDER BY a;
+--INSERT INTO gtest21d (a) VALUES(20);
+--ALTER TABLE gtest21d ADD COLUMN c INT NOT NULL DEFAULT 1234;
+--SELECT * FROM gtest21d ORDER BY a;
+--ALTER TABLE gtest21d ADD COLUMN d INT;
+--INSERT INTO gtest21d (a, c, d) VALUES(30, 12345, 100);
+--SELECT * FROM gtest21d ORDER BY a;
+--DROP TABLE gtest21d;
-- not-null constraint with partitioned table
CREATE TABLE gtestnn_parent (
f1 int,
diff --git a/src/test/regress/expected/generated_virtual.out b/src/test/regress/expected/generated_virtual.out
index b8d5def44db..01ee29fee10 100644
--- a/src/test/regress/expected/generated_virtual.out
+++ b/src/test/regress/expected/generated_virtual.out
@@ -737,6 +737,38 @@ SELECT a, b, c FROM gtest21c;
(1 row)
DROP TABLE gtest21c;
+-- try adding a virtual generated column to an existing table with tuples,
+-- then try adding an atthasmissing column before adding a normal nullable
+-- column.
+CREATE TABLE gtest21d (a int NOT NULL);
+INSERT INTO gtest21d (a) VALUES(10);
+ALTER TABLE gtest21d ADD COLUMN b INT GENERATED ALWAYS AS (a * 10) VIRTUAL NOT NULL;
+SELECT * FROM gtest21d ORDER BY a;
+ a | b
+----+-----
+ 10 | 100
+(1 row)
+
+INSERT INTO gtest21d (a) VALUES(20);
+ALTER TABLE gtest21d ADD COLUMN c INT NOT NULL DEFAULT 1234;
+SELECT * FROM gtest21d ORDER BY a;
+ a | b | c
+----+-----+------
+ 10 | 100 | 1234
+ 20 | 200 | 1234
+(2 rows)
+
+ALTER TABLE gtest21d ADD COLUMN d INT;
+INSERT INTO gtest21d (a, c, d) VALUES(30, 12345, 100);
+SELECT * FROM gtest21d ORDER BY a;
+ a | b | c | d
+----+-----+-------+-----
+ 10 | 100 | 1234 |
+ 20 | 200 | 1234 |
+ 30 | 300 | 12345 | 100
+(3 rows)
+
+DROP TABLE gtest21d;
-- not-null constraint with partitioned table
CREATE TABLE gtestnn_parent (
f1 int,
diff --git a/src/test/regress/sql/generated_stored.sql b/src/test/regress/sql/generated_stored.sql
index 71b0ba6d8d7..85b6212023d 100644
--- a/src/test/regress/sql/generated_stored.sql
+++ b/src/test/regress/sql/generated_stored.sql
@@ -373,6 +373,20 @@ INSERT INTO gtest21b (a) VALUES (0); -- ok now
--INSERT INTO gtest21c (a, c) VALUES (10, 42);
--SELECT a, b, c FROM gtest21c;
--DROP TABLE gtest21c;
+-- try adding a virtual generated column to an existing table with tuples,
+-- then try adding an atthasmissing column before adding a normal nullable
+-- column.
+--CREATE TABLE gtest21d (a int NOT NULL);
+--INSERT INTO gtest21d (a) VALUES(10);
+--ALTER TABLE gtest21d ADD COLUMN b INT GENERATED ALWAYS AS (a * 10) VIRTUAL NOT NULL;
+--SELECT * FROM gtest21d ORDER BY a;
+--INSERT INTO gtest21d (a) VALUES(20);
+--ALTER TABLE gtest21d ADD COLUMN c INT NOT NULL DEFAULT 1234;
+--SELECT * FROM gtest21d ORDER BY a;
+--ALTER TABLE gtest21d ADD COLUMN d INT;
+--INSERT INTO gtest21d (a, c, d) VALUES(30, 12345, 100);
+--SELECT * FROM gtest21d ORDER BY a;
+--DROP TABLE gtest21d;
-- not-null constraint with partitioned table
CREATE TABLE gtestnn_parent (
f1 int,
diff --git a/src/test/regress/sql/generated_virtual.sql b/src/test/regress/sql/generated_virtual.sql
index 9e3dc99c71d..0cb14eb0e36 100644
--- a/src/test/regress/sql/generated_virtual.sql
+++ b/src/test/regress/sql/generated_virtual.sql
@@ -380,6 +380,21 @@ INSERT INTO gtest21c (a, c) VALUES (10, 42);
SELECT a, b, c FROM gtest21c;
DROP TABLE gtest21c;
+-- try adding a virtual generated column to an existing table with tuples,
+-- then try adding an atthasmissing column before adding a normal nullable
+-- column.
+CREATE TABLE gtest21d (a int NOT NULL);
+INSERT INTO gtest21d (a) VALUES(10);
+ALTER TABLE gtest21d ADD COLUMN b INT GENERATED ALWAYS AS (a * 10) VIRTUAL NOT NULL;
+SELECT * FROM gtest21d ORDER BY a;
+INSERT INTO gtest21d (a) VALUES(20);
+ALTER TABLE gtest21d ADD COLUMN c INT NOT NULL DEFAULT 1234;
+SELECT * FROM gtest21d ORDER BY a;
+ALTER TABLE gtest21d ADD COLUMN d INT;
+INSERT INTO gtest21d (a, c, d) VALUES(30, 12345, 100);
+SELECT * FROM gtest21d ORDER BY a;
+DROP TABLE gtest21d;
+
-- not-null constraint with partitioned table
CREATE TABLE gtestnn_parent (
f1 int,
Attachments:
[text/plain] fix_jit_deform_for_virtual_generated_cols_v3.patch (8.3K, 2-fix_jit_deform_for_virtual_generated_cols_v3.patch)
download | inline diff:
diff --git a/src/backend/jit/llvm/llvmjit_deform.c b/src/backend/jit/llvm/llvmjit_deform.c
index 12521e3e46a..3e4ecda7b00 100644
--- a/src/backend/jit/llvm/llvmjit_deform.c
+++ b/src/backend/jit/llvm/llvmjit_deform.c
@@ -104,27 +104,32 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
funcname = llvm_expand_funcname(context, "deform");
/*
- * Check which columns have to exist, so we don't have to check the row's
- * natts unnecessarily.
+ * Check which columns have to exist in all tuples, so we don't have to
+ * check the row's natts unnecessarily.
*/
for (attnum = 0; attnum < desc->natts; attnum++)
{
- CompactAttribute *att = TupleDescCompactAttr(desc, attnum);
+ CompactAttribute *catt = TupleDescCompactAttr(desc, attnum);
+ Form_pg_attribute attr = TupleDescAttr(desc, attnum);
/*
* If the column is declared NOT NULL then it must be present in every
* tuple, unless there's a "missing" entry that could provide a
* non-NULL value for it. That in turn guarantees that the NULL bitmap
* - if there are any NULLable columns - is at least long enough to
- * cover columns up to attnum.
+ * cover columns up to attnum. We treat virtual generated columns
+ * similar to atthasmissing columns, as these columns could either not
+ * be represented in the tuple or could have the column represented as
+ * a NULL in the null bitmap.
*
* Be paranoid and also check !attisdropped, even though the
* combination of attisdropped && attnotnull combination shouldn't
* exist.
*/
- if (att->attnullability == ATTNULLABLE_VALID &&
- !att->atthasmissing &&
- !att->attisdropped)
+ if (catt->attnullability == ATTNULLABLE_VALID &&
+ !catt->atthasmissing &&
+ !catt->attisdropped &&
+ attr->attgenerated != ATTRIBUTE_GENERATED_VIRTUAL)
guaranteed_column_number = attnum;
}
@@ -392,6 +397,8 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
for (attnum = 0; attnum < natts; attnum++)
{
CompactAttribute *att = TupleDescCompactAttr(desc, attnum);
+ Form_pg_attribute attr = TupleDescAttr(desc, attnum);
+
LLVMValueRef v_incby;
int alignto = att->attalignby;
LLVMValueRef l_attno = l_int16_const(lc, attnum);
@@ -435,8 +442,11 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
* Check for nulls if necessary. No need to take missing attributes
* into account, because if they're present the heaptuple's natts
* would have indicated that a slot_getmissingattrs() is needed.
+ * When present in the tuple, virtual generated columns are always
+ * stored as NULL, so we must always perform NULL checks for these.
*/
- if (att->attnullability != ATTNULLABLE_VALID)
+ if (att->attnullability != ATTNULLABLE_VALID ||
+ attr->attgenerated == ATTRIBUTE_GENERATED_VIRTUAL)
{
LLVMBasicBlockRef b_ifnotnull;
LLVMBasicBlockRef b_ifnull;
@@ -614,12 +624,14 @@ slot_compile_deform(LLVMJitContext *context, TupleDesc desc,
known_alignment += att->attlen;
}
else if (att->attnullability == ATTNULLABLE_VALID &&
+ attr->attgenerated != ATTRIBUTE_GENERATED_VIRTUAL &&
(att->attlen % alignto) == 0)
{
/*
- * After a NOT NULL fixed-width column with a length that is a
- * multiple of its alignment requirement, we know the following
- * column is aligned to at least the current column's alignment.
+ * After a NOT NULL (and not virtual generated) fixed-width column
+ * with a length that is a multiple of its alignment requirement,
+ * we know the following column is aligned to at least the current
+ * column's alignment.
*/
Assert(att->attlen > 0);
known_alignment = alignto;
diff --git a/src/test/regress/expected/generated_stored.out b/src/test/regress/expected/generated_stored.out
index f87a756b231..e17ba2f4881 100644
--- a/src/test/regress/expected/generated_stored.out
+++ b/src/test/regress/expected/generated_stored.out
@@ -726,6 +726,20 @@ INSERT INTO gtest21b (a) VALUES (0); -- ok now
--INSERT INTO gtest21c (a, c) VALUES (10, 42);
--SELECT a, b, c FROM gtest21c;
--DROP TABLE gtest21c;
+-- try adding a virtual generated column to an existing table with tuples,
+-- then try adding an atthasmissing column before adding a normal nullable
+-- column.
+--CREATE TABLE gtest21d (a int NOT NULL);
+--INSERT INTO gtest21d (a) VALUES(10);
+--ALTER TABLE gtest21d ADD COLUMN b INT GENERATED ALWAYS AS (a * 10) VIRTUAL NOT NULL;
+--SELECT * FROM gtest21d ORDER BY a;
+--INSERT INTO gtest21d (a) VALUES(20);
+--ALTER TABLE gtest21d ADD COLUMN c INT NOT NULL DEFAULT 1234;
+--SELECT * FROM gtest21d ORDER BY a;
+--ALTER TABLE gtest21d ADD COLUMN d INT;
+--INSERT INTO gtest21d (a, c, d) VALUES(30, 12345, 100);
+--SELECT * FROM gtest21d ORDER BY a;
+--DROP TABLE gtest21d;
-- not-null constraint with partitioned table
CREATE TABLE gtestnn_parent (
f1 int,
diff --git a/src/test/regress/expected/generated_virtual.out b/src/test/regress/expected/generated_virtual.out
index b8d5def44db..01ee29fee10 100644
--- a/src/test/regress/expected/generated_virtual.out
+++ b/src/test/regress/expected/generated_virtual.out
@@ -737,6 +737,38 @@ SELECT a, b, c FROM gtest21c;
(1 row)
DROP TABLE gtest21c;
+-- try adding a virtual generated column to an existing table with tuples,
+-- then try adding an atthasmissing column before adding a normal nullable
+-- column.
+CREATE TABLE gtest21d (a int NOT NULL);
+INSERT INTO gtest21d (a) VALUES(10);
+ALTER TABLE gtest21d ADD COLUMN b INT GENERATED ALWAYS AS (a * 10) VIRTUAL NOT NULL;
+SELECT * FROM gtest21d ORDER BY a;
+ a | b
+----+-----
+ 10 | 100
+(1 row)
+
+INSERT INTO gtest21d (a) VALUES(20);
+ALTER TABLE gtest21d ADD COLUMN c INT NOT NULL DEFAULT 1234;
+SELECT * FROM gtest21d ORDER BY a;
+ a | b | c
+----+-----+------
+ 10 | 100 | 1234
+ 20 | 200 | 1234
+(2 rows)
+
+ALTER TABLE gtest21d ADD COLUMN d INT;
+INSERT INTO gtest21d (a, c, d) VALUES(30, 12345, 100);
+SELECT * FROM gtest21d ORDER BY a;
+ a | b | c | d
+----+-----+-------+-----
+ 10 | 100 | 1234 |
+ 20 | 200 | 1234 |
+ 30 | 300 | 12345 | 100
+(3 rows)
+
+DROP TABLE gtest21d;
-- not-null constraint with partitioned table
CREATE TABLE gtestnn_parent (
f1 int,
diff --git a/src/test/regress/sql/generated_stored.sql b/src/test/regress/sql/generated_stored.sql
index 71b0ba6d8d7..85b6212023d 100644
--- a/src/test/regress/sql/generated_stored.sql
+++ b/src/test/regress/sql/generated_stored.sql
@@ -373,6 +373,20 @@ INSERT INTO gtest21b (a) VALUES (0); -- ok now
--INSERT INTO gtest21c (a, c) VALUES (10, 42);
--SELECT a, b, c FROM gtest21c;
--DROP TABLE gtest21c;
+-- try adding a virtual generated column to an existing table with tuples,
+-- then try adding an atthasmissing column before adding a normal nullable
+-- column.
+--CREATE TABLE gtest21d (a int NOT NULL);
+--INSERT INTO gtest21d (a) VALUES(10);
+--ALTER TABLE gtest21d ADD COLUMN b INT GENERATED ALWAYS AS (a * 10) VIRTUAL NOT NULL;
+--SELECT * FROM gtest21d ORDER BY a;
+--INSERT INTO gtest21d (a) VALUES(20);
+--ALTER TABLE gtest21d ADD COLUMN c INT NOT NULL DEFAULT 1234;
+--SELECT * FROM gtest21d ORDER BY a;
+--ALTER TABLE gtest21d ADD COLUMN d INT;
+--INSERT INTO gtest21d (a, c, d) VALUES(30, 12345, 100);
+--SELECT * FROM gtest21d ORDER BY a;
+--DROP TABLE gtest21d;
-- not-null constraint with partitioned table
CREATE TABLE gtestnn_parent (
f1 int,
diff --git a/src/test/regress/sql/generated_virtual.sql b/src/test/regress/sql/generated_virtual.sql
index 9e3dc99c71d..0cb14eb0e36 100644
--- a/src/test/regress/sql/generated_virtual.sql
+++ b/src/test/regress/sql/generated_virtual.sql
@@ -380,6 +380,21 @@ INSERT INTO gtest21c (a, c) VALUES (10, 42);
SELECT a, b, c FROM gtest21c;
DROP TABLE gtest21c;
+-- try adding a virtual generated column to an existing table with tuples,
+-- then try adding an atthasmissing column before adding a normal nullable
+-- column.
+CREATE TABLE gtest21d (a int NOT NULL);
+INSERT INTO gtest21d (a) VALUES(10);
+ALTER TABLE gtest21d ADD COLUMN b INT GENERATED ALWAYS AS (a * 10) VIRTUAL NOT NULL;
+SELECT * FROM gtest21d ORDER BY a;
+INSERT INTO gtest21d (a) VALUES(20);
+ALTER TABLE gtest21d ADD COLUMN c INT NOT NULL DEFAULT 1234;
+SELECT * FROM gtest21d ORDER BY a;
+ALTER TABLE gtest21d ADD COLUMN d INT;
+INSERT INTO gtest21d (a, c, d) VALUES(30, 12345, 100);
+SELECT * FROM gtest21d ORDER BY a;
+DROP TABLE gtest21d;
+
-- not-null constraint with partitioned table
CREATE TABLE gtestnn_parent (
f1 int,
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Fix tuple deformation with virtual generated NOT NULL columns
2026-06-04 05:57 Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-06 04:50 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-08 13:01 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Andres Freund <[email protected]>
2026-06-09 02:19 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-12 04:26 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 05:04 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 22:23 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Tom Lane <[email protected]>
2026-06-17 22:45 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Tom Lane <[email protected]>
2026-06-17 23:06 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 23:37 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-18 02:35 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-18 05:18 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-18 05:55 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-18 06:27 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
@ 2026-06-18 06:46 ` Chao Li <[email protected]>
2026-06-19 03:30 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: Chao Li @ 2026-06-18 06:46 UTC (permalink / raw)
To: David Rowley <[email protected]>; +Cc: Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Postgres hackers <[email protected]>
> On Jun 18, 2026, at 14:27, David Rowley <[email protected]> wrote:
>
> On Thu, 18 Jun 2026 at 17:56, Chao Li <[email protected]> wrote:
>> I don’t know why you added these commented SQL statements, I guess you have your reason. The problem is, in the 3 SELECTs, gtest21c should probably be gtest21d.
>
> This relates to the "-- keep these tests aligned with
> generated_stored.sql" at the top of the file.
>
> I mentioned in [1];
>> I pushed the patch to add the new test after modifying it to also
>> include the change in the generated_stored test, but with it commented
>> out. This is the method that 83ea6c540 introduced.
>
> and this is meant to follow the same pattern.
Thanks for the explanation.
>
> I've attached a version with the commented table names fixed. Thanks
> for looking.
>
> David
>
> [1] https://postgr.es/m/CAApHDvrJBXEhet4=Es_wHBKdv5PCV5OGCaJOSmJexeaFqfmUHA@mail.gmail.com
> <fix_jit_deform_for_virtual_generated_cols_v3.patch>
This version feels all set.
I also tried to build and run tests, all passed.
Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Fix tuple deformation with virtual generated NOT NULL columns
2026-06-04 05:57 Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-06 04:50 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-08 13:01 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Andres Freund <[email protected]>
2026-06-09 02:19 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-12 04:26 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 05:04 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 22:23 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Tom Lane <[email protected]>
2026-06-17 22:45 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Tom Lane <[email protected]>
2026-06-17 23:06 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 23:37 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-18 02:35 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-18 05:18 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-18 05:55 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-18 06:27 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-18 06:46 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
@ 2026-06-19 03:30 ` David Rowley <[email protected]>
2026-06-19 03:34 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Tom Lane <[email protected]>
0 siblings, 1 reply; 22+ messages in thread
From: David Rowley @ 2026-06-19 03:30 UTC (permalink / raw)
To: Chao Li <[email protected]>; +Cc: Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Postgres hackers <[email protected]>
On Thu, 18 Jun 2026 at 18:46, Chao Li <[email protected]> wrote:
> This version feels all set.
Thanks for looking. I've pushed and backpatched this now.
For the v18 version, I included the tests added by 1f7dfe8c8 in the
same commit. I did it that way so as not to briefly introduce a test
that would fail into REL_18_STABLE.
David
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Fix tuple deformation with virtual generated NOT NULL columns
2026-06-04 05:57 Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-06 04:50 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-08 13:01 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Andres Freund <[email protected]>
2026-06-09 02:19 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-12 04:26 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 05:04 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 22:23 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Tom Lane <[email protected]>
2026-06-17 22:45 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Tom Lane <[email protected]>
2026-06-17 23:06 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-17 23:37 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-18 02:35 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-18 05:18 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-18 05:55 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-18 06:27 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
2026-06-18 06:46 ` Re: Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-19 03:30 ` Re: Fix tuple deformation with virtual generated NOT NULL columns David Rowley <[email protected]>
@ 2026-06-19 03:34 ` Tom Lane <[email protected]>
0 siblings, 0 replies; 22+ messages in thread
From: Tom Lane @ 2026-06-19 03:34 UTC (permalink / raw)
To: David Rowley <[email protected]>; +Cc: Chao Li <[email protected]>; Peter Eisentraut <[email protected]>; Andres Freund <[email protected]>; Postgres hackers <[email protected]>
David Rowley <[email protected]> writes:
> For the v18 version, I included the tests added by 1f7dfe8c8 in the
> same commit. I did it that way so as not to briefly introduce a test
> that would fail into REL_18_STABLE.
+1, doing it the other way would create a landmine for git-bisect
testing.
regards, tom lane
^ permalink raw reply [nested|flat] 22+ messages in thread
end of thread, other threads:[~2026-06-19 03:34 UTC | newest]
Thread overview: 22+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-06-04 05:57 Fix tuple deformation with virtual generated NOT NULL columns Chao Li <[email protected]>
2026-06-04 09:32 ` =?utf-8?B?Y2NhNTUwNw==?= <[email protected]>
2026-06-04 10:30 ` Chao Li <[email protected]>
2026-06-05 08:52 ` =?utf-8?B?Y2NhNTUwNw==?= <[email protected]>
2026-06-06 04:50 ` David Rowley <[email protected]>
2026-06-07 01:43 ` Chao Li <[email protected]>
2026-06-08 13:01 ` Andres Freund <[email protected]>
2026-06-09 02:19 ` Chao Li <[email protected]>
2026-06-12 04:26 ` David Rowley <[email protected]>
2026-06-17 05:04 ` David Rowley <[email protected]>
2026-06-17 06:11 ` Tom Lane <[email protected]>
2026-06-17 22:23 ` Tom Lane <[email protected]>
2026-06-17 22:45 ` Tom Lane <[email protected]>
2026-06-17 23:06 ` David Rowley <[email protected]>
2026-06-17 23:37 ` David Rowley <[email protected]>
2026-06-18 02:35 ` Chao Li <[email protected]>
2026-06-18 05:18 ` David Rowley <[email protected]>
2026-06-18 05:55 ` Chao Li <[email protected]>
2026-06-18 06:27 ` David Rowley <[email protected]>
2026-06-18 06:46 ` Chao Li <[email protected]>
2026-06-19 03:30 ` David Rowley <[email protected]>
2026-06-19 03:34 ` Tom Lane <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox