public inbox for [email protected]
help / color / mirror / Atom feedCOPY FROM ON_ERROR SET_NULL bypasses domain NOT NULL with partial column list
5+ messages / 4 participants
[nested] [flat]
* COPY FROM ON_ERROR SET_NULL bypasses domain NOT NULL with partial column list
@ 2026-04-16 17:09 SATYANARAYANA NARLAPURAM <[email protected]>
2026-04-17 03:58 ` Re: COPY FROM ON_ERROR SET_NULL bypasses domain NOT NULL with partial column list jian he <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: SATYANARAYANA NARLAPURAM @ 2026-04-16 17:09 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>; +Cc: jian he <[email protected]>
HI hackers,
domain_with_constraint[] was allocated with list_length(attnumlist)
elements and indexed sequentially via foreach_current_index(), but
copyfromparse.c accesses it via attnum - 1 (physical attribute index).
With a partial column list targeting high-numbered columns, this caused
an out-of-bounds read that bypassed domain NOT NULL checks, silently
inserting NULL into NOT NULL domain columns.
Fix by allocating with num_phys_attrs and indexing by attnum - 1,
consistent with all other per-column arrays in BeginCopyFrom().
Patch is attached, and added a new test case to cover this scenario.
Repro:
CREATE DOMAIN d_notnull_int AS int NOT NULL;
CREATE TABLE t (
c1 text, c2 text, c3 text, c4 text, c5 text,
c6 text, c7 text, c8 text, c9 text,
c10 d_notnull_int
);
COPY t(c1, c10) FROM stdin WITH (on_error set_null);
hello bad
\.
SELECT c10 IS NULL FROM t;
Thanks,
Satya
Attachments:
[application/octet-stream] 0001-Fix-COPY-FROM-ON_ERROR-SET_NULL-OOB-read-with-partia.patch (5.5K, 3-0001-Fix-COPY-FROM-ON_ERROR-SET_NULL-OOB-read-with-partia.patch)
download | inline diff:
From ff9e5823164714e14cba5e70099c051bc2571e70 Mon Sep 17 00:00:00 2001
From: Satya Narlapuram <[email protected]>
Date: Thu, 16 Apr 2026 16:43:34 +0000
Subject: [PATCH] Fix COPY FROM ON_ERROR SET_NULL OOB read with partial column
list
domain_with_constraint[] was allocated with list_length(attnumlist)
elements and indexed sequentially via foreach_current_index(), but
copyfromparse.c accesses it via attnum - 1 (physical attribute index).
With a partial column list targeting high-numbered columns, this caused
an out-of-bounds read that bypassed domain NOT NULL checks, silently
inserting NULL into NOT NULL domain columns.
Fix by allocating with num_phys_attrs and indexing by attnum - 1,
consistent with all other per-column arrays in BeginCopyFrom().
---
src/backend/commands/copyfrom.c | 8 ++------
src/test/regress/expected/copy2.out | 23 +++++++++++++++++++++++
src/test/regress/sql/copy2.sql | 19 +++++++++++++++++++
3 files changed, 44 insertions(+), 6 deletions(-)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 64ac3063..0087585b 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -1636,8 +1636,6 @@ BeginCopyFrom(ParseState *pstate,
if (cstate->opts.on_error == COPY_ON_ERROR_SET_NULL)
{
- int attr_count = list_length(cstate->attnumlist);
-
/*
* When data type conversion fails and ON_ERROR is SET_NULL, we need
* ensure that the input column allow null values. ExecConstraints()
@@ -1646,15 +1644,13 @@ BeginCopyFrom(ParseState *pstate,
* check must be performed during the initial string-to-datum
* conversion (see CopyFromTextLikeOneRow()).
*/
- cstate->domain_with_constraint = palloc0_array(bool, attr_count);
+ cstate->domain_with_constraint = palloc0_array(bool, num_phys_attrs);
foreach_int(attno, cstate->attnumlist)
{
- int i = foreach_current_index(attno);
-
Form_pg_attribute att = TupleDescAttr(tupDesc, attno - 1);
- cstate->domain_with_constraint[i] = DomainHasConstraints(att->atttypid, NULL);
+ cstate->domain_with_constraint[attno - 1] = DomainHasConstraints(att->atttypid, NULL);
}
}
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index 7600e523..2d6bfdd8 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -839,6 +839,28 @@ SELECT * FROM t_on_error_null ORDER BY a;
13 | 14 | NULL
(3 rows)
+-- Test on_error set_null with partial column list and domain NOT NULL.
+-- Bug: domain_with_constraint[] was indexed by physical attnum, but
+-- allocated with only as many elements as the COPY column list.
+-- With a partial column list targeting a high-numbered column, this
+-- caused an out-of-bounds read that bypassed the domain NOT NULL check.
+CREATE TABLE t_on_error_null_partial (
+ c1 text, c2 text, c3 text, c4 text, c5 text,
+ c6 text, c7 text, c8 text, c9 text,
+ c10 d_int_not_null -- attnum 10, domain NOT NULL
+);
+-- Partial column list: attnumlist = {1, 10}. Should fail because
+-- the domain on c10 does not allow null, and 'bad' is not an integer.
+COPY t_on_error_null_partial(c1, c10) FROM STDIN WITH (on_error set_null); -- fail
+ERROR: domain d_int_not_null does not allow null values
+DETAIL: ON_ERROR SET_NULL cannot be applied because column "c10" (domain d_int_not_null) does not accept null values.
+CONTEXT: COPY t_on_error_null_partial, line 1, column c10: "bad"
+SELECT count(*) AS "expect_0" FROM t_on_error_null_partial;
+ expect_0
+----------
+ 0
+(1 row)
+
\pset null ''
-- tests for on_error option with log_verbosity and null constraint via domain
CREATE DOMAIN dcheck_ign_err2 varchar(15) NOT NULL;
@@ -900,6 +922,7 @@ DROP VIEW instead_of_insert_tbl_view_2;
DROP FUNCTION fun_instead_of_insert_tbl();
DROP TABLE check_ign_err;
DROP TABLE t_on_error_null;
+DROP TABLE t_on_error_null_partial;
DROP DOMAIN d_int_not_null;
DROP DOMAIN d_int_positive_maybe_null;
DROP TABLE check_ign_err2;
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index e0810109..287c8fb8 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -580,6 +580,24 @@ COPY t_on_error_null FROM STDIN WITH (on_error set_null, log_verbosity verbose);
SELECT * FROM t_on_error_null ORDER BY a;
+-- Test on_error set_null with partial column list and domain NOT NULL.
+-- Bug: domain_with_constraint[] was indexed by physical attnum, but
+-- allocated with only as many elements as the COPY column list.
+-- With a partial column list targeting a high-numbered column, this
+-- caused an out-of-bounds read that bypassed the domain NOT NULL check.
+CREATE TABLE t_on_error_null_partial (
+ c1 text, c2 text, c3 text, c4 text, c5 text,
+ c6 text, c7 text, c8 text, c9 text,
+ c10 d_int_not_null -- attnum 10, domain NOT NULL
+);
+-- Partial column list: attnumlist = {1, 10}. Should fail because
+-- the domain on c10 does not allow null, and 'bad' is not an integer.
+COPY t_on_error_null_partial(c1, c10) FROM STDIN WITH (on_error set_null); -- fail
+hello bad
+\.
+
+SELECT count(*) AS "expect_0" FROM t_on_error_null_partial;
+
\pset null ''
-- tests for on_error option with log_verbosity and null constraint via domain
@@ -652,6 +670,7 @@ DROP VIEW instead_of_insert_tbl_view_2;
DROP FUNCTION fun_instead_of_insert_tbl();
DROP TABLE check_ign_err;
DROP TABLE t_on_error_null;
+DROP TABLE t_on_error_null_partial;
DROP DOMAIN d_int_not_null;
DROP DOMAIN d_int_positive_maybe_null;
DROP TABLE check_ign_err2;
--
2.43.0
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: COPY FROM ON_ERROR SET_NULL bypasses domain NOT NULL with partial column list
2026-04-16 17:09 COPY FROM ON_ERROR SET_NULL bypasses domain NOT NULL with partial column list SATYANARAYANA NARLAPURAM <[email protected]>
@ 2026-04-17 03:58 ` jian he <[email protected]>
2026-05-18 13:22 ` Re: COPY FROM ON_ERROR SET_NULL bypasses domain NOT NULL with partial column list Fujii Masao <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: jian he @ 2026-04-17 03:58 UTC (permalink / raw)
To: SATYANARAYANA NARLAPURAM <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On Fri, Apr 17, 2026 at 1:09 AM SATYANARAYANA NARLAPURAM
<[email protected]> wrote:
>
> HI hackers,
>
> domain_with_constraint[] was allocated with list_length(attnumlist)
> elements and indexed sequentially via foreach_current_index(), but
> copyfromparse.c accesses it via attnum - 1 (physical attribute index).
> With a partial column list targeting high-numbered columns, this caused
> an out-of-bounds read that bypassed domain NOT NULL checks, silently
> inserting NULL into NOT NULL domain columns.
>
> Fix by allocating with num_phys_attrs and indexing by attnum - 1,
> consistent with all other per-column arrays in BeginCopyFrom().
>
> Patch is attached, and added a new test case to cover this scenario.
>
The patch looks good to me.
I simplified the tests.
--
jian
https://www.enterprisedb.com/
Attachments:
[text/x-patch] v2-0001-Fix-COPY-FROM-ON_ERROR-SET_NULL-with-selective-column-list.patch (4.0K, 2-v2-0001-Fix-COPY-FROM-ON_ERROR-SET_NULL-with-selective-column-list.patch)
download | inline diff:
From df7ed84769070dcb4c47b8f90cbdf0f3acf4ddeb Mon Sep 17 00:00:00 2001
From: jian he <[email protected]>
Date: Fri, 17 Apr 2026 10:19:15 +0800
Subject: [PATCH v2 1/1] Fix COPY FROM ON_ERROR SET_NULL with selective column
list
When using COPY FROM ... ON_ERROR SET_NULL with a selective column list, the
domain_with_constraint array was incorrectly allocated based on the length of
the target column list. While the array was populated sequentially,
CopyFromTextLikeOneRow attempted to access it using the physical attribute index
(attnum - 1).
This mismatch caused out-of-bounds reads when targeting high-numbered columns,
allowing NULL values to bypass NOT NULL domain checks and be silently inserted.
Fix by allocating the array to match the total number of physical attributes
(num_phys_attrs) and indexing via attnum - 1, bringing it into alignment with
other per-column arrays in BeginCopyFrom.
Reported-by: SATYANARAYANA NARLAPURAM <[email protected]>
Author: SATYANARAYANA NARLAPURAM <[email protected]>
Reviewed-by: jian he <[email protected]>
Discussion: https://postgr.es/m/CAHg+QDdej0c0gWJi2FnbirzhgzyZNPiTwC1P5B_-dSNCzq-91A@mail.gmail.com
---
src/backend/commands/copyfrom.c | 8 ++------
src/test/regress/expected/copy2.out | 4 ++++
src/test/regress/sql/copy2.sql | 4 ++++
3 files changed, 10 insertions(+), 6 deletions(-)
diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c
index 64ac3063c61..0087585b2c4 100644
--- a/src/backend/commands/copyfrom.c
+++ b/src/backend/commands/copyfrom.c
@@ -1636,8 +1636,6 @@ BeginCopyFrom(ParseState *pstate,
if (cstate->opts.on_error == COPY_ON_ERROR_SET_NULL)
{
- int attr_count = list_length(cstate->attnumlist);
-
/*
* When data type conversion fails and ON_ERROR is SET_NULL, we need
* ensure that the input column allow null values. ExecConstraints()
@@ -1646,15 +1644,13 @@ BeginCopyFrom(ParseState *pstate,
* check must be performed during the initial string-to-datum
* conversion (see CopyFromTextLikeOneRow()).
*/
- cstate->domain_with_constraint = palloc0_array(bool, attr_count);
+ cstate->domain_with_constraint = palloc0_array(bool, num_phys_attrs);
foreach_int(attno, cstate->attnumlist)
{
- int i = foreach_current_index(attno);
-
Form_pg_attribute att = TupleDescAttr(tupDesc, attno - 1);
- cstate->domain_with_constraint[i] = DomainHasConstraints(att->atttypid, NULL);
+ cstate->domain_with_constraint[attno - 1] = DomainHasConstraints(att->atttypid, NULL);
}
}
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index 7600e5239d2..919eabd5f78 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -805,6 +805,10 @@ COPY t_on_error_null FROM STDIN WITH (on_error set_null); -- fail
ERROR: domain d_int_not_null does not allow null values
DETAIL: ON_ERROR SET_NULL cannot be applied because column "a" (domain d_int_not_null) does not accept null values.
CONTEXT: COPY t_on_error_null, line 1, column a: null input
+COPY t_on_error_null(c, a) FROM STDIN WITH (on_error set_null); -- fail
+ERROR: domain d_int_not_null does not allow null values
+DETAIL: ON_ERROR SET_NULL cannot be applied because column "a" (domain d_int_not_null) does not accept null values.
+CONTEXT: COPY t_on_error_null, line 1, column a: null input
COPY t_on_error_null FROM STDIN WITH (on_error set_null); -- fail
ERROR: domain d_int_not_null does not allow null values
DETAIL: ON_ERROR SET_NULL cannot be applied because column "a" (domain d_int_not_null) does not accept null values.
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index e0810109473..f853499021d 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -555,6 +555,10 @@ COPY t_on_error_null FROM STDIN WITH (on_error set_null); -- fail
\N 11 13
\.
+COPY t_on_error_null(c, a) FROM STDIN WITH (on_error set_null); -- fail
+11 \N
+\.
+
COPY t_on_error_null FROM STDIN WITH (on_error set_null); -- fail
ss 11 14
\.
--
2.34.1
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: COPY FROM ON_ERROR SET_NULL bypasses domain NOT NULL with partial column list
2026-04-16 17:09 COPY FROM ON_ERROR SET_NULL bypasses domain NOT NULL with partial column list SATYANARAYANA NARLAPURAM <[email protected]>
2026-04-17 03:58 ` Re: COPY FROM ON_ERROR SET_NULL bypasses domain NOT NULL with partial column list jian he <[email protected]>
@ 2026-05-18 13:22 ` Fujii Masao <[email protected]>
2026-05-18 22:27 ` Re: COPY FROM ON_ERROR SET_NULL bypasses domain NOT NULL with partial column list Chao Li <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: Fujii Masao @ 2026-05-18 13:22 UTC (permalink / raw)
To: jian he <[email protected]>; +Cc: SATYANARAYANA NARLAPURAM <[email protected]>; PostgreSQL Hackers <[email protected]>
On Fri, Apr 17, 2026 at 12:59 PM jian he <[email protected]> wrote:
>
> On Fri, Apr 17, 2026 at 1:09 AM SATYANARAYANA NARLAPURAM
> <[email protected]> wrote:
> >
> > HI hackers,
> >
> > domain_with_constraint[] was allocated with list_length(attnumlist)
> > elements and indexed sequentially via foreach_current_index(), but
> > copyfromparse.c accesses it via attnum - 1 (physical attribute index).
> > With a partial column list targeting high-numbered columns, this caused
> > an out-of-bounds read that bypassed domain NOT NULL checks, silently
> > inserting NULL into NOT NULL domain columns.
> >
> > Fix by allocating with num_phys_attrs and indexing by attnum - 1,
> > consistent with all other per-column arrays in BeginCopyFrom().
> >
> > Patch is attached, and added a new test case to cover this scenario.
> >
>
> The patch looks good to me.
> I simplified the tests.
The patch looks good to me. Barring any objections, I will commit it.
Regards,
--
Fujii Masao
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: COPY FROM ON_ERROR SET_NULL bypasses domain NOT NULL with partial column list
2026-04-16 17:09 COPY FROM ON_ERROR SET_NULL bypasses domain NOT NULL with partial column list SATYANARAYANA NARLAPURAM <[email protected]>
2026-04-17 03:58 ` Re: COPY FROM ON_ERROR SET_NULL bypasses domain NOT NULL with partial column list jian he <[email protected]>
2026-05-18 13:22 ` Re: COPY FROM ON_ERROR SET_NULL bypasses domain NOT NULL with partial column list Fujii Masao <[email protected]>
@ 2026-05-18 22:27 ` Chao Li <[email protected]>
2026-05-19 01:13 ` Re: COPY FROM ON_ERROR SET_NULL bypasses domain NOT NULL with partial column list Fujii Masao <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: Chao Li @ 2026-05-18 22:27 UTC (permalink / raw)
To: Fujii Masao <[email protected]>; +Cc: jian he <[email protected]>; SATYANARAYANA NARLAPURAM <[email protected]>; PostgreSQL Hackers <[email protected]>
> On May 18, 2026, at 21:22, Fujii Masao <[email protected]> wrote:
>
> On Fri, Apr 17, 2026 at 12:59 PM jian he <[email protected]> wrote:
>>
>> On Fri, Apr 17, 2026 at 1:09 AM SATYANARAYANA NARLAPURAM
>> <[email protected]> wrote:
>>>
>>> HI hackers,
>>>
>>> domain_with_constraint[] was allocated with list_length(attnumlist)
>>> elements and indexed sequentially via foreach_current_index(), but
>>> copyfromparse.c accesses it via attnum - 1 (physical attribute index).
>>> With a partial column list targeting high-numbered columns, this caused
>>> an out-of-bounds read that bypassed domain NOT NULL checks, silently
>>> inserting NULL into NOT NULL domain columns.
>>>
>>> Fix by allocating with num_phys_attrs and indexing by attnum - 1,
>>> consistent with all other per-column arrays in BeginCopyFrom().
>>>
>>> Patch is attached, and added a new test case to cover this scenario.
>>>
>>
>> The patch looks good to me.
>> I simplified the tests.
>
> The patch looks good to me. Barring any objections, I will commit it.
>
> Regards,
>
> --
> Fujii Masao
I also found this bug and reported it here [1]. I just reviewed v2, the code change of the fix itself is exactly the same as my version in [1]. The new test is slightly different from my version, but I think that also works.
[1] https://www.postgresql.org/message-id/8A72720A-55AC-4D41-B9DF-5610307600E6%40gmail.com
Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: COPY FROM ON_ERROR SET_NULL bypasses domain NOT NULL with partial column list
2026-04-16 17:09 COPY FROM ON_ERROR SET_NULL bypasses domain NOT NULL with partial column list SATYANARAYANA NARLAPURAM <[email protected]>
2026-04-17 03:58 ` Re: COPY FROM ON_ERROR SET_NULL bypasses domain NOT NULL with partial column list jian he <[email protected]>
2026-05-18 13:22 ` Re: COPY FROM ON_ERROR SET_NULL bypasses domain NOT NULL with partial column list Fujii Masao <[email protected]>
2026-05-18 22:27 ` Re: COPY FROM ON_ERROR SET_NULL bypasses domain NOT NULL with partial column list Chao Li <[email protected]>
@ 2026-05-19 01:13 ` Fujii Masao <[email protected]>
0 siblings, 0 replies; 5+ messages in thread
From: Fujii Masao @ 2026-05-19 01:13 UTC (permalink / raw)
To: Chao Li <[email protected]>; +Cc: jian he <[email protected]>; SATYANARAYANA NARLAPURAM <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, May 19, 2026 at 7:28 AM Chao Li <[email protected]> wrote:
> I also found this bug and reported it here [1]. I just reviewed v2, the code change of the fix itself is exactly the same as my version in [1]. The new test is slightly different from my version, but I think that also works.
Thanks for the review! I've pushed the patch.
Regards,
--
Fujii Masao
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2026-05-19 01:13 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-04-16 17:09 COPY FROM ON_ERROR SET_NULL bypasses domain NOT NULL with partial column list SATYANARAYANA NARLAPURAM <[email protected]>
2026-04-17 03:58 ` jian he <[email protected]>
2026-05-18 13:22 ` Fujii Masao <[email protected]>
2026-05-18 22:27 ` Chao Li <[email protected]>
2026-05-19 01:13 ` Fujii Masao <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox