agora inbox for pgsql-hackers@postgresql.orghelp / color / mirror / Atom feed
[PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table 324+ messages / 2 participants [nested] [flat]
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table @ 2025-07-15 10:45 Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 0 siblings, 0 replies; 324+ messages in thread From: Jehan-Guillaume de Rorthais @ 2025-07-15 10:45 UTC (permalink / raw) When a foreign table points to a partitioned table or an inheritance parent on the foreign server, a non-direct DML can affect multiple rows when only one row is intended to be affected. This happens because postgres_fdw uses only ctid to identify a row to work on. Though ctid uniquely identifies a row in a single table, in a partitioned table or in an inheritance hierarchy, there can be be multiple rows, in different partitions, with the same ctid. So DML statement sent to the foreign server by postgres_fdw ends up affecting more than one rows, only one of which is intended to be affected. This commit adds testcases to show the problem. A subsequent commit would have a fix to the problem. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://www.postgresql.org/message-id/flat/CAFjFpRfcgwsHRmpvoOK-GUQi-n8MgAS%2BOxcQo%3DaBDn1COywmcg%4... Rebased by Jehan-Guillaume de Rorthais <jgdr@dalibo.com> --- .../postgres_fdw/expected/postgres_fdw.out | 125 ++++++++++++++++++ contrib/postgres_fdw/sql/postgres_fdw.sql | 58 ++++++++ 2 files changed, 183 insertions(+) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 2185b42bb4f..62019eaa881 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -8954,6 +8954,131 @@ drop foreign table remt2; drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | aaa + fa | (0,1) | bbb +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; + QUERY PLAN +----------------------------------------------------------------------------------------------------------------- + Update on public.fa + Remote SQL: UPDATE public.a SET aa = $2 WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: CASE WHEN (random() <= '1'::double precision) THEN 'zzzz'::text ELSE NULL::text END, ctid, fa.* + Remote SQL: SELECT aa, ctid FROM public.a WHERE ((aa = 'aaa')) FOR UPDATE +(5 rows) + +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+------ + fa | (0,2) | zzzz + fa | (0,1) | bbb +(2 rows) + +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); + QUERY PLAN +--------------------------------------------------------------------------------------------------------------- + Delete on public.fa + Remote SQL: DELETE FROM public.a WHERE ctid = $1 + -> Foreign Scan on public.fa + Output: ctid + Filter: (fa.aa = CASE WHEN (random() <= '1'::double precision) THEN 'aaa'::text ELSE 'bbb'::text END) + Remote SQL: SELECT aa, ctid FROM public.a FOR UPDATE +(6 rows) + +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + tableoid | ctid | aa +----------+-------+----- + fa | (0,1) | bbb +(1 row) + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; +NOTICE: drop cascades to table b +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 1 | 1 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; + QUERY PLAN +------------------------------------------------------------------------------------------------- + Update on public.fplt + Remote SQL: UPDATE public.plt SET b = $2 WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: CASE WHEN (random() <= '1'::double precision) THEN 10 ELSE 20 END, ctid, fplt.* + Remote SQL: SELECT a, b, ctid FROM public.plt WHERE ((a = 1)) FOR UPDATE +(5 rows) + +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+---- + fplt | (0,2) | 1 | 10 + fplt | (0,1) | 2 | 2 +(2 rows) + +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); + QUERY PLAN +--------------------------------------------------------------------------------------------- + Delete on public.fplt + Remote SQL: DELETE FROM public.plt WHERE ctid = $1 + -> Foreign Scan on public.fplt + Output: ctid + Filter: (fplt.a = CASE WHEN (random() <= '1'::double precision) THEN 1 ELSE 10 END) + Remote SQL: SELECT a, ctid FROM public.plt FOR UPDATE +(6 rows) + +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + tableoid | ctid | a | b +----------+-------+---+--- + fplt | (0,1) | 2 | 2 +(1 row) + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index e534b40de3c..ae3777b7edb 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -2494,6 +2494,64 @@ drop table loct1; drop table loct2; drop table parent; +-- test DML statement on a foreign table pointing to an inheritance hierarchy +-- on the remote server +CREATE TABLE a(aa TEXT); +ALTER TABLE a SET (autovacuum_enabled = 'false'); +CREATE TABLE b() INHERITS(a); +ALTER TABLE b SET (autovacuum_enabled = 'false'); +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +CREATE FOREIGN TABLE fa (aa TEXT) SERVER loopback OPTIONS (table_name 'a'); + +SELECT tableoid::regclass, ctid, * FROM fa; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +UPDATE fa SET aa = (CASE WHEN random() <= 1 THEN 'zzzz' ELSE NULL END) WHERE aa = 'aaa'; +SELECT tableoid::regclass, ctid, * FROM fa; +-- repopulate tables so that we have rows with same ctid +TRUNCATE a, b; +INSERT INTO a(aa) VALUES('aaa'); +INSERT INTO b(aa) VALUES('bbb'); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +DELETE FROM fa WHERE aa = (CASE WHEN random() <= 1 THEN 'aaa' ELSE 'bbb' END); +SELECT tableoid::regclass, ctid, * FROM fa; + +-- cleanup +DROP FOREIGN TABLE fa; +DROP TABLE a CASCADE; + +-- =================================================================== +-- test foreign table pointing to a remote partitioned table +-- =================================================================== + +-- test DML statement on foreign table pointing to a foreign partitioned table +CREATE TABLE plt (a int, b int) PARTITION BY LIST(a); +CREATE TABLE plt_p1 PARTITION OF plt FOR VALUES IN (1); +CREATE TABLE plt_p2 PARTITION OF plt FOR VALUES IN (2); +INSERT INTO plt VALUES (1, 1), (2, 2); +CREATE FOREIGN TABLE fplt (a int, b int) SERVER loopback OPTIONS (table_name 'plt'); +SELECT tableoid::regclass, ctid, * FROM fplt; +-- use random() so that DML statement is not pushed down to the foreign +-- server +EXPLAIN (VERBOSE, COSTS OFF) +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +UPDATE fplt SET b = (CASE WHEN random() <= 1 THEN 10 ELSE 20 END) WHERE a = 1; +SELECT tableoid::regclass, ctid, * FROM fplt; +-- repopulate partitioned table so that we have rows with same ctid +TRUNCATE plt; +INSERT INTO plt VALUES (1, 1), (2, 2); +EXPLAIN (VERBOSE, COSTS OFF) +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +DELETE FROM fplt WHERE a = (CASE WHEN random() <= 1 THEN 1 ELSE 10 END); +SELECT tableoid::regclass, ctid, * FROM fplt; + +DROP TABLE plt; +DROP FOREIGN TABLE fplt; + -- =================================================================== -- test tuple routing for foreign-table partitions -- =================================================================== -- 2.50.0 --MP_/4ZRYdF7Ah.pt5w65PuZdaPz Content-Type: text/x-patch Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=v3-0002-Error-out-if-one-iteration-of-non-direct-DML-affe.patch ^ permalink raw reply [nested|flat] 324+ messages in thread
* [PATCH v48 2/7] Give 'options' parameter to table_delete/table_update @ 2026-03-30 11:27 Álvaro Herrera <alvherre@kurilemu.de> 0 siblings, 0 replies; 324+ messages in thread From: Álvaro Herrera @ 2026-03-30 11:27 UTC (permalink / raw) The table_insert() method already has an equivalent argument, so this makes sense just on consistency grounds, for future growth. table_delete() can immediately use it to carry the 'changingPart' boolean (which is arguably misplaced in the current API); for table_update we don't have any options at present, but an upcoming patch would add one. Discussion: https://postgr.es/m/202603171606.kf6pmhscqbqz@alvherre.pgsql --- src/backend/access/heap/heapam.c | 16 ++++++++++------ src/backend/access/heap/heapam_handler.c | 13 ++++++++----- src/backend/access/table/tableam.c | 6 +++--- src/backend/executor/nodeModifyTable.c | 9 +++++++-- src/include/access/heapam.h | 7 ++++--- src/include/access/tableam.h | 23 +++++++++++++++-------- 6 files changed, 47 insertions(+), 27 deletions(-) diff --git a/src/backend/access/heap/heapam.c b/src/backend/access/heap/heapam.c index d34136d2e94..0645b2d5f58 100644 --- a/src/backend/access/heap/heapam.c +++ b/src/backend/access/heap/heapam.c @@ -2862,8 +2862,8 @@ xmax_infomask_changed(uint16 new_infomask, uint16 old_infomask) */ TM_Result heap_delete(Relation relation, const ItemPointerData *tid, - CommandId cid, Snapshot crosscheck, bool wait, - TM_FailureData *tmfd, bool changingPart) + CommandId cid, uint32 options, Snapshot crosscheck, + bool wait, TM_FailureData *tmfd) { TM_Result result; TransactionId xid = GetCurrentTransactionId(); @@ -2876,6 +2876,7 @@ heap_delete(Relation relation, const ItemPointerData *tid, TransactionId new_xmax; uint16 new_infomask, new_infomask2; + bool changingPart = (options & TABLE_DELETE_CHANGING_PARTITION) != 0; bool have_tuple_lock = false; bool iscombo; bool all_visible_cleared = false; @@ -3290,9 +3291,11 @@ simple_heap_delete(Relation relation, const ItemPointerData *tid) TM_FailureData tmfd; result = heap_delete(relation, tid, - GetCurrentCommandId(true), InvalidSnapshot, + GetCurrentCommandId(true), + 0, + InvalidSnapshot, true /* wait for commit */ , - &tmfd, false /* changingPart */ ); + &tmfd); switch (result) { case TM_SelfModified: @@ -3331,7 +3334,7 @@ simple_heap_delete(Relation relation, const ItemPointerData *tid) */ TM_Result heap_update(Relation relation, const ItemPointerData *otid, HeapTuple newtup, - CommandId cid, Snapshot crosscheck, bool wait, + CommandId cid, uint32 options, Snapshot crosscheck, bool wait, TM_FailureData *tmfd, LockTupleMode *lockmode, TU_UpdateIndexes *update_indexes) { @@ -4585,7 +4588,8 @@ simple_heap_update(Relation relation, const ItemPointerData *otid, HeapTuple tup LockTupleMode lockmode; result = heap_update(relation, otid, tup, - GetCurrentCommandId(true), InvalidSnapshot, + GetCurrentCommandId(true), 0, + InvalidSnapshot, true /* wait for commit */ , &tmfd, &lockmode, update_indexes); switch (result) diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index cdd153c6b6d..69debeff516 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -313,21 +313,23 @@ heapam_tuple_complete_speculative(Relation relation, TupleTableSlot *slot, static TM_Result heapam_tuple_delete(Relation relation, ItemPointer tid, CommandId cid, - Snapshot snapshot, Snapshot crosscheck, bool wait, - TM_FailureData *tmfd, bool changingPart) + uint32 options, Snapshot snapshot, Snapshot crosscheck, + bool wait, TM_FailureData *tmfd) { /* * Currently Deleting of index tuples are handled at vacuum, in case if * the storage itself is cleaning the dead tuples by itself, it is the * time to call the index tuple deletion also. */ - return heap_delete(relation, tid, cid, crosscheck, wait, tmfd, changingPart); + return heap_delete(relation, tid, cid, options, crosscheck, wait, + tmfd); } static TM_Result heapam_tuple_update(Relation relation, ItemPointer otid, TupleTableSlot *slot, - CommandId cid, Snapshot snapshot, Snapshot crosscheck, + CommandId cid, uint32 options pg_attribute_unused(), + Snapshot snapshot, Snapshot crosscheck, bool wait, TM_FailureData *tmfd, LockTupleMode *lockmode, TU_UpdateIndexes *update_indexes) { @@ -339,7 +341,8 @@ heapam_tuple_update(Relation relation, ItemPointer otid, TupleTableSlot *slot, slot->tts_tableOid = RelationGetRelid(relation); tuple->t_tableOid = slot->tts_tableOid; - result = heap_update(relation, otid, tuple, cid, crosscheck, wait, + result = heap_update(relation, otid, tuple, cid, options, + crosscheck, wait, tmfd, lockmode, update_indexes); ItemPointerCopy(&tuple->t_self, &slot->tts_tid); diff --git a/src/backend/access/table/tableam.c b/src/backend/access/table/tableam.c index 86481d7c029..68ff0966f1c 100644 --- a/src/backend/access/table/tableam.c +++ b/src/backend/access/table/tableam.c @@ -320,9 +320,9 @@ simple_table_tuple_delete(Relation rel, ItemPointer tid, Snapshot snapshot) result = table_tuple_delete(rel, tid, GetCurrentCommandId(true), - snapshot, InvalidSnapshot, + 0, snapshot, InvalidSnapshot, true /* wait for commit */ , - &tmfd, false /* changingPart */ ); + &tmfd); switch (result) { @@ -369,7 +369,7 @@ simple_table_tuple_update(Relation rel, ItemPointer otid, result = table_tuple_update(rel, otid, slot, GetCurrentCommandId(true), - snapshot, InvalidSnapshot, + 0, snapshot, InvalidSnapshot, true /* wait for commit */ , &tmfd, &lockmode, update_indexes); diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 582bcc367c0..76728f08734 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -1522,14 +1522,18 @@ ExecDeleteAct(ModifyTableContext *context, ResultRelInfo *resultRelInfo, ItemPointer tupleid, bool changingPart) { EState *estate = context->estate; + uint32 options = 0; + + if (changingPart) + options |= TABLE_DELETE_CHANGING_PARTITION; return table_tuple_delete(resultRelInfo->ri_RelationDesc, tupleid, estate->es_output_cid, + options, estate->es_snapshot, estate->es_crosscheck_snapshot, true /* wait for commit */ , - &context->tmfd, - changingPart); + &context->tmfd); } /* @@ -2331,6 +2335,7 @@ lreplace: */ result = table_tuple_update(resultRelationDesc, tupleid, slot, estate->es_output_cid, + 0, estate->es_snapshot, estate->es_crosscheck_snapshot, true /* wait for commit */ , diff --git a/src/include/access/heapam.h b/src/include/access/heapam.h index f46c83e88f3..54067b828e4 100644 --- a/src/include/access/heapam.h +++ b/src/include/access/heapam.h @@ -382,13 +382,14 @@ extern void heap_multi_insert(Relation relation, TupleTableSlot **slots, int ntuples, CommandId cid, uint32 options, BulkInsertState bistate); extern TM_Result heap_delete(Relation relation, const ItemPointerData *tid, - CommandId cid, Snapshot crosscheck, bool wait, - TM_FailureData *tmfd, bool changingPart); + CommandId cid, uint32 options, Snapshot crosscheck, + bool wait, TM_FailureData *tmfd); extern void heap_finish_speculative(Relation relation, const ItemPointerData *tid); extern void heap_abort_speculative(Relation relation, const ItemPointerData *tid); extern TM_Result heap_update(Relation relation, const ItemPointerData *otid, HeapTuple newtup, - CommandId cid, Snapshot crosscheck, bool wait, + CommandId cid, uint32 options, + Snapshot crosscheck, bool wait, TM_FailureData *tmfd, LockTupleMode *lockmode, TU_UpdateIndexes *update_indexes); extern TM_Result heap_lock_tuple(Relation relation, HeapTuple tuple, diff --git a/src/include/access/tableam.h b/src/include/access/tableam.h index 57892152957..c3b4429d9b9 100644 --- a/src/include/access/tableam.h +++ b/src/include/access/tableam.h @@ -288,6 +288,11 @@ typedef struct TM_IndexDeleteOp /* Follow update chain and lock latest version of tuple */ #define TUPLE_LOCK_FLAG_FIND_LAST_VERSION (1 << 1) +/* "options" flag bits for table_tuple_delete */ +#define TABLE_DELETE_CHANGING_PARTITION (1 << 0) + +/* "options" flag bits for table_tuple_update */ +/* XXX none at present */ /* Typedef for callback function for table_index_build_scan */ typedef void (*IndexBuildCallback) (Relation index, @@ -559,17 +564,18 @@ typedef struct TableAmRoutine TM_Result (*tuple_delete) (Relation rel, ItemPointer tid, CommandId cid, + uint32 options, Snapshot snapshot, Snapshot crosscheck, bool wait, - TM_FailureData *tmfd, - bool changingPart); + TM_FailureData *tmfd); /* see table_tuple_update() for reference about parameters */ TM_Result (*tuple_update) (Relation rel, ItemPointer otid, TupleTableSlot *slot, CommandId cid, + uint32 options, Snapshot snapshot, Snapshot crosscheck, bool wait, @@ -1534,12 +1540,12 @@ table_multi_insert(Relation rel, TupleTableSlot **slots, int nslots, */ static inline TM_Result table_tuple_delete(Relation rel, ItemPointer tid, CommandId cid, - Snapshot snapshot, Snapshot crosscheck, bool wait, - TM_FailureData *tmfd, bool changingPart) + uint32 options, Snapshot snapshot, Snapshot crosscheck, + bool wait, TM_FailureData *tmfd) { - return rel->rd_tableam->tuple_delete(rel, tid, cid, + return rel->rd_tableam->tuple_delete(rel, tid, cid, options, snapshot, crosscheck, - wait, tmfd, changingPart); + wait, tmfd); } /* @@ -1579,12 +1585,13 @@ table_tuple_delete(Relation rel, ItemPointer tid, CommandId cid, */ static inline TM_Result table_tuple_update(Relation rel, ItemPointer otid, TupleTableSlot *slot, - CommandId cid, Snapshot snapshot, Snapshot crosscheck, + CommandId cid, uint32 options, + Snapshot snapshot, Snapshot crosscheck, bool wait, TM_FailureData *tmfd, LockTupleMode *lockmode, TU_UpdateIndexes *update_indexes) { return rel->rd_tableam->tuple_update(rel, otid, slot, - cid, snapshot, crosscheck, + cid, options, snapshot, crosscheck, wait, tmfd, lockmode, update_indexes); } -- 2.47.3 --qfkt2ktdpcfeypib Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v48-0003-Rename-cluster.c-h-repack.c-h.patch" ^ permalink raw reply [nested|flat] 324+ messages in thread
end of thread, other threads:[~2026-03-30 11:27 UTC | newest] Thread overview: 324+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2025-07-15 10:45 [PATCH v3 1/2] Test exposing bug when foreign table points to a partitioned table Jehan-Guillaume de Rorthais <jgdr@dalibo.com> 2026-03-30 11:27 [PATCH v48 2/7] Give 'options' parameter to table_delete/table_update Álvaro Herrera <alvherre@kurilemu.de>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox