pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feed From: Alexander Pyhalov <a.pyhalov@postgrespro.ru>
To: Tomas Vondra <tomas.vondra@enterprisedb.com>
Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Andres Freund <andres@anarazel.de>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: postgres_fdw batching vs. (re)creating the tuple slots
Date: Wed, 16 Jun 2021 15:36:13 +0300
Message-ID: <7a8cf8d56b3d18e5c0bccd6cd42d04ac@postgrespro.ru> (raw )
In-Reply-To: <cdff37d6-c57c-e40c-4635-4bcaeccd0d7d@enterprisedb.com >
References: <ebbbcc7d-4286-8c28-0272-61b4753af761@enterprisedb.com >
<20210530205831.vib4p7mt23cymhpc@alap3.anarazel.de >
<2399204.1622409059@sss.pgh.pa.us >
<20210530212616.5k5nxnbunwnwet2a@alap3.anarazel.de >
<2433405.1622416631@sss.pgh.pa.us >
<a3233163-829d-94a0-7ca0-87bdfde92c90@enterprisedb.com >
<4e7ebff8-59ac-c90c-3ecd-8d9dfdfc75ad@enterprisedb.com >
<6dc1c07b-87e6-63e6-10b3-cca4698361fb@enterprisedb.com >
<CALj2ACWC8ecAZz5hOVh7vf+82s3_t8HKoMjQFbuncT+a4jn+vQ@mail.gmail.com >
<ec1c620f-b9ff-26ac-6848-6e39f0a72f6e@enterprisedb.com >
<cdff37d6-c57c-e40c-4635-4bcaeccd0d7d@enterprisedb.com >
Tomas Vondra писал 2021-06-12 00:01:
> On 6/9/21 1:08 PM, Tomas Vondra wrote:
>>
>>
>> On 6/9/21 12:50 PM, Bharath Rupireddy wrote:
>>> On Wed, Jun 9, 2021 at 4:00 PM Tomas Vondra
>>> <tomas.vondra@enterprisedb.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> Here's a v2 fixing a silly bug with reusing the same variable in two
>>>> nested loops (worked for simple postgres_fdw cases, but "make check"
>>>> failed).
>>>
>>> I applied these patches and ran make check in postgres_fdw contrib
>>> module, I saw a server crash. Is it the same failure you were saying
>>> above?
>>>
>>
>> Nope, that was causing infinite loop. This is jut a silly mistake on
>> my
>> side - I forgot to replace the i/j variable inside the loop. Here's
>> v3.
>>
>> regards
>>
>
> FWIW I've pushed this, after improving the comments a little bit.
>
>
> regards
Hi.
It seems this commit
commit b676ac443b6a83558d4701b2dd9491c0b37e17c4
Author: Tomas Vondra <tomas.vondra@postgresql.org>
Date: Fri Jun 11 20:19:48 2021 +0200
Optimize creation of slots for FDW bulk inserts
has broken batch insert for partitions with unique indexes.
Earlier the case worked as expected, inserting 1000 tuples. Now it exits
with
ERROR: duplicate key value violates unique constraint "p0_pkey"
DETAIL: Key (x)=(1) already exists.
CONTEXT: remote SQL command: INSERT INTO public.batch_table_p0(x,
field1, field2) VALUES ($1, $2, $3), ($4, $5, $6), ($7, $8, $9), ($10,
$11, $12), ($13, $14, $15), ($16, $17, $18), ($19, $20, $21), ($22, $23,
$24), ($25, $26, $27), ($28, $29, $30), ($31, $32, $33), ($34, $35,
$36), ($37, $38, $39), ($40, $41, $42), ($43, $44, $45), ($46, $47,
$48), ($49, $50, $51), ($52, $53, $54), ($55, $56, $57), ($58, $59,
$60), ($61, $62, $63), ($64, $65, $66), ($67, $68, $69), ($70, $71,
$72), ($73, $74, $75), ($76, $77, $78), ($79, $80, $81), ($82, $83,
$84), ($85, $86, $87), ($88, $89, $90), ($91, $92, $93), ($94, $95,
$96), ($97, $98, $99), ($100, $101, $102), ($103, $104, $105), ($106,
$107, $108), ($109, $110, $111), ($112, $113, $114), ($115, $116, $117),
($118, $119, $120), ($121, $122, $123), ($124, $125, $126), ($127, $128,
$129), ($130, $131, $132), ($133, $134, $135), ($136, $137, $138),
($139, $140, $141), ($142, $143, $144), ($145, $146, $147), ($148, $149,
$150), ($151, $152, $153), ($154, $155, $156), ($157, $158, $159),
($160, $161, $162), ($163, $164, $165), ($166, $167, $168), ($169, $170,
$171), ($172, $173, $174), ($175, $176, $177), ($178, $179, $180),
($181, $182, $183), ($184, $185, $186), ($187, $188, $189), ($190, $191,
$192), ($193, $194, $195), ($196, $197, $198), ($199, $200, $201),
($202, $203, $204), ($205, $206, $207), ($208, $209, $210), ($211, $212,
$213), ($214, $215, $216), ($217, $218, $219), ($220, $221, $222),
($223, $224, $225), ($226, $227, $228), ($229, $230, $231), ($232, $233,
$234), ($235, $236, $237), ($238, $239, $240), ($241, $242, $243),
($244, $245, $246), ($247, $248, $249), ($250, $251, $252), ($253, $254,
$255), ($256, $257, $258), ($259, $260, $261), ($262, $263, $264),
($265, $266, $267), ($268, $269, $270), ($271, $272, $273), ($274, $275,
$276), ($277, $278, $279), ($280, $281, $282), ($283, $284, $285),
($286, $287, $288), ($289, $290, $291), ($292, $293, $294), ($295, $296,
$297), ($298, $299, $300)
--
Best regards,
Alexander Pyhalov,
Postgres Professional
Attachments:
batch_check_vanilla.patch (text/x-diff, 2.0K) download
diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql
index 8cb2148f1f6..4c280f1e777 100644
--- a/contrib/postgres_fdw/sql/postgres_fdw.sql
+++ b/contrib/postgres_fdw/sql/postgres_fdw.sql
@@ -3083,7 +3083,46 @@ UPDATE batch_cp_upd_test t SET a = 1 FROM (VALUES (1), (2)) s(a) WHERE t.a = s.a
SELECT tableoid::regclass, * FROM batch_cp_upd_test;
-- Clean up
-DROP TABLE batch_table, batch_cp_upd_test CASCADE;
+DROP TABLE batch_table, batch_cp_upd_test, batch_table_p0, batch_table_p1 CASCADE;
+
+-- Use partitioning
+ALTER SERVER loopback OPTIONS (ADD batch_size '100');
+
+CREATE TABLE batch_table ( x int, field1 text, field2 text) PARTITION BY HASH (x);
+
+CREATE TABLE batch_table_p0 (LIKE batch_table);
+ALTER TABLE batch_table_p0 ADD CONSTRAINT p0_pkey PRIMARY KEY (x);
+CREATE FOREIGN TABLE batch_table_p0f
+ PARTITION OF batch_table
+ FOR VALUES WITH (MODULUS 4, REMAINDER 0)
+ SERVER loopback
+ OPTIONS (table_name 'batch_table_p0');
+
+CREATE TABLE batch_table_p1 (LIKE batch_table);
+ALTER TABLE batch_table_p1 ADD CONSTRAINT p1_pkey PRIMARY KEY (x);
+CREATE FOREIGN TABLE batch_table_p1f
+ PARTITION OF batch_table
+ FOR VALUES WITH (MODULUS 4, REMAINDER 1)
+ SERVER loopback
+ OPTIONS (table_name 'batch_table_p1');
+
+CREATE TABLE batch_table_p2
+ PARTITION OF batch_table
+ FOR VALUES WITH (MODULUS 4, REMAINDER 2);
+ALTER TABLE batch_table_p2 ADD CONSTRAINT p2_pkey PRIMARY KEY (x);
+
+CREATE TABLE batch_table_p3 (LIKE batch_table);
+ALTER TABLE batch_table_p3 ADD CONSTRAINT p3_pkey PRIMARY KEY (x);
+CREATE FOREIGN TABLE batch_table_p3f
+ PARTITION OF batch_table
+ FOR VALUES WITH (MODULUS 4, REMAINDER 3)
+ SERVER loopback
+ OPTIONS (table_name 'batch_table_p3');
+
+INSERT INTO batch_table SELECT i, 'test'||i, 'test'|| i FROM generate_series(1, 1000) i;
+SELECT COUNT(*) FROM batch_table;
+
+ALTER SERVER loopback OPTIONS (DROP batch_size);
-- ===================================================================
-- test asynchronous execution
view thread (15+ messages) latest in thread
Message-ID: <7a8cf8d56b3d18e5c0bccd6cd42d04ac@postgrespro.ru>
Permalink: ../7a8cf8d56b3d18e5c0bccd6cd42d04ac@postgrespro.ru/
Also on: postgresql.org/message-id/7a8cf8d56b3d18e5c0bccd6cd42d04ac@postgrespro.ru
copy link · copy postgr.es · copy git am
reply Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-hackers@postgresql.org
Cc: a.pyhalov@postgrespro.ru, tomas.vondra@enterprisedb.com, bharath.rupireddyforpostgres@gmail.com, tgl@sss.pgh.pa.us, andres@anarazel.de
Subject: Re: postgres_fdw batching vs. (re)creating the tuple slots
In-Reply-To: <7a8cf8d56b3d18e5c0bccd6cd42d04ac@postgrespro.ru>
References: <ebbbcc7d-4286-8c28-0272-61b4753af761@enterprisedb.com> <20210530205831.vib4p7mt23cymhpc@alap3.anarazel.de> <2399204.1622409059@sss.pgh.pa.us> <20210530212616.5k5nxnbunwnwet2a@alap3.anarazel.de> <2433405.1622416631@sss.pgh.pa.us> <a3233163-829d-94a0-7ca0-87bdfde92c90@enterprisedb.com> <4e7ebff8-59ac-c90c-3ecd-8d9dfdfc75ad@enterprisedb.com> <6dc1c07b-87e6-63e6-10b3-cca4698361fb@enterprisedb.com> <CALj2ACWC8ecAZz5hOVh7vf+82s3_t8HKoMjQFbuncT+a4jn+vQ@mail.gmail.com> <ec1c620f-b9ff-26ac-6848-6e39f0a72f6e@enterprisedb.com> <cdff37d6-c57c-e40c-4635-4bcaeccd0d7d@enterprisedb.com> <7a8cf8d56b3d18e5c0bccd6cd42d04ac@postgrespro.ru>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox