pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Alexander Pyhalov <a.pyhalov@postgrespro.ru>
To: Tomas Vondra <tomas.vondra@enterprisedb.com>
Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Andres Freund <andres@anarazel.de>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: postgres_fdw batching vs. (re)creating the tuple slots
Date: Wed, 16 Jun 2021 15:36:13 +0300
Message-ID: <7a8cf8d56b3d18e5c0bccd6cd42d04ac@postgrespro.ru> (raw)
In-Reply-To: <cdff37d6-c57c-e40c-4635-4bcaeccd0d7d@enterprisedb.com>
References: <ebbbcc7d-4286-8c28-0272-61b4753af761@enterprisedb.com>
	<20210530205831.vib4p7mt23cymhpc@alap3.anarazel.de>
	<2399204.1622409059@sss.pgh.pa.us>
	<20210530212616.5k5nxnbunwnwet2a@alap3.anarazel.de>
	<2433405.1622416631@sss.pgh.pa.us>
	<a3233163-829d-94a0-7ca0-87bdfde92c90@enterprisedb.com>
	<4e7ebff8-59ac-c90c-3ecd-8d9dfdfc75ad@enterprisedb.com>
	<6dc1c07b-87e6-63e6-10b3-cca4698361fb@enterprisedb.com>
	<CALj2ACWC8ecAZz5hOVh7vf+82s3_t8HKoMjQFbuncT+a4jn+vQ@mail.gmail.com>
	<ec1c620f-b9ff-26ac-6848-6e39f0a72f6e@enterprisedb.com>
	<cdff37d6-c57c-e40c-4635-4bcaeccd0d7d@enterprisedb.com>

Tomas Vondra писал 2021-06-12 00:01:
> On 6/9/21 1:08 PM, Tomas Vondra wrote:
>> 
>> 
>> On 6/9/21 12:50 PM, Bharath Rupireddy wrote:
>>> On Wed, Jun 9, 2021 at 4:00 PM Tomas Vondra
>>> <tomas.vondra@enterprisedb.com> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> Here's a v2 fixing a silly bug with reusing the same variable in two
>>>> nested loops (worked for simple postgres_fdw cases, but "make check"
>>>> failed).
>>> 
>>> I applied these patches and ran make check in postgres_fdw contrib
>>> module, I saw a server crash. Is it the same failure you were saying
>>> above?
>>> 
>> 
>> Nope, that was causing infinite loop. This is jut a silly mistake on 
>> my
>> side - I forgot to replace the i/j variable inside the loop. Here's 
>> v3.
>> 
>> regards
>> 
> 
> FWIW I've pushed this, after improving the comments a little bit.
> 
> 
> regards

Hi.
It seems this commit

commit b676ac443b6a83558d4701b2dd9491c0b37e17c4
Author: Tomas Vondra <tomas.vondra@postgresql.org>
Date:   Fri Jun 11 20:19:48 2021 +0200

     Optimize creation of slots for FDW bulk inserts

has broken batch insert for partitions with unique indexes.

Earlier the case worked as expected, inserting 1000 tuples. Now it exits 
with

ERROR:  duplicate key value violates unique constraint "p0_pkey"
DETAIL:  Key (x)=(1) already exists.
CONTEXT:  remote SQL command: INSERT INTO public.batch_table_p0(x, 
field1, field2) VALUES ($1, $2, $3), ($4, $5, $6), ($7, $8, $9), ($10, 
$11, $12), ($13, $14, $15), ($16, $17, $18), ($19, $20, $21), ($22, $23, 
$24), ($25, $26, $27), ($28, $29, $30), ($31, $32, $33), ($34, $35, 
$36), ($37, $38, $39), ($40, $41, $42), ($43, $44, $45), ($46, $47, 
$48), ($49, $50, $51), ($52, $53, $54), ($55, $56, $57), ($58, $59, 
$60), ($61, $62, $63), ($64, $65, $66), ($67, $68, $69), ($70, $71, 
$72), ($73, $74, $75), ($76, $77, $78), ($79, $80, $81), ($82, $83, 
$84), ($85, $86, $87), ($88, $89, $90), ($91, $92, $93), ($94, $95, 
$96), ($97, $98, $99), ($100, $101, $102), ($103, $104, $105), ($106, 
$107, $108), ($109, $110, $111), ($112, $113, $114), ($115, $116, $117), 
($118, $119, $120), ($121, $122, $123), ($124, $125, $126), ($127, $128, 
$129), ($130, $131, $132), ($133, $134, $135), ($136, $137, $138), 
($139, $140, $141), ($142, $143, $144), ($145, $146, $147), ($148, $149, 
$150), ($151, $152, $153), ($154, $155, $156), ($157, $158, $159), 
($160, $161, $162), ($163, $164, $165), ($166, $167, $168), ($169, $170, 
$171), ($172, $173, $174), ($175, $176, $177), ($178, $179, $180), 
($181, $182, $183), ($184, $185, $186), ($187, $188, $189), ($190, $191, 
$192), ($193, $194, $195), ($196, $197, $198), ($199, $200, $201), 
($202, $203, $204), ($205, $206, $207), ($208, $209, $210), ($211, $212, 
$213), ($214, $215, $216), ($217, $218, $219), ($220, $221, $222), 
($223, $224, $225), ($226, $227, $228), ($229, $230, $231), ($232, $233, 
$234), ($235, $236, $237), ($238, $239, $240), ($241, $242, $243), 
($244, $245, $246), ($247, $248, $249), ($250, $251, $252), ($253, $254, 
$255), ($256, $257, $258), ($259, $260, $261), ($262, $263, $264), 
($265, $266, $267), ($268, $269, $270), ($271, $272, $273), ($274, $275, 
$276), ($277, $278, $279), ($280, $281, $282), ($283, $284, $285), 
($286, $287, $288), ($289, $290, $291), ($292, $293, $294), ($295, $296, 
$297), ($298, $299, $300)


-- 
Best regards,
Alexander Pyhalov,
Postgres Professional

Attachments:

batch_check_vanilla.patch (text/x-diff, 2.0K) download diff --git a/contrib/postgres_fdw/sql/postgres_fdw.sql b/contrib/postgres_fdw/sql/postgres_fdw.sql index 8cb2148f1f6..4c280f1e777 100644 --- a/contrib/postgres_fdw/sql/postgres_fdw.sql +++ b/contrib/postgres_fdw/sql/postgres_fdw.sql @@ -3083,7 +3083,46 @@ UPDATE batch_cp_upd_test t SET a = 1 FROM (VALUES (1), (2)) s(a) WHERE t.a = s.a SELECT tableoid::regclass, * FROM batch_cp_upd_test; -- Clean up -DROP TABLE batch_table, batch_cp_upd_test CASCADE; +DROP TABLE batch_table, batch_cp_upd_test, batch_table_p0, batch_table_p1 CASCADE; + +-- Use partitioning +ALTER SERVER loopback OPTIONS (ADD batch_size '100'); + +CREATE TABLE batch_table ( x int, field1 text, field2 text) PARTITION BY HASH (x); + +CREATE TABLE batch_table_p0 (LIKE batch_table); +ALTER TABLE batch_table_p0 ADD CONSTRAINT p0_pkey PRIMARY KEY (x); +CREATE FOREIGN TABLE batch_table_p0f + PARTITION OF batch_table + FOR VALUES WITH (MODULUS 4, REMAINDER 0) + SERVER loopback + OPTIONS (table_name 'batch_table_p0'); + +CREATE TABLE batch_table_p1 (LIKE batch_table); +ALTER TABLE batch_table_p1 ADD CONSTRAINT p1_pkey PRIMARY KEY (x); +CREATE FOREIGN TABLE batch_table_p1f + PARTITION OF batch_table + FOR VALUES WITH (MODULUS 4, REMAINDER 1) + SERVER loopback + OPTIONS (table_name 'batch_table_p1'); + +CREATE TABLE batch_table_p2 + PARTITION OF batch_table + FOR VALUES WITH (MODULUS 4, REMAINDER 2); +ALTER TABLE batch_table_p2 ADD CONSTRAINT p2_pkey PRIMARY KEY (x); + +CREATE TABLE batch_table_p3 (LIKE batch_table); +ALTER TABLE batch_table_p3 ADD CONSTRAINT p3_pkey PRIMARY KEY (x); +CREATE FOREIGN TABLE batch_table_p3f + PARTITION OF batch_table + FOR VALUES WITH (MODULUS 4, REMAINDER 3) + SERVER loopback + OPTIONS (table_name 'batch_table_p3'); + +INSERT INTO batch_table SELECT i, 'test'||i, 'test'|| i FROM generate_series(1, 1000) i; +SELECT COUNT(*) FROM batch_table; + +ALTER SERVER loopback OPTIONS (DROP batch_size); -- =================================================================== -- test asynchronous execution

view thread (15+ messages)  latest in thread

Message-ID: <7a8cf8d56b3d18e5c0bccd6cd42d04ac@postgrespro.ru>
Permalink:  ../7a8cf8d56b3d18e5c0bccd6cd42d04ac@postgrespro.ru/
Also on:    postgresql.org/message-id/7a8cf8d56b3d18e5c0bccd6cd42d04ac@postgrespro.ru

 ·  · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-hackers@postgresql.org
  Cc: a.pyhalov@postgrespro.ru, tomas.vondra@enterprisedb.com, bharath.rupireddyforpostgres@gmail.com, tgl@sss.pgh.pa.us, andres@anarazel.de
  Subject: Re: postgres_fdw batching vs. (re)creating the tuple slots
  In-Reply-To: <7a8cf8d56b3d18e5c0bccd6cd42d04ac@postgrespro.ru>
  References: <ebbbcc7d-4286-8c28-0272-61b4753af761@enterprisedb.com> <20210530205831.vib4p7mt23cymhpc@alap3.anarazel.de> <2399204.1622409059@sss.pgh.pa.us> <20210530212616.5k5nxnbunwnwet2a@alap3.anarazel.de> <2433405.1622416631@sss.pgh.pa.us> <a3233163-829d-94a0-7ca0-87bdfde92c90@enterprisedb.com> <4e7ebff8-59ac-c90c-3ecd-8d9dfdfc75ad@enterprisedb.com> <6dc1c07b-87e6-63e6-10b3-cca4698361fb@enterprisedb.com> <CALj2ACWC8ecAZz5hOVh7vf+82s3_t8HKoMjQFbuncT+a4jn+vQ@mail.gmail.com> <ec1c620f-b9ff-26ac-6848-6e39f0a72f6e@enterprisedb.com> <cdff37d6-c57c-e40c-4635-4bcaeccd0d7d@enterprisedb.com> <7a8cf8d56b3d18e5c0bccd6cd42d04ac@postgrespro.ru>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox