pg.ddx.io pgsql-hackers@postgresql.org mailing list archive help / color / mirror / Atom feed From: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
To: tsunakawa.takay@fujitsu.com <tsunakawa.takay@fujitsu.com>
Cc: tanghy.fnst@cn.fujitsu.com <tanghy.fnst@cn.fujitsu.com>
Cc: Alexey Kondratov <a.kondratov@postgrespro.ru>
Cc: Michael Paquier <michael@paquier.xyz>
Cc: Ashutosh Bapat <ashutosh.bapat@2ndquadrant.com>
Cc: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>
Cc: Etsuro Fujita <etsuro.fujita@gmail.com>
Cc: PostgreSQL-Dev <pgsql-hackers@postgresql.org>
Cc: Tomas Vondra <tomas.vondra@enterprisedb.com>
Cc: houzj.fnst@cn.fujitsu.com <houzj.fnst@cn.fujitsu.com>
Cc: Amit Langote <amitlangote09@gmail.com>
Subject: Re: [POC] Fast COPY FROM command for the table with foreign partitions
Date: Tue, 9 Feb 2021 10:57:44 +0500
Message-ID: <737f24c8-3f0c-4201-e3da-b763dbf9c8b9@postgrespro.ru> (raw )
In-Reply-To: <TYAPR01MB29908A830017784F6E8A468FFE8E9@TYAPR01MB2990.jpnprd01.prod.outlook.com >
References: <ae25e79e-6122-720e-1349-fb832c6021df@postgrespro.ru >
<TYAPR01MB29900B0604FEB2D7AD9EE61BFEF40@TYAPR01MB2990.jpnprd01.prod.outlook.com >
<CA+HiwqFJG-cJtdWZAqyEQEnA4e-rd=H2dzF+uNSbh5O-1O9tyw@mail.gmail.com >
<9bff5ccc-8649-9699-1cdd-82b53164f9bb@postgrespro.ru >
<7166ded145a947fbb11b67ebe68041f3@G08CNEXMBPEKD05.g08.fujitsu.local >
<c46c1929-d44f-0ce3-a74b-4cfaddf9da82@postgrespro.ru >
<2b465f4cdeeb4a78b3096d31e8af7c77@G08CNEXMBPEKD05.g08.fujitsu.local >
<ca933cda-b8c3-9e9c-23a0-91af2e2a2624@postgrespro.ru >
<9862b608-884b-da1f-afb5-2ee5f2396257@enterprisedb.com >
<6160196a-88a4-ded8-1047-cec16537a111@postgrespro.ru >
<d4d502d42ded45acb3b070a94552a962@G08CNEXMBPEKD05.g08.fujitsu.local >
<TYAPR01MB29906DF706E88C5B2E574477FEB59@TYAPR01MB2990.jpnprd01.prod.outlook.com >
<95b3e446-92ec-f01f-2035-5f5509e6e016@postgrespro.ru >
<TYAPR01MB29906738B11565E139F0984AFEB59@TYAPR01MB2990.jpnprd01.prod.outlook.com >
<TYAPR01MB29908A830017784F6E8A468FFE8E9@TYAPR01MB2990.jpnprd01.prod.outlook.com >
On 2/9/21 9:35 AM, tsunakawa.takay@fujitsu.com wrote:
> From: tsunakawa.takay@fujitsu.com <tsunakawa.takay@fujitsu.com>
>> From: Andrey Lepikhov <a.lepikhov@postgrespro.ru>
>> Also, I might defer working on the extended part (v9 0003 and 0004) and further
>> separate them in a different thread, if it seems to take longer.
>
> I reviewed them but haven't rebased them (it seems to take more labor.)
> Andrey-san, could you tell us:
>
> * Why is a separate FDW connection established for each COPY? To avoid using the same FDW connection for multiple foreign table partitions in a single COPY run?
With separate connection you can init a 'COPY FROM' session for each
foreign partition just one time on partition initialization.
>
> * In what kind of test did you get 2-4x performance gain? COPY into many foreign table partitions where the input rows are ordered randomly enough that many rows don't accumulate in the COPY buffer?
I used 'INSERT INTO .. SELECT * FROM generate_series(1, N)' to generate
test data and HASH partitioning to avoid skews.
--
regards,
Andrey Lepikhov
Postgres Professional
view thread (84+ messages) latest in thread
Message-ID: <737f24c8-3f0c-4201-e3da-b763dbf9c8b9@postgrespro.ru>
Permalink: ../737f24c8-3f0c-4201-e3da-b763dbf9c8b9@postgrespro.ru/
Also on: postgresql.org/message-id/737f24c8-3f0c-4201-e3da-b763dbf9c8b9@postgrespro.ru
copy link · copy postgr.es
reply Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: a.lepikhov@postgrespro.ru, tsunakawa.takay@fujitsu.com, tanghy.fnst@cn.fujitsu.com, a.kondratov@postgrespro.ru, michael@paquier.xyz, ashutosh.bapat@2ndquadrant.com, ashutosh.bapat.oss@gmail.com, etsuro.fujita@gmail.com, tomas.vondra@enterprisedb.com, houzj.fnst@cn.fujitsu.com, amitlangote09@gmail.com
Subject: Re: [POC] Fast COPY FROM command for the table with foreign partitions
In-Reply-To: <737f24c8-3f0c-4201-e3da-b763dbf9c8b9@postgrespro.ru>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox