pg.ddx.io pgsql-hackers@postgresql.org mailing list archive help / color / mirror / Atom feed From: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
To: tsunakawa.takay@fujitsu.com <tsunakawa.takay@fujitsu.com>
Cc: tanghy.fnst@cn.fujitsu.com <tanghy.fnst@cn.fujitsu.com>
Cc: Alexey Kondratov <a.kondratov@postgrespro.ru>
Cc: Michael Paquier <michael@paquier.xyz>
Cc: Ashutosh Bapat <ashutosh.bapat@2ndquadrant.com>
Cc: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>
Cc: Etsuro Fujita <etsuro.fujita@gmail.com>
Cc: PostgreSQL-Dev <pgsql-hackers@postgresql.org>
Cc: Tomas Vondra <tomas.vondra@enterprisedb.com>
Cc: houzj.fnst@cn.fujitsu.com <houzj.fnst@cn.fujitsu.com>
Cc: Amit Langote <amitlangote09@gmail.com>
Subject: Re: [POC] Fast COPY FROM command for the table with foreign partitions
Date: Tue, 9 Feb 2021 13:22:33 +0500
Message-ID: <4397cd58-1fcc-73f2-6565-4b3d1bd432bf@postgrespro.ru> (raw )
In-Reply-To: <TYAPR01MB2990D01994E41A47D69F3F0EFE8E9@TYAPR01MB2990.jpnprd01.prod.outlook.com >
References: <ae25e79e-6122-720e-1349-fb832c6021df@postgrespro.ru >
<9bff5ccc-8649-9699-1cdd-82b53164f9bb@postgrespro.ru >
<7166ded145a947fbb11b67ebe68041f3@G08CNEXMBPEKD05.g08.fujitsu.local >
<c46c1929-d44f-0ce3-a74b-4cfaddf9da82@postgrespro.ru >
<2b465f4cdeeb4a78b3096d31e8af7c77@G08CNEXMBPEKD05.g08.fujitsu.local >
<ca933cda-b8c3-9e9c-23a0-91af2e2a2624@postgrespro.ru >
<9862b608-884b-da1f-afb5-2ee5f2396257@enterprisedb.com >
<6160196a-88a4-ded8-1047-cec16537a111@postgrespro.ru >
<d4d502d42ded45acb3b070a94552a962@G08CNEXMBPEKD05.g08.fujitsu.local >
<TYAPR01MB29906DF706E88C5B2E574477FEB59@TYAPR01MB2990.jpnprd01.prod.outlook.com >
<95b3e446-92ec-f01f-2035-5f5509e6e016@postgrespro.ru >
<TYAPR01MB29906738B11565E139F0984AFEB59@TYAPR01MB2990.jpnprd01.prod.outlook.com >
<TYAPR01MB29908A830017784F6E8A468FFE8E9@TYAPR01MB2990.jpnprd01.prod.outlook.com >
<737f24c8-3f0c-4201-e3da-b763dbf9c8b9@postgrespro.ru >
<TYAPR01MB2990D01994E41A47D69F3F0EFE8E9@TYAPR01MB2990.jpnprd01.prod.outlook.com >
On 2/9/21 12:47 PM, tsunakawa.takay@fujitsu.com wrote:
> From: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
> I guess you used many hash partitions. Sadly, The current COPY implementation only accumulates either 1,000 rows or 64 KB of input data (very small!) before flushing all CopyMultiInsertBuffers. One CopyMultiInsertBuffer corresponds to one partition. Flushing a CopyMultiInsertBuffer calls ExecForeignCopy() once, which connects to a remote database, runs COPY FROM STDIN, and disconnects. Here, the flushing trigger (1,000 rows or 64 KB input data, whichever comes first) is so small that if there are many target partitions, the amount of data for each partition is small.
I tried to use 1E4 - 1E8 rows in a tuple buffer. But the results weren't
impressive.
We can use one more GUC instead of a precompiled constant.
> Why don't we focus on committing the basic part and addressing the extended part (0003 and 0004) separately later?
I focused only on the 0001 and 0002 patches.
> As Tang-san and you showed, the basic part already demonstrated impressive improvement. If there's no objection, I'd like to make this ready for committer in a few days.
Good.
--
regards,
Andrey Lepikhov
Postgres Professional
view thread (84+ messages) latest in thread
Message-ID: <4397cd58-1fcc-73f2-6565-4b3d1bd432bf@postgrespro.ru>
Permalink: ../4397cd58-1fcc-73f2-6565-4b3d1bd432bf@postgrespro.ru/
Also on: postgresql.org/message-id/4397cd58-1fcc-73f2-6565-4b3d1bd432bf@postgrespro.ru
copy link · copy postgr.es
reply Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: a.lepikhov@postgrespro.ru, tsunakawa.takay@fujitsu.com, tanghy.fnst@cn.fujitsu.com, a.kondratov@postgrespro.ru, michael@paquier.xyz, ashutosh.bapat@2ndquadrant.com, ashutosh.bapat.oss@gmail.com, etsuro.fujita@gmail.com, tomas.vondra@enterprisedb.com, houzj.fnst@cn.fujitsu.com, amitlangote09@gmail.com
Subject: Re: [POC] Fast COPY FROM command for the table with foreign partitions
In-Reply-To: <4397cd58-1fcc-73f2-6565-4b3d1bd432bf@postgrespro.ru>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox