From: Tomas Vondra <tomas.vondra@enterprisedb.com>
To: tsunakawa.takay@fujitsu.com <tsunakawa.takay@fujitsu.com>
To: Craig Ringer <craig.ringer@enterprisedb.com>
Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: Andrey V. Lepikhov <a.lepikhov@postgrespro.ru>
Cc: pgsql-hackers@postgresql.org <pgsql-hackers@postgresql.org>
Cc: Andres Freund <andres@anarazel.de>
Subject: Re: POC: postgres_fdw insert batching
Date: Wed, 25 Nov 2020 21:04:51 +0100
Message-ID: <6358b1e4-7f0a-6e0a-84bf-3cf890fe1073@enterprisedb.com> (raw)
In-Reply-To: <TYAPR01MB2990FC82356124F692545B53FEFA0@TYAPR01MB2990.jpnprd01.prod.outlook.com>
References: <20200628151002.7x5laxwpgvkyiu3q@development>
<0300d7df-854b-6f72-1e93-dfbf98f3fc8a@postgrespro.ru>
<20200712001101.x65f3hvsanvamtgj@development>
<OSBPR01MB2982039EA967F0304CC6A3ECFE0B0@OSBPR01MB2982.jpnprd01.prod.outlook.com>
<CAGRY4nzSZTn-EbvNJq+ZRVKrMv0pHGqb6xfi-SAiM5+LrX0Ebw@mail.gmail.com>
<TYAPR01MB2990FC82356124F692545B53FEFA0@TYAPR01MB2990.jpnprd01.prod.outlook.com>
On 11/25/20 7:31 AM, tsunakawa.takay@fujitsu.com wrote:
> From: Craig Ringer <craig.ringer@enterprisedb.com>
>> I suggest that when developing this, you keep in mind the ongoing
>> work on the libpq pipelining/batching enhancements, and also the
>> way many interfaces to foreign data sources support asynchronous,
>> concurrent operations.
>
> Yes, thank you, I bear it in mind. I understand it's a feature for
> batching multiple kinds of SQL statements like DBC's batch updates.
>
I haven't followed the libpq pipelining thread very closely. It does
seem related, but I'm not sure if it's a good match for this patch, or
how far is it from being committable ...
>
>> I'd argue it's pretty much vital for decent performance when
>> talking to a cloud database from an on-prem server for example, or
>> any other time that round-trip-time reduction is important.
>
> Yeah, I'm thinking of the data migration and integration as the
> prominent use case.
>
Well, good that we all agree this is a useful feature to have (in
general). The question is whether postgres_fdw should be doing batching
on it's onw (per this thread) or rely on some other feature (libpq
pipelining). I haven't followed the other thread, so I don't have an
opinion on that.
Note however we're doing two things here, actually - we're implementing
custom batching for postgres_fdw, but we're also extending the FDW API
to allow other implementations do the same thing. And most of them won't
be able to rely on the connection library providing that, I believe.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: tomas.vondra@enterprisedb.com, tsunakawa.takay@fujitsu.com, craig.ringer@enterprisedb.com, tomas.vondra@2ndquadrant.com, a.lepikhov@postgrespro.ru, andres@anarazel.de
Subject: Re: POC: postgres_fdw insert batching
In-Reply-To: <6358b1e4-7f0a-6e0a-84bf-3cf890fe1073@enterprisedb.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox