public inbox for [email protected]
help / color / mirror / Atom feedFrom: Tomas Vondra <[email protected]>
To: Tom Lane <[email protected]>
Cc: Bharath Rupireddy <[email protected]>
Cc: [email protected] <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Subject: Re: Fdw batch insert error out when set batch_size > 65535
Date: Wed, 9 Jun 2021 12:22:15 +0200
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
References: <OS0PR01MB571603973C0AC2874AD6BF2594299@OS0PR01MB5716.jpnprd01.prod.outlook.com>
<OS0PR01MB5716BFBD760428859058FCBD94299@OS0PR01MB5716.jpnprd01.prod.outlook.com>
<CALj2ACV=qgpccTEip7NHJWDe3Set8wXwDmNnRzy=GYmx9x94XQ@mail.gmail.com>
<OS0PR01MB57160ADFD5010254C8D1CE6594259@OS0PR01MB5716.jpnprd01.prod.outlook.com>
<CALj2ACVvrd6LGga6UzOu17GzW0cmaS-F30KMBSxKsA+Zw8Z=BQ@mail.gmail.com>
<CALj2ACXBJ3a7MLbCAGA=5dneYsD720KPp1Pot-kKVM1YFPkESA@mail.gmail.com>
<[email protected]>
<CALj2ACXq8a=FdrPDa=_wMrvOwcZjrNfCdeyfG-2dt17LyBEc=g@mail.gmail.com>
<OS0PR01MB571674E988B480E82066EA7794239@OS0PR01MB5716.jpnprd01.prod.outlook.com>
<[email protected]>
<CALj2ACUxf+PRaTVPEY-pV_nNgsB0_U2uU3u8GqMKcF-6M2exjw@mail.gmail.com>
<[email protected]>
<CALj2ACXoxbNhpKx2r29OyBdx+fA51Xsv80B-9-xyp1CeBL3ckQ@mail.gmail.com>
<1! [email protected]>
<[email protected]>
On 6/9/21 8:28 AM, Tom Lane wrote:
> I wrote:
>> Bharath Rupireddy <[email protected]> writes:
>>>> I've added a simple regression test to postgres_fdw, testing that batch
>>>> sizes > 65535 work fine, and pushed the fix.
>
>>> I was earlier thinking of adding one, but stopped because it might
>>> increase the regression test execution time. It looks like that's true
>>> - with and without the test case it takes 17 sec and 4 sec
>>> respectively on my dev system which is 4X slower. I'm not sure if this
>>> is okay.
>
>> The cost, versus the odds of ever detecting a problem, doesn't
>> seem like a good tradeoff.
>
> I took a quick look and noted that on buildfarm member longfin
> (to take a random example that's sitting a few feet from me),
> the time for contrib-install-check went from 34 seconds before
> this patch to 40 seconds after. I find that completely
> unacceptable compared to the likely value of this test case.
>
Note that the problem here is [1] - we're creating a lot of slots
referencing the same tuple descriptor, which inflates the duration.
There's a fix in the other thread, which eliminates ~99% of the
overhead. I plan to push that fix soon (a day or two).
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
view thread (29+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: Fdw batch insert error out when set batch_size > 65535
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox