Date: Mon, 16 Sep 2019 22:57:19 +0200
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
Cc: Alexey Kondratov <a.kondratov@postgrespro.ru>,
	Peter Eisentraut <peter.eisentraut@2ndquadrant.com>,
	Erik Rijkers <er@xs4all.nl>,
	Masahiko Sawada <sawada.mshk@gmail.com>,
	PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Subject: Re: PATCH: logical_work_mem and logical streaming of large
 in-progress transactions
Message-ID: <20190916205719.mnyytrsredcuf2or@development>
References: <126e3a30-a7f5-ede4-2bd0-b217749c8b33@postgrespro.ru>
 <c511125d-33d7-5209-98ce-5c9977f3e437@2ndquadrant.com>
 <f4cf3883-5ce2-6829-bb42-3038bfc7b4d9@postgrespro.ru>
 <8eda5118-2dd0-79a1-4fe9-eec7e334de17@postgrespro.ru>
 <20190828190646.s3bjs22urff32k56@development>
 <bb84dcf8-0a7d-5c8b-9cea-7186dc3acc3c@postgrespro.ru>
 <20190829184824.kmrbchrk2ged6vjw@development>
 <322e40c4-8ca7-6c34-2544-28a6d95989c2@postgrespro.ru>
 <b25ce80e-f536-78c8-d5c8-a5df3e230785@postgrespro.ru>
 <50a87b09-a5cd-bb13-1d10-41d4147a8a95@postgrespro.ru>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Disposition: inline
In-Reply-To: <50a87b09-a5cd-bb13-1d10-41d4147a8a95@postgrespro.ru>
User-Agent: NeoMutt/20180716
Precedence: bulk

On Mon, Sep 16, 2019 at 10:29:18PM +0300, Konstantin Knizhnik wrote:
>
>
>On 16.09.2019 19:54, Alexey Kondratov wrote:
>>On 30.08.2019 18:59, Konstantin Knizhnik wrote:
>>>
>>>I think that instead of defining savepoints it is simpler and more 
>>>efficient to use
>>>
>>>BeginInternalSubTransaction + 
>>>ReleaseCurrentSubTransaction/RollbackAndReleaseCurrentSubTransaction
>>>
>>>as it is done in PL/pgSQL (pl_exec.c).
>>>Not sure if it can pr
>>>
>>
>>Both BeginInternalSubTransaction and DefineSavepoint use 
>>PushTransaction() internally for a normal subtransaction start. So 
>>they seems to be identical from the performance perspective, which 
>>is also stated in the comment section:
>
>Yes, definitely them are using the same mechanism and most likely 
>provides similar performance.
>But BeginInternalSubTransaction does not require to generate some 
>savepoint name which seems to be redundant in this case.
>
>
>>
>>Anyway, I've performed a profiling of my apply worker (flamegraph is 
>>attached) and it spends the vast amount of time (>90%) applying 
>>changes. So the problem is not in the savepoints their-self, but in 
>>the fact that we first apply all changes and then abort all the 
>>work. Not sure, that it is possible to do something in this case.
>>
>
>Looks like the only way to increase apply speed is to do it in 
>parallel: make it possible to concurrently execute non-conflicting 
>transactions.
>

True, although it seems like a massive can of worms to me. I'm not aware
a way to identify non-conflicting transactions in advance, so it would
have to be implemented as optimistic apply, with a detection and
recovery from conflicts.

I'm not against doing that, and I'm willing to spend some time on revies
etc. but it seems like a completely separate effort.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services