Message-ID: <7f4f4783-a9fd-6b1b-b39d-3c5f72d54961@enterprisedb.com>
Date: Sat, 2 Apr 2022 13:58:43 +0200
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.7.0
Subject: Re: logical decoding and replication of sequences
Content-Language: en-US
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: Petr Jelinek <petr.jelinek@enterprisedb.com>,
 Peter Eisentraut <peter.eisentraut@enterprisedb.com>,
 PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
References: <d045f3c2-6cfb-06d3-5540-e63c320df8bc@enterprisedb.com>
 <CAA4eK1Jn-DttQ=4Pdh9YCe1w+zGbgC+0uR0sfg2RtkjiAPmB9g@mail.gmail.com>
 <BDDB28EE-84D3-4BA8-A5BD-DFF56CE584BC@enterprisedb.com>
 <aeb2ba8d-e6f4-5486-cc4c-0d4982c291cb@enterprisedb.com>
 <74729cc5-960b-9295-912a-a7c37356a0c4@enterprisedb.com>
 <CAA4eK1JxDNKGBSNTyN-Xj2JRjzFo+ziSqJbjH==vuO0YF_CQrg@mail.gmail.com>
 <9d7635f8-2402-e038-64da-cbc3732ce8cf@enterprisedb.com>
 <CAA4eK1KG2hj-=5y6Ufc67q8O2P=OgRA584DahK7Sz6UuQvf0qA@mail.gmail.com>
 <c0d61b5f-c122-a057-8063-b7303fb8bdbd@enterprisedb.com>
 <83c682bf-7164-5976-608c-385cc394bb6c@enterprisedb.com>
 <CAA4eK1JrCVD8fQeNQmYo1YnPay_6Mh9fsTYMrAHkNo89Pxr4LA@mail.gmail.com>
 <494bb2d9-0c1c-b429-e6bb-231f00f2c36f@enterprisedb.com>
 <CAA4eK1KAFdQEULk+4C=ieWA5UPSUtf_gtqKsFj9J9f2c=8hm4g@mail.gmail.com>
 <de8cc7c3-4b61-94d5-87dd-ed317b245655@enterprisedb.com>
 <00708727-d856-1886-48e3-811296c7ba8c@enterprisedb.com>
 <CAA4eK1JKdJZKMrpU49UPzS=d+O=_t919EMqhjGC49mAzOT2AQw@mail.gmail.com>
From: Tomas Vondra <tomas.vondra@enterprisedb.com>
In-Reply-To: <CAA4eK1JKdJZKMrpU49UPzS=d+O=_t919EMqhjGC49mAzOT2AQw@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Archived-At: <https://www.postgresql.org/message-id/7f4f4783-a9fd-6b1b-b39d-3c5f72d54961%40enterprisedb.com>
Precedence: bulk


On 4/2/22 12:43, Amit Kapila wrote:
> On Sat, Apr 2, 2022 at 5:47 AM Tomas Vondra
> <tomas.vondra@enterprisedb.com> wrote:
>>
>> On 4/1/22 17:02, Tomas Vondra wrote:
>>
>> The only option I see is reworking the decoding so that it does not need
>> the snapshot at all. We'll need to stash the changes just like any other
>> change, apply them at end of transaction, and the main difference
>> between transactional and non-transactional case would be what happens
>> at abort. Transactional changes would be discarded, non-transactional
>> would be applied anyway.
>>
> 
> I think in the above I am not following how we can make it work
> without considering *snapshot at all* because based on that we would
> have done the initial sync (copy_sequence) and if we don't follow that
> later it can lead to inconsistency. I might be missing something here.
> 

Well, what I meant to say is that we can't consider the snapshot at this
phase of decoding. We'd still consider it later, at commit/abort time,
of course. I.e. it'd be fairly similar to what heap_decode/DecodeInsert
does, for example. AFAIK this does not build the snapshot anywhere.

>> The challenge is this reorders the sequence changes, so we'll need to
>> reconcile them somehow. One option would be to simply (1) apply the
>> change with the highest LSN in the transaction, and then walk all other
>> in-progress transactions and changes for that sequence with a lower LSN.
>> Not sure how complex/expensive that would be, though. Another problem is
>> not all increments are WAL-logged, of course, not sure about that.
>>
>> Another option might be revisiting the approach proposed by Hannu in
>> September [1], i.e. tracking sequences touched in a transaction, and
>> then replicating the current state at that particular moment.
>>
> 
> I'll think about that approach as well.
> 

Thanks!

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company