Message-ID: <86cea9a2-9a79-0228-bae4-287cebe54bc7@enterprisedb.com>
Date: Wed, 6 Dec 2023 14:47:11 +0100
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.15.1
Subject: Re: logical decoding and replication of sequences, take 2
Content-Language: en-US
To: Dilip Kumar <dilipbalaut@gmail.com>, Amit Kapila <amit.kapila16@gmail.com>
Cc: "Hayato Kuroda (Fujitsu)" <kuroda.hayato@fujitsu.com>,
 "Zhijie Hou (Fujitsu)" <houzj.fnst@fujitsu.com>,
 Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>,
 PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>,
 Masahiko Sawada <sawada.mshk@gmail.com>,
 Peter Eisentraut <peter.eisentraut@enterprisedb.com>
References: <c5e671bc-3400-c2f4-14d3-1ae06c462dad@enterprisedb.com>
 <36f3a200-f185-887c-da08-fdc866436e6c@enterprisedb.com>
 <CAA4eK1LkZ+99rKhp1Nxhg9iGBppf1FgLEFmC5SntNFtc1xJFTg@mail.gmail.com>
 <195be236-7580-72c6-97c7-59f06c4e093c@enterprisedb.com>
 <4954a4f8-ba9d-d352-571b-7b94372ea5ae@enterprisedb.com>
 <CAA4eK1+dsdB_g=Y9UnBf8GQk1VnHnugbPWp3Ebt1JXtETcQUgQ@mail.gmail.com>
 <418ff5ac-1664-9038-8ac6-0dd308b5df07@enterprisedb.com>
 <e66ff0de-6f2d-baf9-aee9-ba412868df75@enterprisedb.com>
 <TY3PR01MB9889D457278B254CA87D1325F581A@TY3PR01MB9889.jpnprd01.prod.outlook.com>
 <c6a08e9d-7b08-76d4-197d-fab54039ed91@enterprisedb.com>
 <OS3PR01MB9882CCE9A9F037309078DC3DF587A@OS3PR01MB9882.jpnprd01.prod.outlook.com>
 <8a2f25d8-23d4-4ab8-2a23-7c8d2d976209@enterprisedb.com>
 <CAFiTN-vm3-bGfm-uJdzRLERMHozW8xjZHu4rdmtWR-rP-SJYMQ@mail.gmail.com>
 <CAA4eK1KxrWN35-BRHB6WLjpOLrLWpqabBp7BvK86waoCcFqfxw@mail.gmail.com>
 <CAFiTN-vY61cK+-XrcYEmWKDDOGBjbbwj3-H3qcMQ1KHKRyaPmQ@mail.gmail.com>
From: Tomas Vondra <tomas.vondra@enterprisedb.com>
In-Reply-To: <CAFiTN-vY61cK+-XrcYEmWKDDOGBjbbwj3-H3qcMQ1KHKRyaPmQ@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Archived-At: <https://www.postgresql.org/message-id/86cea9a2-9a79-0228-bae4-287cebe54bc7%40enterprisedb.com>
Precedence: bulk

On 12/6/23 12:05, Dilip Kumar wrote:
> On Wed, Dec 6, 2023 at 3:36 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>
>>> Why can't we use the same concept of
>>> SnapBuildDistributeNewCatalogSnapshot(), I mean we keep queuing the
>>> non-transactional changes (have some base snapshot before the first
>>> change), and whenever there is any catalog change, queue new snapshot
>>> change also in the queue of the non-transactional sequence change so
>>> that while sending it to downstream whenever it is necessary we will
>>> change the historic snapshot?
>>>
>>
>> Oh, do you mean maintain different historic snapshots and then switch
>> based on the change we are processing? I guess the other thing we need
>> to consider is the order of processing the changes if we maintain
>> separate queues that need to be processed.
> 
> I mean we will not specifically maintain the historic changes, but if
> there is any catalog change where we are pushing the snapshot to all
> the transaction's change queue, at the same time we will push this
> snapshot in the non-transactional sequence queue as well.  I am not
> sure what is the problem with the ordering? because we will be
> queueing all non-transactional sequence changes in a separate queue in
> the order they arrive and as soon as we process the next commit we
> will process all the non-transactional changes at that time.  Do you
> see issue with that?
> 

Isn't this (in principle) the idea of queuing the non-transactional
changes and then applying them on the next commit? Yes, I didn't get
very far with that, but I got stuck exactly on tracking which snapshot
to use, so if there's a way to do that, that'd fix my issue.

Also, would this mean we don't need to track the relfilenodes, if we're
able to query the catalog? Would we be able to check if the relfilenode
was created by the current xact?


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company