Message-ID: <f217341d-6841-390d-3dd2-89f9d1de17fe@enterprisedb.com>
Date: Wed, 16 Mar 2022 18:24:58 +0100
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.6.2
Subject: Re: Column Filtering in Logical Replication
Content-Language: en-US
From: Tomas Vondra <tomas.vondra@enterprisedb.com>
To: Amit Kapila <amit.kapila16@gmail.com>,
 "houzj.fnst@fujitsu.com" <houzj.fnst@fujitsu.com>
Cc: Peter Eisentraut <peter.eisentraut@enterprisedb.com>,
 Alvaro Herrera <alvherre@alvh.no-ip.org>,
 Justin Pryzby <pryzby@telsasoft.com>, Rahila Syed <rahilasyed90@gmail.com>,
 Peter Smith <smithpb2250@gmail.com>,
 pgsql-hackers <pgsql-hackers@postgresql.org>,
 "shiy.fnst@fujitsu.com" <shiy.fnst@fujitsu.com>
References: <202201120041.p24wvsfcsope@alvherre.pgsql>
 <CAA4eK1JyoVty_BZOMJdx7VMkBTadvv8oOYrh9xTQ5BY+k-3+OA@mail.gmail.com>
 <d26d24dd-2fab-3c48-0162-2b7f84a9c893@enterprisedb.com>
 <CAA4eK1JJ5qgqbjfYQU9F4R1+OABxec-DQechQXus0fanDjaAVg@mail.gmail.com>
 <OS0PR01MB571614E4A39DD33669D5AFD6940A9@OS0PR01MB5716.jpnprd01.prod.outlook.com>
 <e4b0fb99-3d74-7333-1e71-74f990b641ee@enterprisedb.com>
 <78b8d385-57cb-9a3f-cfe5-9f300f15eced@enterprisedb.com>
 <822a8e40-287c-59ff-0ea9-35eb759f4fe6@enterprisedb.com>
 <CAA4eK1J9UDnM-xN7viu4eO_DDL_PqMG9F6ZsskqEU7snujXi+A@mail.gmail.com>
 <2b8471f9-3849-c91f-ef82-9181d4bca0a7@enterprisedb.com>
 <CAA4eK1KUpOnFam+2bBuGGX-_pZbmO_ShJV6G=GT-RyM7gQSvWg@mail.gmail.com>
 <16e83ce3-8415-b732-37b2-c3329a2c13a1@enterprisedb.com>
 <OS0PR01MB57164514E1DE1E579B0B6C46940F9@OS0PR01MB5716.jpnprd01.prod.outlook.com>
 <CAA4eK1KvFAygPbFu=cUvnqhmTLPpjzqp8p4UNSsLkiZmvFp8oQ@mail.gmail.com>
 <bc6918cc-2025-6c91-2ca4-c237af67e97d@enterprisedb.com>
In-Reply-To: <bc6918cc-2025-6c91-2ca4-c237af67e97d@enterprisedb.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Archived-At: <https://www.postgresql.org/message-id/f217341d-6841-390d-3dd2-89f9d1de17fe%40enterprisedb.com>
Precedence: bulk


On 3/15/22 09:30, Tomas Vondra wrote:
> 
> 
> On 3/15/22 05:43, Amit Kapila wrote:
>> On Mon, Mar 14, 2022 at 4:42 PM houzj.fnst@fujitsu.com
>> <houzj.fnst@fujitsu.com> wrote:
>>>
>>> On Monday, March 14, 2022 5:08 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
>>>>
>>>> On 3/12/22 05:30, Amit Kapila wrote:
>>>>>> ...
>>>>>
>>>>> Okay, please find attached. I have done basic testing of this, if we
>>>>> agree with this approach then this will require some more testing.
>>>>>
>>>>
>>>> Thanks, the proposed changes seem like a clear improvement, so I've
>>>> added them, with some minor tweaks (mostly to comments).
>>>
>>> Hi,
>>>
>>> Thanks for updating the patches !
>>> And sorry for the row filter bug caused by my mistake.
>>>
>>> I looked at the two fixup patches. I am thinking would it be better if we
>>> add one testcase for these two bugs? Maybe like the attachment.
>>>
>>
>> Your tests look good to me. We might want to add some comments for
>> each test but I guess that can be done before committing. Tomas, it
>> seems you are planning to push these bug fixes, do let me know if you
>> want me to take care of these while you focus on the main patch? I
>> think the first patch needs to be backpatched till 13 and the second
>> one is for just HEAD.
>>
> 
> Yeah, I plan to push the fixes later today. I'll polish them a bit
> first, and merge the tests (shared by Hou zj) into the patches etc.
> 

I've pushed (and backpatched to 13+) the fix for the publish_as_relid
issue, including the test. I tweaked the test a bit, to check both
orderings of the publication list.

While doing that, I discovered yet ANOTHER bug in the publish_as_relid
loop, affecting 12+13. There was a break once all actions were
replicated, but skipping additional publications ignores the fact that
the publications may replicate a different (higher-up) ancestor.

I removed the break, if anyone thinks this optimization is worth it we
could still do that once we replicate the top-most ancestor.


I'll push the second fix soon.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company