Message-ID: <3536d621-e3fc-c248-c03b-dd7bd0cfc4fc@enterprisedb.com>
Date: Fri, 11 Mar 2022 13:36:11 +0100
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.5.0
Subject: Re: Column Filtering in Logical Replication
Content-Language: en-US
To: "wangw.fnst@fujitsu.com" <wangw.fnst@fujitsu.com>,
 "houzj.fnst@fujitsu.com" <houzj.fnst@fujitsu.com>,
 Amit Kapila <amit.kapila16@gmail.com>
Cc: Peter Eisentraut <peter.eisentraut@enterprisedb.com>,
 Alvaro Herrera <alvherre@alvh.no-ip.org>,
 Justin Pryzby <pryzby@telsasoft.com>, Rahila Syed <rahilasyed90@gmail.com>,
 Peter Smith <smithpb2250@gmail.com>,
 pgsql-hackers <pgsql-hackers@postgresql.org>,
 "shiy.fnst@fujitsu.com" <shiy.fnst@fujitsu.com>
References: <202201120041.p24wvsfcsope@alvherre.pgsql>
 <ca91dc91-80ba-e954-213e-b4170a6160f5@enterprisedb.com>
 <d37d6709-7e9c-f422-f925-fa66ce1fc0e8@enterprisedb.com>
 <CAA4eK1KONbrck=hQTFSFgwio5rqu-82KZba2HLE_AcVa_RWYcQ@mail.gmail.com>
 <cac80284-7196-6c77-adb9-ef0976a3e21f@enterprisedb.com>
 <CAA4eK1JyoVty_BZOMJdx7VMkBTadvv8oOYrh9xTQ5BY+k-3+OA@mail.gmail.com>
 <d26d24dd-2fab-3c48-0162-2b7f84a9c893@enterprisedb.com>
 <CAA4eK1JJ5qgqbjfYQU9F4R1+OABxec-DQechQXus0fanDjaAVg@mail.gmail.com>
 <OS0PR01MB571614E4A39DD33669D5AFD6940A9@OS0PR01MB5716.jpnprd01.prod.outlook.com>
 <e4b0fb99-3d74-7333-1e71-74f990b641ee@enterprisedb.com>
 <78b8d385-57cb-9a3f-cfe5-9f300f15eced@enterprisedb.com>
 <822a8e40-287c-59ff-0ea9-35eb759f4fe6@enterprisedb.com>
 <OS3PR01MB6275E91117035094F53095C79E0C9@OS3PR01MB6275.jpnprd01.prod.outlook.com>
From: Tomas Vondra <tomas.vondra@enterprisedb.com>
In-Reply-To: 
 <OS3PR01MB6275E91117035094F53095C79E0C9@OS3PR01MB6275.jpnprd01.prod.outlook.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Archived-At: 
 <https://www.postgresql.org/message-id/3536d621-e3fc-c248-c03b-dd7bd0cfc4fc%40enterprisedb.com>
Precedence: bulk


On 3/11/22 08:05, wangw.fnst@fujitsu.com wrote:
> On Fri, Mar 11, 2022 at 9:57 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
>>
> Hi Tomas,
> Thanks for your patches.
> 
> On Mon, Mar 9, 2022 at 9:53 PM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
>> On Wed, Mar 9, 2022 at 6:04 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>> On Mon, Mar 7, 2022 at 11:18 PM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
>>>> On Fri, Mar 4, 2022 at 6:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>>>> Fetching column filter info in tablesync.c is quite expensive. It
>>>>> seems to be using four round-trips to get the complete info whereas
>>>>> for row-filter we use just one round trip. I think we should try to
>>>>> get both row filter and column filter info in just one round trip.
>>>>>
>>>>
>>>> Maybe, but I really don't think this is an issue.
>>>>
>>>
>>> I am not sure but it might matter for small tables. Leaving aside the
>>> performance issue, I think the current way will get the wrong column
>>> list in many cases: (a) The ALL TABLES IN SCHEMA case handling won't
>>> work for partitioned tables when the partitioned table is part of one
>>> schema and partition table is part of another schema. (b) The handling
>>> of partition tables in other cases will fetch incorrect lists as it
>>> tries to fetch the column list of all the partitions in the hierarchy.
>>>
>>> One of my colleagues has even tested these cases both for column
>>> filters and row filters and we find the behavior of row filter is okay
>>> whereas for column filter it uses the wrong column list. We will share
>>> the tests and results with you in a later email. We are trying to
>>> unify the column filter queries with row filter to make their behavior
>>> the same and will share the findings once it is done. I hope if we are
>>> able to achieve this that we will reduce the chances of bugs in this
>>> area.
>>>
>>
>> OK, I'll take a look at that email.
> I tried to get both the column filters and the row filters with one SQL, but
> it failed because I think the result is not easy to parse.
> 
> I noted that we use two SQLs to get column filters in the latest
> patches(20220311). I think maybe we could use one SQL to get column filters to
> reduce network cost. Like the SQL in the attachment.
> 

I'll take a look. But as I said before - I very much prefer SQL that is
easy to understand, and I don't think the one extra round trip is an
issue during tablesync (which is a very rare action).


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company