From: Tomas Vondra <tomas.vondra@enterprisedb.com>
To: wangw.fnst@fujitsu.com <wangw.fnst@fujitsu.com>
To: houzj.fnst@fujitsu.com <houzj.fnst@fujitsu.com>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
Cc: Alvaro Herrera <alvherre@alvh.no-ip.org>
Cc: Justin Pryzby <pryzby@telsasoft.com>
Cc: Rahila Syed <rahilasyed90@gmail.com>
Cc: Peter Smith <smithpb2250@gmail.com>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Cc: shiy.fnst@fujitsu.com <shiy.fnst@fujitsu.com>
Subject: Re: Column Filtering in Logical Replication
Date: Fri, 11 Mar 2022 13:36:11 +0100
Message-ID: <3536d621-e3fc-c248-c03b-dd7bd0cfc4fc@enterprisedb.com> (raw)
In-Reply-To: <OS3PR01MB6275E91117035094F53095C79E0C9@OS3PR01MB6275.jpnprd01.prod.outlook.com>
References: <202201120041.p24wvsfcsope@alvherre.pgsql>
<ca91dc91-80ba-e954-213e-b4170a6160f5@enterprisedb.com>
<d37d6709-7e9c-f422-f925-fa66ce1fc0e8@enterprisedb.com>
<CAA4eK1KONbrck=hQTFSFgwio5rqu-82KZba2HLE_AcVa_RWYcQ@mail.gmail.com>
<cac80284-7196-6c77-adb9-ef0976a3e21f@enterprisedb.com>
<CAA4eK1JyoVty_BZOMJdx7VMkBTadvv8oOYrh9xTQ5BY+k-3+OA@mail.gmail.com>
<d26d24dd-2fab-3c48-0162-2b7f84a9c893@enterprisedb.com>
<CAA4eK1JJ5qgqbjfYQU9F4R1+OABxec-DQechQXus0fanDjaAVg@mail.gmail.com>
<OS0PR01MB571614E4A39DD33669D5AFD6940A9@OS0PR01MB5716.jpnprd01.prod.outlook.com>
<e4b0fb99-3d74-7333-1e71-74f990b641ee@enterprisedb.com>
<78b8d385-57cb-9a3f-cfe5-9f300f15eced@enterprisedb.com>
<822a8e40-287c-59ff-0ea9-35eb759f4fe6@enterprisedb.com>
<OS3PR01MB6275E91117035094F53095C79E0C9@OS3PR01MB6275.jpnprd01.prod.outlook.com>
On 3/11/22 08:05, wangw.fnst@fujitsu.com wrote:
> On Fri, Mar 11, 2022 at 9:57 AM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
>>
> Hi Tomas,
> Thanks for your patches.
>
> On Mon, Mar 9, 2022 at 9:53 PM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
>> On Wed, Mar 9, 2022 at 6:04 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>> On Mon, Mar 7, 2022 at 11:18 PM Tomas Vondra <tomas.vondra@enterprisedb.com> wrote:
>>>> On Fri, Mar 4, 2022 at 6:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>>>>> Fetching column filter info in tablesync.c is quite expensive. It
>>>>> seems to be using four round-trips to get the complete info whereas
>>>>> for row-filter we use just one round trip. I think we should try to
>>>>> get both row filter and column filter info in just one round trip.
>>>>>
>>>>
>>>> Maybe, but I really don't think this is an issue.
>>>>
>>>
>>> I am not sure but it might matter for small tables. Leaving aside the
>>> performance issue, I think the current way will get the wrong column
>>> list in many cases: (a) The ALL TABLES IN SCHEMA case handling won't
>>> work for partitioned tables when the partitioned table is part of one
>>> schema and partition table is part of another schema. (b) The handling
>>> of partition tables in other cases will fetch incorrect lists as it
>>> tries to fetch the column list of all the partitions in the hierarchy.
>>>
>>> One of my colleagues has even tested these cases both for column
>>> filters and row filters and we find the behavior of row filter is okay
>>> whereas for column filter it uses the wrong column list. We will share
>>> the tests and results with you in a later email. We are trying to
>>> unify the column filter queries with row filter to make their behavior
>>> the same and will share the findings once it is done. I hope if we are
>>> able to achieve this that we will reduce the chances of bugs in this
>>> area.
>>>
>>
>> OK, I'll take a look at that email.
> I tried to get both the column filters and the row filters with one SQL, but
> it failed because I think the result is not easy to parse.
>
> I noted that we use two SQLs to get column filters in the latest
> patches(20220311). I think maybe we could use one SQL to get column filters to
> reduce network cost. Like the SQL in the attachment.
>
I'll take a look. But as I said before - I very much prefer SQL that is
easy to understand, and I don't think the one extra round trip is an
issue during tablesync (which is a very rare action).
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: tomas.vondra@enterprisedb.com, wangw.fnst@fujitsu.com, houzj.fnst@fujitsu.com, amit.kapila16@gmail.com, peter.eisentraut@enterprisedb.com, alvherre@alvh.no-ip.org, pryzby@telsasoft.com, rahilasyed90@gmail.com, smithpb2250@gmail.com, shiy.fnst@fujitsu.com
Subject: Re: Column Filtering in Logical Replication
In-Reply-To: <3536d621-e3fc-c248-c03b-dd7bd0cfc4fc@enterprisedb.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox