public inbox for [email protected]
help / color / mirror / Atom feedFrom: Amit Kapila <[email protected]>
To: Masahiko Sawada <[email protected]>
Cc: [email protected] <[email protected]>
Cc: [email protected] <[email protected]>
Cc: Peter Smith <[email protected]>
Cc: Dilip Kumar <[email protected]>
Cc: [email protected] <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Subject: Re: Perform streaming logical transactions by background workers and parallel apply
Date: Wed, 12 Oct 2022 11:34:43 +0530
Message-ID: <CAA4eK1JYFXEoFhJAvg1qU=nZrZLw_87X=2YWQGFBbcBGirAUwA@mail.gmail.com> (raw)
In-Reply-To: <CAD21AoDm3224e=se7=ZYt=R+v0_ZJ4E9dd5y2816_rTTCV+G+Q@mail.gmail.com>
References: <OS0PR01MB5716730C06159452335D870D947F9@OS0PR01MB5716.jpnprd01.prod.outlook.com>
<CAA4eK1Lu-6oXMk7ZaGYLwm3CRLBuzueGbasyHnNpJxu6Mq3mmg@mail.gmail.com>
<OS3PR01MB6275F145878B4A44586C46CE9E499@OS3PR01MB6275.jpnprd01.prod.outlook.com>
<CAA4eK1LMVdS6uM7Tw7ANL0BetAd76TKkmAXNNQa0haTe2tax6g@mail.gmail.com>
<OS3PR01MB6275EFC4B707650DAB9392859E4D9@OS3PR01MB6275.jpnprd01.prod.outlook.com>
<CAHut+PuVjRgGr4saN7qwq0oB8DANHVR7UfDiciB1Q3cYN54F6A@mail.gmail.com>
<OS3PR01MB627567CF4B96A23DAFAF77269E4E9@OS3PR01MB6275.jpnprd01.prod.outlook.com>
<CAA4eK1KEf6w1azJytYnLhd2Lo-=rPkRh1Jv2ePcvs=SKqNQnAQ@mail.gmail.com>
<CAA4eK1KjGNA8T8O77rRhkv6bRT6OsdQaEy--2hNrJFCc80bN0A@mail.gmail.com>
<OS0PR01MB571620741B90C769D2727D7F94559@OS0PR01MB5716.jpnprd01.prod.outlook.com>
<CAD21AoBLPDPCE4jj16ZRE9J8AoxLtvo3yasToaP5kRp6++=yyg@mail.gmail.com>
<OS0PR01MB57161EFFC1D03B9080A9B812945C9@OS0PR01MB5716.jpnprd01.prod.outlook.com>
<CAD21AoCRDWMebeDNeMNgf4LpTA23si_1xU0T_F2Rh+BD9HU6=Q@mail.gmail.com>
<CAA4eK1LJoA868HCMmrzPqfcFL=bcFtAi6WWSJ0NKKQp8gspYVQ@mail.gmail.com>
<CAD21AoDm3224e=se7=ZYt=R+v0_ZJ4E9dd5y2816_rTTCV+G+Q@mail.gmail.com>
On Tue, Oct 11, 2022 at 5:52 AM Masahiko Sawada <[email protected]> wrote:
>
> On Fri, Oct 7, 2022 at 2:00 PM Amit Kapila <[email protected]> wrote:
> >
> > About your point that having different partition structures for
> > publisher and subscriber, I don't know how common it will be once we
> > have DDL replication. Also, the default value of
> > publish_via_partition_root is false which doesn't seem to indicate
> > that this is a quite common case.
>
> So how can we consider these concurrent issues that could happen only
> when streaming = 'parallel'? Can we restrict some use cases to avoid
> the problem or can we have a safeguard against these conflicts?
>
Yeah, right now the strategy is to disallow parallel apply for such
cases as you can see in *0003* patch.
> We
> could find a new problematic scenario in the future and if it happens,
> logical replication gets stuck, it cannot be resolved only by apply
> workers themselves.
>
I think users can change streaming option to on/off and internally the
parallel apply worker can detect and restart to allow replication to
proceed. Having said that, I think that would be a bug in the code and
we should try to fix it. We may need to disable parallel apply in the
problematic case.
The other ideas that occurred to me in this regard are (a) provide a
reloption (say parallel_apply) at table level and we can use that to
bypass various checks like different Unique Key between
publisher/subscriber, constraints/expressions having mutable
functions, Foreign Key (when enabled on subscriber), operations on
Partitioned Table. We can't detect whether those are safe or not
(primarily because of a different structure in publisher and
subscriber) so we prohibit parallel apply but if users use this
option, we can allow it even in those cases. (b) While enabling the
parallel option in the subscription, we can try to match all the
table(s) information of the publisher/subscriber. It will be tricky to
make this work because say even if match some trigger function name,
we won't be able to match the function body. The other thing is when
at a later point the table definition is changed on the subscriber, we
need to again validate the information between publisher and
subscriber which I think would be difficult as we would be already in
between processing some message and getting information from the
publisher at that stage won't be possible.
Thoughts?
--
With Regards,
Amit Kapila.
view thread (625+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: Perform streaming logical transactions by background workers and parallel apply
In-Reply-To: <CAA4eK1JYFXEoFhJAvg1qU=nZrZLw_87X=2YWQGFBbcBGirAUwA@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox