public inbox for [email protected]
help / color / mirror / Atom feedFrom: Amit Kapila <[email protected]>
To: Masahiko Sawada <[email protected]>
Cc: Greg Nancarrow <[email protected]>
Cc: vignesh C <[email protected]>
Cc: Peter Eisentraut <[email protected]>
Cc: [email protected] <[email protected]>
Cc: [email protected] <[email protected]>
Cc: [email protected] <[email protected]>
Cc: Alexey Lesovsky <[email protected]>
Cc: PostgreSQL-development <[email protected]>
Subject: Re: Skipping logical replication transactions on subscriber side
Date: Wed, 5 Jan 2022 09:00:57 +0530
Message-ID: <CAA4eK1L2-CEz1Mgad9ZuoQ59BVieGmj5Rmcj8GOT6LSdP-3TbA@mail.gmail.com> (raw)
In-Reply-To: <CAD21AoB2mYLQx6jmcUVtsQ0iR7YOH2CtJ+p9OieDyhH14ynmyw@mail.gmail.com>
References: <CAA4eK1KzZp5YC34exW3Yb4Vk9v5iMW0YuJCQsWeTOeF7ejLdXw@mail.gmail.com>
<CAA4eK1JfJBJZyTEmG-z=Uu32aMTx8d+aLLorzDXt6s=ERa_ykA@mail.gmail.com>
<CALDaNm2_WY2t2GnzW1-QL823+xk5cuC0zc4Eep8fjYeLB0jfRA@mail.gmail.com>
<CAA4eK1JheT_MTmQiGfWZJeHSEJGzd-mgPPwBwA=YWjhcTfykPA@mail.gmail.com>
<CAD21AoBu2f2R6Bc9qWpky8v_h822zdkm0419j60Ph_T9ciaVsw@mail.gmail.com>
<CAD21AoAnexmfLz5dA4mPXT10b=wt0DEsp-CcAbLCZbhuh8zZoQ@mail.gmail.com>
<OS0PR01MB5716D985D4F2430E817CE30B94689@OS0PR01MB5716.jpnprd01.prod.outlook.com>
<CAA4eK1K_yFm_vLyNi-unqJNYk1edNkuOrqjBRWVTF5bFaTSsQg@mail.gmail.com>
<CAD21AoAUogU_4+yPrwJeaZhOUV06USE+TFJHt5OD48O43NFvnQ@mail.gmail.com>
<CAA4eK1JOsqPHZTOeifp+Q8pWgrn1qzks3ojZnSHp-1QZFcjyQQ@mail.gmail.com>
<CAD21AoAqDVCVzSpv6DOMLqLeO+m4Jh=dKv9a8Cc4qthdpYzqFw@mail.gmail.com>
<OS0PR01MB571603F90202D6709B5DF5B594689@OS0PR01MB5716.jpnprd01.prod.outlook.com>
<CAA4eK1KD_C_0LSxaYB0UbG59VOgjf4mXBeSYbVWCLXAnnuqnPw@mail.gmail.com>
<[email protected]>
<CAA4eK1LpbjYgEjK0=v+1wAyWQMXALPoh=BS5cQeFam7ye3rQKg@mail.gmail.com>
<CAD21AoAzxB+7U8QWjHnaHY-oa-SApBLc4VqDjVewQgkem1Prmw@mail.gmail.com>
<CAA4eK1K2yKC-Eve+tACyfAkniy4VvLqiGU-F9Vtg0yNO7MjZzw@mail.gmail.com>
<CAD21AoBNPPEySdB8M7mG9KSg5+F9nfbo8LrWU7d=5uvQ18Uumw@mail.gmail.com>
<CAA4eK1LasocmhFeBb0D4ixf_J=pDr1OYdyTnEvbhcYToqA=GMw@mail.gmail.com>
<CAD21AoBnxmRGMv5CZHVosp+g-vUzZtcJ0ecxfvk+aLEqTKFa-g@mail.gmail.com>
<CAA4eK1Lfn2VPEK2J=+JOZkmkdtTwVd2GyPwvr7=iVUaSMgNTXw@mail.gmail.com>
<CAD21AoAGiUYhiNgyr4A9928gDmbU95ai9K0Od7c2MeP_dXuDbA@mail.gmail.com>
<CAA4eK1LCfWf8JVnzdmk3PORZHgfC=DUuJh5dUD9PP5QeNknJMQ@mail.gmail.com>
<CAD21AoCAL08VPW9GeVdr5E=mShs2PG65KhujJfP6Gntz==zknQ@mail.gmail.com>
<CAA4eK1+ET_L03k_GW2ruOpLK5bfJLxaY+4sqo161FgoUyim4kw@mail.gmail.com>
<CAD21AoCG+X+xdbW=ZUxaqGN4G+b6JJW4vKdxbu6oRxa6NY8XNw@mail.gmail.com>
<CAA4eK1La0Oum41L9=bgdU6AjJXY7TvPXEWhgm+sKtdg3JCY27w@mail.gmail.com>
<CAD21AoBRDU3dzUbw0zeQfKfJ79rWCbYKEaTLoJbHpP6VomiY_Q@mail.gmail.com>
<CALDaNm0YD=AqGpPBk6ZtvUiY6XpiTXrzGrrfzPShOhAcVnJ8Fg@mail.gmail.com>
<CAJcOf-f4AHJ4BTSa_u3VE7XHDb4BA4-J_-FWcHUM5O=x1yXBpQ@mail.gmail.com>
<CAD21AoCjmg=m=hQyfdXCi7Ywt14jz8LkbShn0Eyw9VJEwnLD2g@mail.gmail.com>
<CAA4eK1J2-R5vZ9SEZAu2yXKv-g2KDYZuM6MyjQX8xmiFt75+eQ@mail.gmail.com>
<CAD21AoD==vwyNWsrndRSbQWEnHujpOTC059u-arTFF3hxAye5g@mail.gmail.com>
<CAA4eK1+s4AeEHrSnJV8WjdYBDoBHLqpAU8ZyU=0LWAbk4KSGqA@mail.gmail.com>
<CAD21AoB7U3DorHAJEezTqKL_70KJOWr2rO75eMMNUMbE4OsJSA@mail.gmail.com>
<CAA4eK1Ln6X=OKoNJg1yrtfPmUFFSTeEonou5C-6ncrGT0hO6Xw@mail.gmail.com>
<CAD21AoC96A=vUYTRFs7hyrSk_5YEX7rgMVwfYKdNTXQvsomAdQ@mail.gmail.com>
<CAD21AoB2mYLQx6jmcUVtsQ0iR7YOH2CtJ+p9OieDyhH14ynmyw@mail.gmail.com>
On Mon, Dec 27, 2021 at 9:54 AM Masahiko Sawada <[email protected]> wrote:
>
> On Thu, Dec 16, 2021 at 2:42 PM Masahiko Sawada <[email protected]> wrote:
> >
> > On Thu, Dec 16, 2021 at 2:21 PM Amit Kapila <[email protected]> wrote:
> > >
> > > On Thu, Dec 16, 2021 at 10:37 AM Masahiko Sawada <[email protected]> wrote:
> > > >
> > > > On Thu, Dec 16, 2021 at 11:43 AM Amit Kapila <[email protected]> wrote:
> > > > >
> > > > > I thought we just want to lock before clearing the skip_xid something
> > > > > like take the lock, check if the skip_xid in the catalog is the same
> > > > > as we have skipped, if it is the same then clear it, otherwise, leave
> > > > > it as it is. How will that disallow users to change skip_xid when we
> > > > > are skipping changes?
> > > >
> > > > Oh I thought we wanted to keep holding the lock while skipping changes
> > > > (changing skip_xid requires acquiring the lock).
> > > >
> > > > So if skip_xid is already changed, the apply worker would do
> > > > replorigin_advance() with WAL logging, instead of committing the
> > > > catalog change?
> > > >
> > >
> > > Right. BTW, how are you planning to advance the origin? Normally, a
> > > commit transaction would do it but when we are skipping all changes,
> > > the commit might not do it as there won't be any transaction id
> > > assigned.
> >
> > I've not tested it yet but replorigin_advance() with wal_log = true
> > seems to work for this case.
>
> I've tested it and realized that we cannot use replorigin_advance()
> for this purpose without changes. That is, the current
> replorigin_advance() doesn't allow to advance the origin by the owner:
>
> /* Make sure it's not used by somebody else */
> if (replication_state->acquired_by != 0)
> {
> ereport(ERROR,
> (errcode(ERRCODE_OBJECT_IN_USE),
> errmsg("replication origin with OID %d is already
> active for PID %d",
> replication_state->roident,
> replication_state->acquired_by)));
> }
>
> So we need to change it so that the origin owner can advance its
> origin, which makes sense to me.
>
> Also, when we have to update the origin instead of committing the
> catalog change while updating the origin, we cannot record the origin
> timestamp.
>
Is it because we currently update the origin timestamp with commit record?
> This behavior makes sense to me because we skipped the
> transaction. But ISTM it’s not good if we emit the origin timestamp
> only when directly updating the origin. So probably we need to always
> omit origin timestamp.
>
Do you mean to say that you want to omit it even when we are
committing the changes?
> Apart from that, I'm vaguely concerned that the logic seems to be
> getting complex. Probably it comes from the fact that we store
> skip_xid in the catalog and update the catalog to clear/set the
> skip_xid. It might be worth revisiting the idea of storing skip_xid on
> shmem (e.g., ReplicationState)?
>
IIRC, the problem with that idea was that we won't remember skip_xid
information after server restart and the user won't even know that it
has to set it again.
--
With Regards,
Amit Kapila.
view thread (2+ messages)
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: Skipping logical replication transactions on subscriber side
In-Reply-To: <CAA4eK1L2-CEz1Mgad9ZuoQ59BVieGmj5Rmcj8GOT6LSdP-3TbA@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox