MIME-Version: 1.0
References: <CAA4eK1KdVRffYEYiv9=4tVWhP=Jkfh1h6FM0A1R0NEe8cLT8+g@mail.gmail.com>
 <CAA4eK1JU4J3-Z=fcMcyQw-iG-DNr=u6w_g-iM_4xwg1pKupkFQ@mail.gmail.com>
 <OS0PR01MB57163693482C5470C41914AF94EA9@OS0PR01MB5716.jpnprd01.prod.outlook.com>
 <CAHut+PuYJz-26QWMSRN_HwiJNfYG4ZxEV3RGE-VyZtXXeFm+cw@mail.gmail.com>
 <OS0PR01MB5716C1C1D8244BA86311193894EB9@OS0PR01MB5716.jpnprd01.prod.outlook.com>
 <CAA4eK1LrADDNXKRVYox4Q+zSFQvtiQu-7piyd_PCmVjg1gsN2w@mail.gmail.com>
 <OS0PR01MB57169EF567017861EACBD10D94E99@OS0PR01MB5716.jpnprd01.prod.outlook.com>
 <OS0PR01MB571623EFD8903C93424EFADC94E99@OS0PR01MB5716.jpnprd01.prod.outlook.com>
 <OS0PR01MB57166C4E79559C9A5570838C94EC9@OS0PR01MB5716.jpnprd01.prod.outlook.com>
 <CAA4eK1+NV9BDPjh5XB9PnZYNdnQht8+2JGRcerJpSUXtYPtQsg@mail.gmail.com>
 <OS3PR01MB6275B61076717E4CE9E079D19EED9@OS3PR01MB6275.jpnprd01.prod.outlook.com>
 <CAA4eK1LXch0EDB9RCDiqqmM3S9_cam=svMMK08i8Fmspms=6Xg@mail.gmail.com>
 <OS3PR01MB62756CC6FFBDD431F2878ACE9EF29@OS3PR01MB6275.jpnprd01.prod.outlook.com>
 <CAA4eK1Konv5gm2bZhej_CZqotr=qhUHnhhiJixKLRmCMsg+Egw@mail.gmail.com>
 <OS3PR01MB627586DA752A9E07F02FB7609EF09@OS3PR01MB6275.jpnprd01.prod.outlook.com>
 <CAA4eK1+63gaq1ruaUTvhM6-GRWi0waZT7X+TJL2tx+dWzisveQ@mail.gmail.com>
 <OS3PR01MB6275D4B22266412FEEBDB3AE9EF49@OS3PR01MB6275.jpnprd01.prod.outlook.com>
 <CAD21AoC=PCkeURevhZdNjcKHYws=paPcotgG1fAnq9g8GdwGXw@mail.gmail.com>
 <OS0PR01MB5716087C537FA8311F208B6094F59@OS0PR01MB5716.jpnprd01.prod.outlook.com>
 <CAFiTN-vy5aD2qEe2g8ssh+n=iyusFm84FxPsUyWOMeQdvbN3ow@mail.gmail.com>
 <CAA4eK1LdtevYSW_Lk5MJJeCMWqQtUgKCtPkX4FNKVS-ptEBGxw@mail.gmail.com>
 <CAFiTN-u_hFNEB6iQTtY9sLV_0fYX87OGH5BWbjTALYgfoHnxhw@mail.gmail.com>
 <OS0PR01MB5716C2CC59810717EC9601EE94FA9@OS0PR01MB5716.jpnprd01.prod.outlook.com>
 <CAFiTN-sTYk=h75Jn1a7ee+5hOcdQFjKGDvF_0NWQQXmoBv4A+A@mail.gmail.com> <OS0PR01MB57162495C97CAEA88AC8E9DB94FA9@OS0PR01MB5716.jpnprd01.prod.outlook.com>
In-Reply-To: <OS0PR01MB57162495C97CAEA88AC8E9DB94FA9@OS0PR01MB5716.jpnprd01.prod.outlook.com>
From: Dilip Kumar <dilipbalaut@gmail.com>
Date: Thu, 5 Jan 2023 17:23:36 +0530
Message-ID: <CAFiTN-srQ2qZsx2uK6egT_jz9weBGm-Ff+gE8ObMu3sWYNKpFQ@mail.gmail.com>
Subject: Re: Perform streaming logical transactions by background workers and
 parallel apply
To: "houzj.fnst@fujitsu.com" <houzj.fnst@fujitsu.com>
Cc: Amit Kapila <amit.kapila16@gmail.com>, Masahiko Sawada <sawada.mshk@gmail.com>, 
	"wangw.fnst@fujitsu.com" <wangw.fnst@fujitsu.com>, Peter Smith <smithpb2250@gmail.com>, 
	"shiy.fnst@fujitsu.com" <shiy.fnst@fujitsu.com>, 
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Archived-At: <https://www.postgresql.org/message-id/CAFiTN-srQ2qZsx2uK6egT_jz9weBGm-Ff%2BgE8ObMu3sWYNKpFQ%40mail.gmail.com>
Precedence: bulk

On Thu, Jan 5, 2023 at 5:03 PM houzj.fnst@fujitsu.com
<houzj.fnst@fujitsu.com> wrote:
>
> On Thursday, January 5, 2023 4:22 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> >

> Thanks for reporting the problem.
>
> After analyzing the behavior, I think it's a bug on publisher side which
> is not directly related to parallel apply.
>
> I think the root reason is that we didn't try to send a stream end(stream
> abort) message to subscriber for the crashed transaction which was streamed
> before.
> The behavior is that, after restarting, the publisher will start to decode the
> transaction that aborted due to crash, and when try to stream the first change
> of that transaction, it will send a stream start message but then it realizes
> that the transaction was aborted, so it will enter the PG_CATCH block of
> ReorderBufferProcessTXN() and call ReorderBufferResetTXN() which send the
> stream stop message. And in this case, there would be a parallel apply worker
> started on subscriber waiting for stream end message which will never come.

I suspected it but didn't analyze this.

> I think the same behavior happens for the non-parallel mode which will cause
> a stream file left on subscriber and will not be cleaned until the apply worker is
> restarted.
> To fix it, I think we need to send a stream abort message when we are cleaning
> up crashed transaction on publisher(e.g., in ReorderBufferAbortOld()). And here
> is a tiny patch which change the same. I have confirmed that the bug is fixed
> and all regression tests pass.
>
> What do you think ?
> I will start a new thread and try to write a testcase if possible
> after reaching a consensus.

I think your analysis looks correct and we can raise this in a new thread.


-- 
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com