From: =?utf-8?q?=D0=94=D0=B0=D0=B2=D1=8B=D0=B4=D0=BE=D0=B2_=D0=92=D0=B8=D1=82=D0=B0=D0=BB=D0=B8=D0=B9?= <v.davydov@postgrespro.ru>
In-Reply-To: <CAA4eK1KOs3s6syZqUgrd2WvjTz64SGf0ToZcRoPMCKKH+M0YFQ@mail.gmail.com>
Content-Type: multipart/alternative; boundary="----=_=-_OpenGroupware_org_NGMime-630258-1708708306.652385-0------"
References: <8fab8-65d74c80-1-2f28e880@39088166> <CAA4eK1KOs3s6syZqUgrd2WvjTz64SGf0ToZcRoPMCKKH+M0YFQ@mail.gmail.com>
Date: Fri, 23 Feb 2024 20:11:46 +0300
Cc: pgsql-hackers@lists.postgresql.org
To: "Amit Kapila" <amit.kapila16@gmail.com>
MIME-Version: 1.0
Message-ID: <99df2-65d8d200-1-63c26880@172687752>
Subject: =?utf-8?q?Re=3A?= Slow catchup of 2PC (twophase) transactions on replica in 
 LR
User-Agent: SOGoMail 5.9.1
Archived-At: <https://www.postgresql.org/message-id/99df2-65d8d200-1-63c26880%40172687752>
Precedence: bulk

------=_=-_OpenGroupware_org_NGMime-630258-1708708306.652385-0------
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Content-Length: 2818


Hi Amit,
Amit Kapila <amit.kapila16@gmail.com> wrote:
I don't see we do anything specific for 2PC transactions to make them b=
ehave differently than regular transactions with respect to synchronous=
=5Fcommit setting. What makes you think so? Can you pin point the code =
you are referring to?Yes, sure. The function RecordTransactionCommitPre=
pared is called on prepared transaction commit (twophase.c). It calls X=
LogFlush unconditionally. The function RecordTransactionCommit (for reg=
ular transactions, xact.c) calls XLogFlush if synchronous=5Fcommit > OF=
F, otherwise it calls=C2=A0XLogSetAsyncXactLSN.

There is some comment in RecordTransactionCommitPrepared (by Bruce Momj=
ian) that shows that async commit is not supported yet:
/*
* We don't currently try to sleep before flush here ... nor is there an=
y
* support for async commit of a prepared xact (the very idea is probabl=
y
* a contradiction)
*/
/* Flush XLOG to disk */
XLogFlush(recptr);
Right, I think for this we need to implement parallel apply.Yes, parall=
el apply is a good point. But, I believe, it will not work if asynchron=
ous commit is not supported. You have only one receiver process which s=
hould dispatch incoming messages to parallel workers. I guess, you will=
 never reach such rate of parallel execution on replica as on the maste=
r with multiple backends.
=C2=A0
Can you be a bit more specific about what exactly you have in mind to a=
chieve the above solutions?My proposal is to implement async commit for=
 2PC transactions as it is for regular transactions. It should signific=
antly speedup the catchup process. Then, think how to apply in parallel=
, which is much diffcult to do. The current problem is to get 2PC state=
 from the WAL on commit prepared. At this moment, the WAL is not flushe=
d yet, commit function waits until WAL with 2PC state is to be flushed.=
 I just tried to do it in my sandbox and found such a problem. Inabilit=
y to get 2PC state from unflushed WAL stops me right now. I think about=
 possible solutions.

The idea with enableFsync is not a suitable solution, in general, I thi=
nk. I just pointed it as an alternate idea. You just do enableFsync =3D=
 false before prepare or commit prepared and do enableFsync =3D true af=
ter these functions. In this case, 2PC records will not be fsync-ed, bu=
t FlushPtr will be increased. Thus, 2PC state can be read from WAL on c=
ommit prepared without waiting. To make it work correctly, I guess, we =
have to do some additional work to keep more wal on the master and filt=
er some duplicate transactions on the replica, if replica restarts duri=
ng catchup.
=E2=80=8B=E2=80=8B=E2=80=8B=E2=80=8B=E2=80=8B=E2=80=8B
With best regards,
=E2=80=8B=E2=80=8B=E2=80=8B=E2=80=8B=E2=80=8B=E2=80=8BVitaly Davydov

=C2=A0

------=_=-_OpenGroupware_org_NGMime-630258-1708708306.652385-0------
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Content-Length: 3075

<html>Hi Amit,<blockquote><p>Amit Kapila &lt;amit.kapila16@gmail.com&gt=
; wrote:</p></blockquote><blockquote><p>I don't see we do anything spec=
ific for 2PC transactions to make them behave differently than regular =
transactions with respect to synchronous=5Fcommit setting. What makes y=
ou think so? Can you pin point the code you are referring to?</p></bloc=
kquote>Yes, sure. The function RecordTransactionCommitPrepared is calle=
d on prepared transaction commit (twophase.c). It calls XLogFlush uncon=
ditionally. The function RecordTransactionCommit (for regular transacti=
ons, xact.c) calls XLogFlush if synchronous=5Fcommit &gt; OFF, otherwis=
e it calls&nbsp;XLogSetAsyncXactLSN.<br /><br />There is some comment i=
n RecordTransactionCommitPrepared (by Bruce Momjian) that shows that as=
ync commit is not supported yet:<br />/*<br />* We don't currently try =
to sleep before flush here ... nor is there any<br />* support for asyn=
c commit of a prepared xact (the very idea is probably<br />* a contrad=
iction)<br />*/<br />/* Flush XLOG to disk */<br />XLogFlush(recptr);<b=
lockquote><p>Right, I think for this we need to implement parallel appl=
y.</p></blockquote>Yes, parallel apply is a good point. But, I believe,=
 it will not work if asynchronous commit is not supported. You have onl=
y one receiver process which should dispatch incoming messages to paral=
lel workers. I guess, you will never reach such rate of parallel execut=
ion on replica as on the master with multiple backends.<br />&nbsp;<blo=
ckquote><p>Can you be a bit more specific about what exactly you have i=
n mind to achieve the above solutions?</p></blockquote>My proposal is t=
o implement async commit for 2PC transactions as it is for regular tran=
sactions. It should significantly speedup the catchup process. Then, th=
ink how to apply in parallel, which is much diffcult to do. The current=
 problem is to get 2PC state from the WAL on commit prepared. At this m=
oment, the WAL is not flushed yet, commit function waits until WAL with=
 2PC state is to be flushed. I just tried to do it in my sandbox and fo=
und such a problem. Inability to get 2PC state from unflushed WAL stops=
 me right now. I think about possible solutions.<br /><br />The idea wi=
th enableFsync is not a suitable solution, in general, I think. I just =
pointed it as an alternate idea. You just do enableFsync =3D false befo=
re prepare or commit prepared and do enableFsync =3D true after these f=
unctions. In this case, 2PC records will not be fsync-ed, but FlushPtr =
will be increased. Thus, 2PC state can be read from WAL on commit prepa=
red without waiting. To make it work correctly, I guess, we have to do =
some additional work to keep more wal on the master and filter some dup=
licate transactions on the replica, if replica restarts during catchup.=
<br />=E2=80=8B=E2=80=8B=E2=80=8B=E2=80=8B=E2=80=8B=E2=80=8B<br />With =
best regards,<br />=E2=80=8B=E2=80=8B=E2=80=8B=E2=80=8B=E2=80=8B=E2=80=8B=
Vitaly Davydov<br /><br />&nbsp;</html>

------=_=-_OpenGroupware_org_NGMime-630258-1708708306.652385-0--------