MIME-Version: 1.0
References: <CAB+=1TV-b8=RGKVFN0qj5Nxdgbj-JYCNh4ZqFjxxmQPpFD6=PQ@mail.gmail.com>
 <fd6afa3b-f5db-4632-8e25-678ef66703d5@aklaver.com> <CAB+=1TXn-TZFC5QAFDshpRpnNhGrZ847sdaivNJRgot-6BuerA@mail.gmail.com>
 <CAB+=1TWhGgybtCdU7oR3oghhXG6vULUEkebG1DbT+bc1v7yp4w@mail.gmail.com>
 <586d988d-978b-4b08-84b4-ef3ebb8736a8@aklaver.com> <CAB+=1TXcLJsZoGZ2qzv5ehauOa3GX_-cHth4NvZ2mgFsJPCffA@mail.gmail.com>
 <CAKAnmmJhsNL=f+z7R018L6NrLQuq_7+quqZkGUKbzHRctHaY3w@mail.gmail.com>
 <bc8363b5-28d0-4218-b3d6-73c7d78717da@aklaver.com> <CAB+=1TXEJTMtypsZrNhGVEZxRV6jBop4hmMaR5DDsTLOoSqsdQ@mail.gmail.com>
 <4cfe720e-4283-43fc-8d72-8fa11e2f741a@aklaver.com> <20240920221718.ctq5whqfyqlrjwr2@hjp.at>
In-Reply-To: <20240920221718.ctq5whqfyqlrjwr2@hjp.at>
From: veem v <veema0000@gmail.com>
Date: Sat, 21 Sep 2024 15:06:45 +0530
Message-ID: <CAB+=1TU7FcET2RDV0W969B3q0UB3xwiEVdPmqLybBmfqU2VC8w@mail.gmail.com>
Subject: Re: IO related waits
To: pgsql-general@lists.postgresql.org
Content-Type: multipart/alternative; boundary="000000000000a2a62406229de53a"
Archived-At: <https://www.postgresql.org/message-id/CAB%2B%3D1TU7FcET2RDV0W969B3q0UB3xwiEVdPmqLybBmfqU2VC8w%40mail.gmail.com>
Precedence: bulk

--000000000000a2a62406229de53a
Content-Type: text/plain; charset="UTF-8"

On Sat, 21 Sept 2024 at 03:47, Peter J. Holzer <hjp-pgsql@hjp.at> wrote:

> On 2024-09-20 14:11:38 -0700, Adrian Klaver wrote:
> > On 9/20/24 1:01 PM, veem v wrote:
> > > Able to reproduce this deadlock graph as below.  Now my question is ,
> > > this is a legitimate scenario in which the same ID can get inserted
> from
> > > multiple sessions and in such cases it's expected to skip that (thus
> "On
> > > conflict Do nothing" is used) row. But as we see it's breaking the code
> >
> > Yeah, as I see it that would not work with concurrent uncommitted
> sessions
> > as it would be unresolved whether a conflict actually exists until at
> least
> > one of the sessions completes.
> >
> > > with deadlock error during race conditions where a lot of parallel
> > > threads are operating. So how should we handle this scenario? Will
> > > setting the "lock_timeout" parameter at session level will help us
> > > anyway here?
> >
> > Serializable transaction?:
> >
> >
> https://www.postgresql.org/docs/current/transaction-iso.html#XACT-SERIALIZABLE
>
> Doesn't help here, at least not directly. It would help indirectly
> because isolation level serializable makes it very proable that
> serialization errors occur. So an application designed for serializable
> would have some kind of retry logic already in place.
>
> SO that leads as to another solution:
>
> Retry each batch (possibly after reducing the batch size) until it
> succeeds.
>
>
Actually here the application is using kafka and  flink stream and is using
one of existing code in which it was doing row by row commit which is now
changed to Batch commit i.e. the commit point is shifted from row by row to
batch now. There are multiple sessions spawning at the same time to process
the incoming messages 24/7. And also as described in another ongoing thread
and also we saw in the past we did not see much difference between "batch
commit" and "Batch insert" performance. We only realized the row by row
commit is having worse performance.

Now, in such a scenario when the messages are coming from a streaming
framework using kafka +flink and also the insert is happening using row by
row only (but just the commit is shifted to batch), i don't see any way to
sort the ID columns in this streaming process, so that they won't overlap
across session.

In such a situation , the only way seems to have the messages replayed for
which the deadlock error happens , as I think during a deadlock error, one
session gets terminated by postgres and that messages perhap we can save in
some exception table and then replay?

--000000000000a2a62406229de53a
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">=
<div dir=3D"ltr" class=3D"gmail_attr">On Sat, 21 Sept 2024 at 03:47, Peter =
J. Holzer &lt;<a href=3D"mailto:hjp-pgsql@hjp.at">hjp-pgsql@hjp.at</a>&gt; =
wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0=
px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 2024-0=
9-20 14:11:38 -0700, Adrian Klaver wrote:<br>
&gt; On 9/20/24 1:01 PM, veem v wrote:<br>
&gt; &gt; Able to reproduce this deadlock graph as below.=C2=A0 Now my ques=
tion is ,<br>
&gt; &gt; this is a legitimate=C2=A0scenario in which the same ID can get i=
nserted from<br>
&gt; &gt; multiple sessions and in such cases it&#39;s expected to skip tha=
t (thus &quot;On<br>
&gt; &gt; conflict Do=C2=A0nothing&quot; is used) row. But as we see it&#39=
;s breaking the code<br>
&gt; <br>
&gt; Yeah, as I see it that would not work with concurrent uncommitted sess=
ions<br>
&gt; as it would be unresolved whether a conflict actually exists until at =
least<br>
&gt; one of the sessions completes.<br>
&gt; <br>
&gt; &gt; with deadlock error during race=C2=A0conditions where a lot of pa=
rallel<br>
&gt; &gt; threads are operating. So how should we handle this scenario? Wil=
l<br>
&gt; &gt; setting the &quot;lock_timeout&quot; parameter at session level w=
ill help us<br>
&gt; &gt; anyway here?<br>
&gt; <br>
&gt; Serializable transaction?:<br>
&gt; <br>
&gt; <a href=3D"https://www.postgresql.org/docs/current/transaction-iso.htm=
l#XACT-SERIALIZABLE" rel=3D"noreferrer" target=3D"_blank">https://www.postg=
resql.org/docs/current/transaction-iso.html#XACT-SERIALIZABLE</a><br>
<br>
Doesn&#39;t help here, at least not directly. It would help indirectly<br>
because isolation level serializable makes it very proable that<br>
serialization errors occur. So an application designed for serializable<br>
would have some kind of retry logic already in place. <br>
<br>
SO that leads as to another solution:<br>
<br>
Retry each batch (possibly after reducing the batch size) until it<br>
succeeds.<br><br></blockquote><div><br></div><div>Actually here the applica=
tion is using kafka and=C2=A0 flink stream and is using one of existing cod=
e in=C2=A0which it was doing row by row=C2=A0commit which is now changed to=
 Batch commit i.e. the commit point is shifted from row by row to batch now=
. There are multiple sessions spawning=C2=A0at the same time to process the=
 incoming messages 24/7. And also as described in another ongoing thread an=
d also we saw in the past we did not see much difference between &quot;batc=
h commit&quot; and &quot;Batch insert&quot; performance. We only realized t=
he row by row commit is having worse=C2=A0performance.</div><div><br></div>=
<div>Now, in such a scenario when the messages=C2=A0are coming from a strea=
ming framework using kafka=C2=A0+flink and also the insert is happening usi=
ng row by row only (but just the commit is shifted to batch), i don&#39;t s=
ee any way to sort the ID columns in this streaming process, so that they w=
on&#39;t=C2=A0overlap across session.</div><div><br></div><div>In such a si=
tuation , the only way seems to have the messages replayed for which the de=
adlock error=C2=A0happens ,=C2=A0as I think during a deadlock error, one se=
ssion gets terminated by postgres and that messages perhap we can save in s=
ome exception table and then replay?=C2=A0</div><div><br></div></div></div>

--000000000000a2a62406229de53a--