Date: Sat, 21 Sep 2024 15:19:31 +0200
From: "Peter J. Holzer" <hjp-pgsql@hjp.at>
To: pgsql-general@lists.postgresql.org
Subject: Re: IO related waits
Message-ID: <20240921131931.bxik5havgrhu5a5l@hjp.at>
Mail-Followup-To: pgsql-general@lists.postgresql.org
References: <CAB+=1TXn-TZFC5QAFDshpRpnNhGrZ847sdaivNJRgot-6BuerA@mail.gmail.com>
 <CAB+=1TWhGgybtCdU7oR3oghhXG6vULUEkebG1DbT+bc1v7yp4w@mail.gmail.com>
 <586d988d-978b-4b08-84b4-ef3ebb8736a8@aklaver.com>
 <CAB+=1TXcLJsZoGZ2qzv5ehauOa3GX_-cHth4NvZ2mgFsJPCffA@mail.gmail.com>
 <CAKAnmmJhsNL=f+z7R018L6NrLQuq_7+quqZkGUKbzHRctHaY3w@mail.gmail.com>
 <bc8363b5-28d0-4218-b3d6-73c7d78717da@aklaver.com>
 <CAB+=1TXEJTMtypsZrNhGVEZxRV6jBop4hmMaR5DDsTLOoSqsdQ@mail.gmail.com>
 <4cfe720e-4283-43fc-8d72-8fa11e2f741a@aklaver.com>
 <20240920221718.ctq5whqfyqlrjwr2@hjp.at>
 <CAB+=1TU7FcET2RDV0W969B3q0UB3xwiEVdPmqLybBmfqU2VC8w@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="ilxdws6fgccajh5z"
Content-Disposition: inline
In-Reply-To: <CAB+=1TU7FcET2RDV0W969B3q0UB3xwiEVdPmqLybBmfqU2VC8w@mail.gmail.com>
Archived-At: <https://www.postgresql.org/message-id/20240921131931.bxik5havgrhu5a5l%40hjp.at>
Precedence: bulk


--ilxdws6fgccajh5z
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2024-09-21 15:06:45 +0530, veem v wrote:
> On Sat, 21 Sept 2024 at 03:47, Peter J. Holzer <hjp-pgsql@hjp.at> wrote:
>=20
>     On 2024-09-20 14:11:38 -0700, Adrian Klaver wrote:
>     > On 9/20/24 1:01 PM, veem v wrote:
>     > > Able to reproduce this deadlock graph as below.=A0 Now my
>     > > question is , this is a legitimate=A0scenario in which the same
>     > > ID can get inserted from multiple sessions and in such cases
>     > > it's expected to skip that (thus "On conflict Do=A0nothing" is
>     > > used) row. But as we see it's breaking the code
>     >
>     > Yeah, as I see it that would not work with concurrent
>     > uncommitted sessions as it would be unresolved whether a
>     > conflict actually exists until at least one of the sessions
>     > completes.
>     >
>     > > with deadlock error during race=A0conditions where a lot of
>     > > parallel threads are operating. So how should we handle this
>     > > scenario? Will setting the "lock_timeout" parameter at session
>     > > level will help us anyway here?
[...]
>     SO that leads as to another solution:
>=20
>     Retry each batch (possibly after reducing the batch size) until it
>     succeeds.
>=20
>=20
>=20
> Actually here the application is using kafka and=A0 flink stream and is
> using one of existing code in=A0which it was doing row by row=A0commit
> which is now changed to Batch commit i.e. the commit point is shifted
> from row by row to batch now. There are multiple sessions spawning=A0at
> the same time to process the incoming messages 24/7. And also as
> described in another ongoing thread and also we saw in the past we did
> not see much difference between "batch commit" and "Batch insert"
> performance. We only realized the row by row commit is having
> worse=A0performance.
>=20
> Now, in such a scenario when the messages=A0are coming from a streaming
> framework using kafka=A0+flink and also the insert is happening using
> row by row only (but just the commit is shifted to batch), i don't see
> any way to sort the ID columns in this streaming process, so that they
> won't=A0overlap across session.

I have never used Kafka or Flink, but I imagine that they can stream to
various targets, not just to PostgreSQL.

So I would write a program which can receive such a stream. This program
would then buffer rows until either a number of rows or some timeout was
exceeded. It could then preprocess those rows (e.g. by sorting them) and
then open a transaction and try to insert them. If the transaction fails
because of a deadlock, serialization error or similar, simply retry the
transaction. If everything went well, go back to receiving the next
batch of rows.

> In such a situation , the only way seems to have the messages replayed
> for which the deadlock error=A0happens ,=A0as I think during a deadlock
> error, one session gets terminated by postgres

No, the *session* doesn't get terminated. The transaction fails. It can
be rolled back and tried again.

> and that messages perhap we can save in some exception table and then
> replay?=A0

Not sure what you mean by "exception table", but if you mean that you
keep all the rows from that transaction around until the commit succeeds
(so that you can try to insert them again if the transaction fails),
then yes, that's what I meant.

        hp

--=20
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | hjp@hjp.at         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"

--ilxdws6fgccajh5z
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEETtJbRjyPwVTYGJ5k8g5IURL+KF0FAmbux90ACgkQ8g5IURL+
KF1moBAAi+DhMbWn4a50cGMiYYkh4HU871ZZ8NEjWHnN1I6QEYehtm9RDE+sJu3C
GC+cfg9sMK6wE8HtQDS5KP3kavM7TGw8iVzOtUrqy9tVTGbDl2fusRVF8YO+ieGI
rEU1vDf0oAWVQrf4A6NLQpQpIg/00XIAO+R4K6QKHLdq7Y3rvfNhHQGMtRBkUD8m
3i201l1VXKjQQO9JZXHJoG7zBNTEBz63Dlo5qKe/C2L0nZ1PwzBxrqV91ZAwkOv/
bJ+CA1T/4haTsTsW3fFCF4xhNB+Dbbxvz+SWg3WjCfjwTBD//Lkgypx9aDMaSxP4
t5blfMbpzfN1Yi8n4tk94k7ylFbZwLi9Y69FDjLxG6TnAnOFSQwf3hst4KSse/T0
eKq5G62ko4/IO77cYzZ/Rzp8ek4ZPHND4fXaMrCVINlHx2D4RSGOvhB7zZGD9pvH
ZwwBMuKOemeheq6X00/JrnobH6mXMzQvtW5LWkoly2fcw42SO6l5uIT6pO2BKX55
EWvU7c8tqEyAAyyu74bt/pLhwN6QLPwk5XfPA2owvmGBAOqxKW2QI8A8ybCWx5Bu
z8yZhNH2y7PqF/wGWUm7U1ys4GuTQR1+cK8Rsm7bcD8zXwe0m+DrVIspZ3RuSmrI
9C5f4CMjTYccFoKcD1IkPFemq0NSpmGnyKCS9dJJKpF6hero4Vw=
=3XHe
-----END PGP SIGNATURE-----

--ilxdws6fgccajh5z--