From: Eden Aharoni <edena@legitsecurity.com>
To: Christophe Pettus <xof@thebuild.com>
CC: "pgsql-general@lists.postgresql.org" <pgsql-general@lists.postgresql.org>
Subject: Re: [EXTERNAL] RDS IO Read time
Thread-Topic: [EXTERNAL] RDS IO Read time
Thread-Index: AQHbomR5I1Yf228l+k24YWTefbhcZ7ONhuDJ
Date: Mon, 31 Mar 2025 17:54:08 +0000
Message-ID:
 <PA6PR03MB102417C1CF1AB598D6AF4ABFBBBAD2@PA6PR03MB10241.eurprd03.prod.outlook.com>
References:
 <DBBPR03MB102227BDDC40A11527BF6C0C0BBAD2@DBBPR03MB10222.eurprd03.prod.outlook.com>
 <DA3CFE03-C653-41D8-8F85-604A948C71F1@thebuild.com>
 <PA6PR03MB102418866F5C140FB0604E1EBBBAD2@PA6PR03MB10241.eurprd03.prod.outlook.com>
 <D9C6ADBF-2314-4D3F-B054-C285AB9C9C9D@thebuild.com>
In-Reply-To: <D9C6ADBF-2314-4D3F-B054-C285AB9C9C9D@thebuild.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0:
 =?us-ascii?Q?vXDMuWspQ7kjc2OCX+o4mKEpn193Wc2U6/ZwQf2vXLerMpS/YBKfCViqD1+v?=
 =?us-ascii?Q?g7eCywhC+0QIOoyve2zLwoJt5+C94pLYj0Mrc3+n858pzk5RPCiSzVPs7PuN?=
 =?us-ascii?Q?Cmcs3GPL1DEucqvjzakId3JB6WARp6P2EqjD1vbaCO5S00V3qdf0uiWSvd8h?=
 =?us-ascii?Q?Hji9fZ9ka2CEH049jmTwcAi09pAPwaw3BrRvYQ/Rwd7WTfxziLCBAf9eW/3q?=
 =?us-ascii?Q?6rmtlWHcvqQwCIUpARSxtiYkkr+jCh/fol89Qytera2r3dwBS+HChgeWLcyh?=
 =?us-ascii?Q?+AHevdLLMbsQ5Fee0E6+N8is/9wGwBdrpBunmk9VQmhy4Ojg7VVFR6h6LvSh?=
 =?us-ascii?Q?MfLsJzxXaE9nwAdTKyZ72fRbXxAXAx4PLfeFbaQWTXx03PMlrjwTQ2VUDJ8U?=
 =?us-ascii?Q?0gkjeV1YwIfHcouha8151hJ6kpKtAYS8a+r85FTLBL540Yo1p2oHxjFY6v2s?=
 =?us-ascii?Q?WxbJlqdsA/azfEio5ylOlADXY/d5yHAA/XZNThx6gIVTbrbMbVL/ISptwTV5?=
 =?us-ascii?Q?NdaVKSkXixl7zbFQT/R6DViwBI7qEB0tBcr6+BxZTeObHZU0a3/JH2WaWSlG?=
 =?us-ascii?Q?Zg1Q79aAecljvRlQ52qydPHuOoGfVg+C/i5AesnLRlPE8OGLmUKiMsyF2l9v?=
 =?us-ascii?Q?4OkMeVC66AvKHuuOvuK9IoEgjCLwr5TkX4s5WRKHZbos4H5uHUoHRzNnahkM?=
 =?us-ascii?Q?8tkoby3hYj8N3sTSCLJtOjWpz8/RRXQjm13L8AYjS6qZhmnfF8aORLmou1d7?=
 =?us-ascii?Q?9fES8v7CjTbQtyFU4rGiuplMhtlf1HhrKIGuE4KZuN+bGk/iq0f8GCCVw7eV?=
 =?us-ascii?Q?1TuFwQbyEbJ+ozeMVfprvoPXtIXOyAb4d03ag8GUFto7NHlwye6r3bM0JjLg?=
 =?us-ascii?Q?CxIlx1nSyx/t6MmvqH3TT+i8KqWlkVWB9svHbY1eyKPY66RQBk70ob1HkiF7?=
 =?us-ascii?Q?uXDObCFODEBNOckmqmJCh5rPJRoyM3eLIFyFebw0czQZsr+Am3/rqU2nFZJ5?=
 =?us-ascii?Q?/JPZRpLhgBUML0ITiLMwwTEZSmeZOwbQqMtXotQjGnkVS6dEmVorEmnZQT/3?=
 =?us-ascii?Q?zP2VZzqR6Xs1/PKoD6VtOtYrlOhdIcrQLw8Fb7fQihT4VnCCmvnCLtp3mDCR?=
 =?us-ascii?Q?cet9H2jBu5UwIHO/m/zDzz5dBZ79e+GDo7pcby1R++uuAnqECJK5Xfv9C5Fm?=
 =?us-ascii?Q?SGm7LPyY5AFp5aM9uG5OezxHPHsaB9h5T41YiLvszh4lkrYShF2JKUaVlsiL?=
 =?us-ascii?Q?01gMyMrS2lURHhsgjWj+q0B2c2ctR3DPkA9GT5Elz07mUM8f9d+ZCm1IR7GF?=
 =?us-ascii?Q?Q4q6YLJ+UPpZipYYogEKNSPE6AOET16aRya+W8TrLy+tYDSsxpLTEfMR+lLi?=
 =?us-ascii?Q?EKZ00lj5jjpl3KHD89fha5XYyFOKO16baaQpZx2Fg1dLNRLXk3x2ft9OcEd6?=
 =?us-ascii?Q?L25cBCt96jHoLRtFXSrVpHTgoehEz+gO5/E5mZJQJz0AkqDzYkTn6DbJnQ6Q?=
 =?us-ascii?Q?6A1jQGIwcfEZKhsObdW7LRdnxztWFZtUcQq/rcFLNhCFXEBisvgxdr1qSjbi?=
 =?us-ascii?Q?hfVKKx9TuR8itD1NwqUW7tePz54j9veev8WdpjO/Hri2NR1mtOVb6VV8KOFy?=
 =?us-ascii?Q?2g=3D=3D?=
Content-Type: multipart/alternative;
	boundary="_000_PA6PR03MB102417C1CF1AB598D6AF4ABFBBBAD2PA6PR03MB10241eu_"
MIME-Version: 1.0
X-OriginatorOrg: legitsecurity.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PA6PR03MB10241.eurprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 44e15e9c-727c-4590-132b-08dd707d090a
X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Mar 2025 17:54:08.1013
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 1d036cb2-b1cc-4321-8a87-6f43158d9e1a
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 6aSF2fOQfGs5ULLD6iuK5KW+P1V01V52wKWOLSA7ydzW8qAUsLnT95xvyMI+pzSvQnp0JXQXzm4bPnkHBNf4Ug==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR03MB6259
List-Id: <pgsql-general.lists.postgresql.org>
List-Help: <https://lists.postgresql.org/manage/>
List-Subscribe: <https://lists.postgresql.org/manage/>
List-Post: <mailto:pgsql-general@lists.postgresql.org>
List-Owner: <mailto:pgsql-general-owner@lists.postgresql.org>
List-Archive: <https://www.postgresql.org/list/pgsql-general>
Archived-At: <https://www.postgresql.org/message-id/PA6PR03MB102417C1CF1AB598D6AF4ABFBBBAD2%40PA6PR03MB10241.eurprd03.prod.outlook.com>
Precedence: bulk

--_000_PA6PR03MB102417C1CF1AB598D6AF4ABFBBBAD2PA6PR03MB10241eu_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

So you believe it's strictly an EBS issue?

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Christophe Pettus <xof@thebuild.com>
Sent: Monday, March 31, 2025 8:43:19 PM
To: Eden Aharoni <edena@legitsecurity.com>
Cc: pgsql-general@lists.postgresql.org <pgsql-general@lists.postgresql.org>
Subject: Re: [EXTERNAL] RDS IO Read time

[You don't often get email from xof@thebuild.com. Learn why this is importa=
nt at https://aka.ms/LearnAboutSenderIdentification ]

> On Mar 31, 2025, at 10:32, Eden Aharoni <edena@legitsecurity.com> wrote:
>
> First, thanks for the reply :)
> So, I do know which part is taking a lot of IO time and it's to be honest=
 any node that reads from the disk.. of course, we're running EXPLAIN on ou=
r queries (to be more specific we use auto_explain) but we can't seem to fi=
nd what could cause an index scan that reads 34 MB to take more than a seco=
nd (1.2 sec).. we do know that our dataset doesn't fit the memory and we we=
re ok with that but with IO being so slow we just don't know what to do any=
more. Any other suggestions other than contact AWS (which we did but it see=
ms this path won't lead anywhere).

The performance of EBS is definitely a question for AWS.  I can say that, i=
n our experience, EBS mounts for RDS almost never approach the stated maxim=
um throughput, although io2 tends to be closer than gp2 or gp3.

--_000_PA6PR03MB102417C1CF1AB598D6AF4ABFBBBAD2PA6PR03MB10241eu_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
</head>
<body>
<div dir=3D"auto">So you believe it's strictly an EBS issue?</div>
<div id=3D"ms-outlook-mobile-body-separator-line" dir=3D"auto"><br>
</div>
<div id=3D"ms-outlook-mobile-signature" dir=3D"auto">Get <a href=3D"https:/=
/aka.ms/AAb9ysg">
Outlook for Android</a></div>
<hr style=3D"display:inline-block;width:98%" tabindex=3D"-1">
<div id=3D"divRplyFwdMsg" dir=3D"ltr"><font face=3D"Calibri, sans-serif" st=
yle=3D"font-size:11pt" color=3D"#000000"><b>From:</b> Christophe Pettus &lt=
;xof@thebuild.com&gt;<br>
<b>Sent:</b> Monday, March 31, 2025 8:43:19 PM<br>
<b>To:</b> Eden Aharoni &lt;edena@legitsecurity.com&gt;<br>
<b>Cc:</b> pgsql-general@lists.postgresql.org &lt;pgsql-general@lists.postg=
resql.org&gt;<br>
<b>Subject:</b> Re: [EXTERNAL] RDS IO Read time</font>
<div>&nbsp;</div>
</div>
<div class=3D"BodyFragment"><font size=3D"2"><span style=3D"font-size:11pt;=
">
<div class=3D"PlainText">[You don't often get email from xof@thebuild.com. =
Learn why this is important at
<a href=3D"https://aka.ms/LearnAboutSenderIdentification">https://aka.ms/Le=
arnAboutSenderIdentification</a> ]<br>
<br>
&gt; On Mar 31, 2025, at 10:32, Eden Aharoni &lt;edena@legitsecurity.com&gt=
; wrote:<br>
&gt;<br>
&gt; First, thanks for the reply :)<br>
&gt; So, I do know which part is taking a lot of IO time and it's to be hon=
est any node that reads from the disk.. of course, we're running EXPLAIN on=
 our queries (to be more specific we use auto_explain) but we can't seem to=
 find what could cause an index scan
 that reads 34 MB to take more than a second (1.2 sec).. we do know that ou=
r dataset doesn't fit the memory and we were ok with that but with IO being=
 so slow we just don't know what to do anymore. Any other suggestions other=
 than contact AWS (which we did
 but it seems this path won't lead anywhere).<br>
<br>
The performance of EBS is definitely a question for AWS.&nbsp; I can say th=
at, in our experience, EBS mounts for RDS almost never approach the stated =
maximum throughput, although io2 tends to be closer than gp2 or gp3.<br>
</div>
</span></font></div>
</body>
</html>

--_000_PA6PR03MB102417C1CF1AB598D6AF4ABFBBBAD2PA6PR03MB10241eu_--