From: Pietro Pugni <pietro.pugni@gmail.com>
To: pgsql-performance@postgresql.org <pgsql-performance@postgresql.org>
Subject: Suggestions for a HBA controller (6 x SSDs + madam RAID10)
Date: Tue, 21 Feb 2017 14:49:39 +0100
Message-ID: <DEBA08E2-240E-4332-8318-391097B613D0@gmail.com> (raw)
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-performance>
Hi there,
I configured an IBM X3650 M4 for development and testing purposes. It’s composed by:
- 2 x Intel Xeon E5-2690 @ 2.90Ghz (2 x 8 physical Cores + HT)
- 96GB RAM DDR3 1333MHz (12 x 8GB)
- 2 x 146GB SAS HDDs @ 15k rpm configured in RAID1 (mdadm)
- 6 x 525GB SATA SSDs (over-provisioned at 25%, so 393GB available)
I’ve done a lot of testing focusing on 4k and 8k workloads and found that the IOPS of those SSDs are half the expected. On serverfault.com someone suggested me that probably the bottle neck is the embedded RAID controller, a IBM ServeRaid m5110e, which mounts a LSI 2008 controller.
I’m using the disks in JBOD mode with mdadm software RAID, which is blazing fast. The CPU is also very fast, so I don’t mind having a little overhead due to software RAID.
My typical workload is Postgres run as a DWH with 1 to 2 billions of rows, big indexes, partitions and so on, but also intensive statistical computations.
Here’s my post on serverfault.com ( http://serverfault.com/questions/833642/slow-ssd-performance-ibm-x3650-m4-7915 )
and here’s a graph of those six SSDs evaluated using fio as stand-alone disks (outside of the RAID):
All those IOPS should be doubled if all was working correctly. The curve trend is correct for increasing IO Depths.
Anyway, I would like to buy a HBA controller that leverages those 6 SSDs. Each SSD should deliver about 80k to 90k IOPS, so in RAID10 I should get ~240k IOPS (6 x 80k / 2) and in RAID0 ~480k IOPS (6 x 80k). I’ve seen that mdadm effectively scales performance, but the controller limits the overal IOPS at ~120k (exactly the half of the expected IOPS).
What HBA controller would you suggest me able to handle 500k IOPS?
My server is able to handle 8 more SSDs, for a total of 14 SSDs and 1260k theoretical IOPS. If we imagine adding only 2 more disks, I will achieve 720k theoretical IOPS in RAID0.
What HBA controller would you suggest me able to handle more than 700k IOPS?
Have you got some advices about using mdadm RAID software on SATAIII SSDs and plain HBA?
Thank you everyone
Pietro Pugni
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: pietro.pugni@gmail.com
Subject: Re: Suggestions for a HBA controller (6 x SSDs + madam RAID10)
In-Reply-To: <DEBA08E2-240E-4332-8318-391097B613D0@gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox