Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cgAqq-00050x-1H for pgsql-performance@arkaria.postgresql.org; Tue, 21 Feb 2017 13:51:32 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.84_2) (envelope-from ) id 1cgAqp-0005Mu-KO for pgsql-performance@arkaria.postgresql.org; Tue, 21 Feb 2017 13:51:31 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1cgAp8-000224-7W for pgsql-performance@postgresql.org; Tue, 21 Feb 2017 13:49:46 +0000 Received: from mail-wm0-x234.google.com ([2a00:1450:400c:c09::234]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.84_2) (envelope-from ) id 1cgAp5-000347-Dm for pgsql-performance@postgresql.org; Tue, 21 Feb 2017 13:49:45 +0000 Received: by mail-wm0-x234.google.com with SMTP id t18so29141355wmt.0 for ; Tue, 21 Feb 2017 05:49:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:subject:message-id:date:to:mime-version; bh=R2zHwRdOiJlbokg4YA544fv7+GvAJWGnk+A00SZhEwY=; b=MHmir7Br+zeceTQTG0ejG9xW+2pv78ZbK6KS0R5NyAjLUJIUtxJBWrBAq22T8VJoP5 KEZZAtk5AfUmPIQ21cri8xRXV2VgKF5K8nZ3PNm/KfPFYbdd05cgxvzWpnQk+RcRuObA Xa8t2M4IoKc1YyGW+Fa31paLTJYmA/ZJTsY/HSib2QcP1/tcpAQbDlmVGXqp+4YKT1+e FrxWI7ICW5mEpOAlmN8dalDrPFAQEQifZ8f1gMvvO+Zwb7TMzp2EmGAt/4GuaWtd1aJW SQyYmoiaEheE3Vr4I0ivqee8hSTW1xrU702lb+C/ydl4xKfJ63dUQSfMwGrbSFfDspHO Q5sQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:subject:message-id:date:to:mime-version; bh=R2zHwRdOiJlbokg4YA544fv7+GvAJWGnk+A00SZhEwY=; b=ON9mpqbg7IvGT7BbrM35h++mvMbH/IIRFZs0wjAQMvEUUEQmTyTLBd1temPfqlACNQ FQe/EhIKjepeqK1ZUWFGcvuotBRcEOtBHGbvx2PiNl8DP4LRNoirWLX7C6QRT+0EKnTR kz1pp1a0UW3YlRAJh1xqb9g4ytMty/orDVrJrKsvZ53veZt73IuM5Ne4d5rNqJVqZDiC c99kkFiXqTc8py+aPpWtEE+2n5Kn8nNuhM7/FqWlGnGbV3E7vVjMnLIhAvQfJqoEP8QA PKWiwxvEadl/b8xtZC812yiiFUySvJ6x4PACYarkVWiTtZwvaHU7QslDC4N3QdY/wH+s eiNg== X-Gm-Message-State: AMke39kpymFLoB7OyVvf8isHyO7DgjrUXuQf+KdUlTRZjgMJC7iQtt2DB/P85qeTKeOQlg== X-Received: by 10.28.31.130 with SMTP id f124mr13974867wmf.130.1487684981511; Tue, 21 Feb 2017 05:49:41 -0800 (PST) Received: from [192.168.2.5] (2-230-140-245.ip202.fastwebnet.it. [2.230.140.245]) by smtp.gmail.com with ESMTPSA id k187sm14353162wmb.13.2017.02.21.05.49.40 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 21 Feb 2017 05:49:40 -0800 (PST) From: Pietro Pugni Content-Type: multipart/alternative; boundary="Apple-Mail=_4D6FDBD6-38EB-4874-AA98-458A9BC1274F" Subject: Suggestions for a HBA controller (6 x SSDs + madam RAID10) Message-Id: Date: Tue, 21 Feb 2017 14:49:39 +0100 To: "pgsql-performance@postgresql.org" Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) X-Mailer: Apple Mail (2.3124) X-Pg-Spam-Score: -2.7 (--) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-performance Precedence: bulk Sender: pgsql-performance-owner@postgresql.org --Apple-Mail=_4D6FDBD6-38EB-4874-AA98-458A9BC1274F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi there, I configured an IBM X3650 M4 for development and testing purposes. = It=E2=80=99s composed by: - 2 x Intel Xeon E5-2690 @ 2.90Ghz (2 x 8 physical Cores + HT) - 96GB RAM DDR3 1333MHz (12 x 8GB) - 2 x 146GB SAS HDDs @ 15k rpm configured in RAID1 (mdadm) - 6 x 525GB SATA SSDs (over-provisioned at 25%, so 393GB available) I=E2=80=99ve done a lot of testing focusing on 4k and 8k workloads and = found that the IOPS of those SSDs are half the expected. On = serverfault.com someone suggested me that probably the bottle neck is = the embedded RAID controller, a IBM ServeRaid m5110e, which mounts a LSI = 2008 controller. I=E2=80=99m using the disks in JBOD mode with mdadm software RAID, which = is blazing fast. The CPU is also very fast, so I don=E2=80=99t mind = having a little overhead due to software RAID. My typical workload is Postgres run as a DWH with 1 to 2 billions of = rows, big indexes, partitions and so on, but also intensive statistical = computations. Here=E2=80=99s my post on serverfault.com ( = http://serverfault.com/questions/833642/slow-ssd-performance-ibm-x3650-m4-= 7915 ) and here=E2=80=99s a graph of those six SSDs evaluated using fio as = stand-alone disks (outside of the RAID): All those IOPS should be doubled if all was working correctly. The curve = trend is correct for increasing IO Depths. Anyway, I would like to buy a HBA controller that leverages those 6 = SSDs. Each SSD should deliver about 80k to 90k IOPS, so in RAID10 I = should get ~240k IOPS (6 x 80k / 2) and in RAID0 ~480k IOPS (6 x 80k). = I=E2=80=99ve seen that mdadm effectively scales performance, but the = controller limits the overal IOPS at ~120k (exactly the half of the = expected IOPS). What HBA controller would you suggest me able to handle 500k IOPS?=20 My server is able to handle 8 more SSDs, for a total of 14 SSDs and = 1260k theoretical IOPS. If we imagine adding only 2 more disks, I will = achieve 720k theoretical IOPS in RAID0.=20 What HBA controller would you suggest me able to handle more than 700k = IOPS?=20 Have you got some advices about using mdadm RAID software on SATAIII = SSDs and plain HBA? Thank you everyone Pietro Pugni --Apple-Mail=_4D6FDBD6-38EB-4874-AA98-458A9BC1274F Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi there,
I configured an IBM X3650 M4 for = development and testing purposes. It=E2=80=99s composed by:
 - 2 x Intel Xeon E5-2690 @ 2.90Ghz (2 x 8 physical = Cores + HT)
 - 96GB RAM DDR3 1333MHz (12 x = 8GB)
 - 2 x 146GB SAS HDDs @ 15k rpm = configured in RAID1 (mdadm)
 - 6 x 525GB SATA = SSDs (over-provisioned at 25%, so 393GB available)

I=E2=80=99ve done a lot = of testing focusing on 4k and 8k workloads and found that the IOPS of = those SSDs are half the expected. On serverfault.com someone suggested me that probably the = bottle neck is the embedded RAID controller, a IBM ServeRaid m5110e, = which mounts a LSI 2008 controller.

I=E2=80=99m using the disks in JBOD = mode with mdadm software RAID, which is blazing fast. The CPU is also = very fast, so I don=E2=80=99t mind having a little overhead due to = software RAID.

My typical workload is Postgres run as a DWH with 1 to 2 = billions of rows, big indexes, partitions and so on, but also intensive = statistical computations.


Here=E2=80=99s my post = on serverfault.com = ( http://serverfault.com/questions/833642/slow-ssd-performance-ib= m-x3650-m4-7915 )
and here=E2=80=99s a graph of = those six SSDs evaluated using fio as stand-alone disks (outside of the = RAID):

3D"https://i.stack.imgur.com/ZMhUJ.png"

All those IOPS should be doubled if all was working = correctly. The curve trend is correct for increasing IO = Depths.


Anyway, I would like to buy a HBA = controller that leverages those 6 SSDs. Each SSD should deliver about = 80k to 90k IOPS, so in RAID10 I should get ~240k IOPS (6 x 80k / 2) and = in RAID0 ~480k IOPS (6 x 80k). I=E2=80=99ve seen that mdadm effectively = scales performance, but the controller limits the overal IOPS at ~120k = (exactly the half of the expected IOPS).

What HBA controller would = you suggest me able to handle 500k IOPS? 


My server is able to handle 8 more SSDs, for = a total of 14 SSDs and 1260k theoretical IOPS. If we imagine adding only = 2 more disks, I will achieve 720k theoretical IOPS in = RAID0. 

What HBA controller would you = suggest me able to handle more than 700k IOPS? 

Have you got some advices about using mdadm RAID software on = SATAIII SSDs and plain HBA?

Thank you everyone
   Pietro Pugni



= --Apple-Mail=_4D6FDBD6-38EB-4874-AA98-458A9BC1274F--