Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cgK8E-0002C4-0v for pgsql-performance@arkaria.postgresql.org; Tue, 21 Feb 2017 23:46:06 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.84_2) (envelope-from ) id 1cgK8D-0004nz-Fk for pgsql-performance@arkaria.postgresql.org; Tue, 21 Feb 2017 23:46:05 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1cgK6W-0002CS-RN for pgsql-performance@postgresql.org; Tue, 21 Feb 2017 23:44:21 +0000 Received: from mail-wm0-x233.google.com ([2a00:1450:400c:c09::233]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.84_2) (envelope-from ) id 1cgK6T-00062T-J6 for pgsql-performance@postgresql.org; Tue, 21 Feb 2017 23:44:19 +0000 Received: by mail-wm0-x233.google.com with SMTP id v77so88003635wmv.0 for ; Tue, 21 Feb 2017 15:44:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc:message-id:references :to; bh=WHxen9E8TeZZy3E+gRGqeFpDqM1MkYATmSJzsqWun9U=; b=o0WZNuR+SRZM76Ql2X7rWq6Nt8cWxa+DOHIinzdQ33/vdQkdGXQ+Oj3qXHMrPjrjrf 7J50MadJH8lQMcjy7zaZAqcxUm4cZeW25d1chLU6h2hYsgx+a1PhYvy9gZN38GAnAB/J i3TJlcDCX5WYDJo50/PSMXSeFDNZ2gQvxYDAq8LDnJmwuOVyf99vbzk+PogTVLFaZhQ2 i12imwz3hCarK0qDCAIoPYPB2OaGOYTGYQb4Jg7LmcvDa3DNvd1U5SSRkVQc7PBBjLCA 2Cz4vR51HRdP3n2OS0oxMf/A1eWDuiZ8qAeZeGSrRdLDvAKgdofH+GV2wtUXxnHnX2iO nK1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=WHxen9E8TeZZy3E+gRGqeFpDqM1MkYATmSJzsqWun9U=; b=pFHGBvs4W/dS/DxBq/Fv4gcLV/a/Lll83LtkzkRTqvb2DGwyRF31iMWiCn+Px4n/8r EVmxdhQ//XDtvbL89CDL/T5o262vKhBBVT1PJsYJhiMrbOlX0hwgO+prBccBs3GpM7dJ bQrkepCPfWoc3ZudAP53nanRZD0E552UjWVc1twV5nXGHbkFcUwJChCBZIlOGkZlLaqE JcuWXneq/N/z/T/ddGbKYsqP4NJP8NuDXh7TmocNRWT5Uw9ikjNIlXpAGoseMSIVVXaR NBbSJJLvIIviyAZfIrGKr4GxysTORsFZpfR6yXGg2XZcj7GnEPfxs/QeHG+2qxYlvuIR AccA== X-Gm-Message-State: AMke39mCSykzbpZ5eBPhy0hY/wKfBcG5O4bwfYi39SpoZStAuXWMhrX/s+nPJl8n1whALQ== X-Received: by 10.28.1.213 with SMTP id 204mr26009188wmb.70.1487720655951; Tue, 21 Feb 2017 15:44:15 -0800 (PST) Received: from [192.168.2.5] (2-230-140-245.ip202.fastwebnet.it. [2.230.140.245]) by smtp.gmail.com with ESMTPSA id 186sm232316wmw.24.2017.02.21.15.44.15 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 21 Feb 2017 15:44:15 -0800 (PST) Content-Type: multipart/alternative; boundary="Apple-Mail=_F74DEA63-B849-4C35-A119-3D78D7C071F8" Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Suggestions for a HBA controller (6 x SSDs + madam RAID10) From: Pietro Pugni In-Reply-To: <85c31c1250404e10985fe0baf943ef38@bowex17a.micron.com> Date: Wed, 22 Feb 2017 00:44:14 +0100 Cc: Merlin Moncure , "pgsql-performance@postgresql.org" Message-Id: <800BBCD8-FF7C-421D-8EB1-AF022E16C259@gmail.com> References: <85c31c1250404e10985fe0baf943ef38@bowex17a.micron.com> To: "Wes Vaske (wvaske)" X-Mailer: Apple Mail (2.3124) X-Pg-Spam-Score: -2.7 (--) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-performance Precedence: bulk Sender: pgsql-performance-owner@postgresql.org --Apple-Mail=_F74DEA63-B849-4C35-A119-3D78D7C071F8 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > Disclaimer: I=E2=80=99ve done extensive testing (FIO and postgres) = with a few different RAID controllers and HW RAID vs mdadm. We (micron) = are crucial but I don=E2=80=99t personally work with the consumer = drives. > =20 > Verify whether you have your disk write cache enabled or disabled. If = it=E2=80=99s disabled, that will have a large impact on write = performance.=20 What an honor :) My SSDs are Crucial MX300 (consumer drives) but, as previously stated, = they gave ~90k IOPS in all benchmarks I found on the web, while mine = tops at ~40k IOPS. Being 6 devices bought from 4 different sellers = it=E2=80=99s impossible that they are all defective. > Is this the *exact* string you used? `fio --filename=3D/dev/sdx = --direct=3D1 --rw=3Drandrw --refill_buffers --norandommap --randrepeat=3D0= --ioengine=3Dlibaio --bs=3D4k --rwmixread=3D100 --iodepth=3D16 = --numjobs=3D16 --runtime=3D60 --group_reporting --name=3D4ktest` > =20 > With FIO, you need to multiply iodepth by numjobs to get the final = queue depth its pushing. (in this case, 256). Make sure you=E2=80=99re = looking at the correct data. I used =E2=80=94numjobs=3D1 because I needed the time series values for = bandwidth, latencies and iops. The command string was the same, except = from varying IO Depth and numjobs=3D1. > Few other things: > - Mdadm will give better performance than HW RAID for = specific benchmarks. > - Performance is NOT linear with drive count for synthetic = benchmarks. > - It is often nearly linear for application performance. mdadm RAID10 scaled linearly while mdadm RAID0 scaled much less. > - HW RAID can give better performance if your drives do not = have a capacitor backed cache (like the MX300) AND the controller has a = battery backed cache. *Consumer drives can often get better performance = from HW RAID*. (otherwise MDADM has been faster in all of my testing) My RAID controller doesn=E2=80=99t have a BBU. > - Mdadm RAID10 has a bug where reads are not properly = distributed between the mirror pairs. (It uses head position calculated = from the last IO to determine which drive in a mirror pair should get = the next read. It results in really weird behavior of most read IO going = to half of your drives instead of being evenly split as should be the = case for SSDs). You can see this by running iostat while you=E2=80=99ve = got a load running and you=E2=80=99ll see uneven distribution of IOs. = FYI, the RAID1 implementation has an exception where it does NOT use = head position for SSDs. I have yet to test this but you should be able = to get better performance by manually striping a RAID0 across multiple = RAID1s instead of using the default RAID10 implementation. Very interesting. I will double check this after buying and mounting the = new HBA. I heard of someone doing what you are suggesting but never = tried. > - Don=E2=80=99t focus on 4k Random Read. Do something more = similar to a PG workload (64k 70/30 R/W @ QD=3D4 is *reasonably* close = to what I see for heavy OLTP). Why 64k and QD=3D4? I thought of 8k and larger QD. Will test as soon as = possible and report here the results :) > I=E2=80=99ve tested multiple controllers based on the LSI 3108 and = found that default settings from one vendor to another provide = drastically different performance profiles. Vendor A had much better = benchmark performance (2x IOPS of B) while vendor B gave better = application performance (20% better OLTP performance in Postgres). (I = got equivalent performance from A & B when using the same settings).=20 Do you have some HBA card to suggest? What do you think of LSI SAS3008? = I think it=E2=80=99s the same as the 3108 without RAID On Chip feature. = Probably I will buy a Lenovo HBA card with that chip. It seems blazing = fast (1mln IOPS) compared to the actual embedded RAID controller (LSI = 2008). I don=E2=80=99t know if I can connect a 12Gb/s HBA directly to my = existing 6Gb/s expander/backplane.. sure I will have the right cables = but don=E2=80=99t know if it will work without changing the = expander/backplane. Thank you a lot for your time Pietro Pugni --Apple-Mail=_F74DEA63-B849-4C35-A119-3D78D7C071F8 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
Disclaimer: I=E2=80=99ve done extensive testing (FIO and = postgres) with a few different RAID controllers and HW RAID vs mdadm. We = (micron) are crucial but I don=E2=80=99t personally work with the = consumer drives.
 
Verify whether you have your disk write cache = enabled or disabled. If it=E2=80=99s disabled, that will have a large = impact on write performance. 

What an honor :)
My SSDs are Crucial = MX300 (consumer drives) but, as previously stated, they gave ~90k IOPS = in all benchmarks I found on the web, while mine tops at ~40k IOPS. = Being 6 devices bought from 4 different sellers it=E2=80=99s impossible = that they are all defective.

Is this the *exact* string you = used? `fio --filename=3D/dev/sdx --direct=3D1 --rw=3Drandrw = --refill_buffers --norandommap --randrepeat=3D0 --ioengine=3Dlibaio = --bs=3D4k --rwmixread=3D100 --iodepth=3D16 --numjobs=3D16 --runtime=3D60 = --group_reporting --name=3D4ktest`
 
With FIO, you need to multiply iodepth by = numjobs to get the final queue depth its pushing. (in this case, 256). = Make sure you=E2=80=99re looking at the correct = data.

I = used =E2=80=94numjobs=3D1 because I needed the time series values for = bandwidth, latencies and iops. The command string was the same, except = from varying IO Depth and numjobs=3D1.


Few other things:
-          Mdadm will give better performance than HW RAID for specific = benchmarks.
-          Performance is NOT linear with drive count for synthetic = benchmarks.
-          It= is often nearly linear for application = performance.

mdadm RAID10 scaled linearly while mdadm RAID0 = scaled much less.


-          HW= RAID can give better performance if your drives do not have a capacitor = backed cache (like the MX300) AND the controller has a battery backed = cache. *Consumer drives can often get better performance = from HW RAID*. (otherwise MDADM has been faster in all of my = testing)

My RAID controller doesn=E2=80=99t have a = BBU.


-          Mdadm RAID10 has a bug where reads are not properly = distributed between the mirror pairs. (It uses head position calculated = from the last IO to determine which drive in a mirror pair should get = the next read. It results in really weird behavior of most read IO going = to half of your drives instead of being evenly split as should be the = case for SSDs).  You can see this by running iostat while you=E2=80=99= ve got a load running and you=E2=80=99ll see uneven distribution of IOs. = FYI, the RAID1 implementation has an exception where it does NOT use = head position for SSDs. I have yet to test this but you should be able = to get better performance by manually striping a RAID0 across multiple = RAID1s instead of using the default RAID10 = implementation.

Very interesting. I will double check this after = buying and mounting the new HBA. I heard of someone doing what you are = suggesting but never tried.


-          Don=E2=80=99t focus on 4k Random Read. Do something more = similar to a PG workload (64k 70/30 R/W @ QD=3D4 is *reasonably* close to what I see for = heavy OLTP).

Why 64k and QD=3D4? I thought of 8k and larger QD. = Will test as soon as possible and report here the results = :)


I=E2=80=99ve tested = multiple controllers based on the LSI 3108 and found that default = settings from one vendor to another provide drastically different = performance profiles. Vendor A had much better benchmark performance (2x = IOPS of B) while vendor B gave better application performance (20% = better OLTP performance in Postgres). (I got equivalent performance from = A & B when using the same settings). 

Do you have some HBA card to suggest? What do you = think of LSI SAS3008? I think it=E2=80=99s the same as the 3108 without = RAID On Chip feature. Probably I will buy a Lenovo HBA card with that = chip. It seems blazing fast (1mln IOPS) compared to the actual embedded = RAID controller (LSI 2008).
I don=E2=80=99t know if I can = connect a 12Gb/s HBA directly to my existing 6Gb/s expander/backplane.. = sure I will have the right cables but don=E2=80=99t know if it will work = without changing the expander/backplane.


Thank you a lot for your = time
 Pietro Pugni



= --Apple-Mail=_F74DEA63-B849-4C35-A119-3D78D7C071F8--