Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cjYdu-0004rm-4r for pgsql-performance@arkaria.postgresql.org; Thu, 02 Mar 2017 21:52:10 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.84_2) (envelope-from ) id 1cjYdt-0003iK-Nj for pgsql-performance@arkaria.postgresql.org; Thu, 02 Mar 2017 21:52:09 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84_2) (envelope-from ) id 1cjYds-0003dz-7Z for pgsql-performance@postgresql.org; Thu, 02 Mar 2017 21:52:08 +0000 Received: from mail-wm0-x22d.google.com ([2a00:1450:400c:c09::22d]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.84_2) (envelope-from ) id 1cjYdo-0007pj-3y for pgsql-performance@postgresql.org; Thu, 02 Mar 2017 21:52:07 +0000 Received: by mail-wm0-x22d.google.com with SMTP id n11so2197135wma.0 for ; Thu, 02 Mar 2017 13:52:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc:message-id:references :to; bh=akNGwlfx2YrWrjT189TWX74MsR4IeznPsacoP169epw=; b=XnOTHIGBhssAxVa/eUwCfYhyLwniM1Eaxc/EHqX8Gk047/Esm7YpZ+rRCbFw9RuVh/ DhRYDJCuOTol52nHV+fxF3Udrufo+R0jzhYTUvYlna49lwizCxxltdzWKPdaDmWjCiD9 C74NaiVL0IPyefzqhJjHnd2zp4bNTpCJ0oCCwO0kNmcdUo1LOm311DEiFIZHGJfzNgNN CKfO7qRA/yQ9OGPX2TnJXiakwlG9YAFhsgVh0tClMFdDWl6V92QK8icGQoBr671gTedV nZZtSo2F9cwunhT4k0+De7ooiADp4yKfOxAZxpTcgGEmYcWrh7wdCUWOp1takeQWuOs9 we+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=akNGwlfx2YrWrjT189TWX74MsR4IeznPsacoP169epw=; b=YUKBGl1B4oVpJZ0St22/K61eb48BBMYTwhX8lqMqatUpc2wME+kRIMXBHzalTlU1g/ W+vxPt9Yr0HyPn2YeOG+5ZcUi7FxDIn6BuKQTc+75xZI1N6QBtlQlZPQQa/xYlYqrjAq mz/tif3H15ba1+nmm4nN3y5d+28RyiXucCORBhO5wD6GbC3yq9vLrifjiVBzJ1t80obf cDN7Bj9Jdy7ryGJ1NCMA4qH2TDOR1xALfipt30Rnr0vv9tA7D4c4FiYhoZZLaLyEiJvd 73HaPtYvwGRtmjfIzMn93U8Us5umQPHVE3K/g82rt4txDi1gg19CW7ziiZdxHXnZvmBs atdg== X-Gm-Message-State: AMke39nvZdxte6JcNbHtO2SKDATzN9BzQ41Mw0jEwmutJV7r00Osj1B4TP5+TnfagYPtqg== X-Received: by 10.28.238.66 with SMTP id m63mr305801wmh.121.1488491522722; Thu, 02 Mar 2017 13:52:02 -0800 (PST) Received: from [192.168.2.5] (2-230-140-245.ip202.fastwebnet.it. [2.230.140.245]) by smtp.gmail.com with ESMTPSA id k195sm324655wmd.7.2017.03.02.13.52.01 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 02 Mar 2017 13:52:01 -0800 (PST) Content-Type: multipart/alternative; boundary="Apple-Mail=_9138C8B3-C857-4027-88B6-730C64FE46BF" Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Suggestions for a HBA controller (6 x SSDs + madam RAID10) From: Pietro Pugni In-Reply-To: Date: Thu, 2 Mar 2017 22:51:56 +0100 Cc: "pgsql-performance@postgresql.org" Message-Id: <1EBE383A-C04C-40F7-AC5B-03F75F33CF09@gmail.com> References: <85c31c1250404e10985fe0baf943ef38@bowex17a.micron.com> <800BBCD8-FF7C-421D-8EB1-AF022E16C259@gmail.com> To: "Wes Vaske (wvaske)" X-Mailer: Apple Mail (2.3124) X-Pg-Spam-Score: -2.7 (--) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-performance Precedence: bulk Sender: pgsql-performance-owner@postgresql.org --Apple-Mail=_9138C8B3-C857-4027-88B6-730C64FE46BF Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 I just mounted and configured my brand new LSI 3008-8i. This server had = 1 SAS expander connected to 2 backplanes (8 disks to the first backplane = and no disks connected to the second backplane). After some testing I = found the SAS expander was a bottleneck, so I removed it and connected = the first backplane directly to the controller.=20 The following results are from 4k 100% random reads (32QD) run in = parallel on each single SSD: Raw SSDs [ 4k, 100% random reads, 32 Queue Depth] ServeRaid m5110e (with SAS expander) [numjob=3D1] read : io=3D5111.2MB, bw=3D87227KB/s, iops=3D21806, runt=3D 60002msec read : io=3D4800.6MB, bw=3D81927KB/s, iops=3D20481, runt=3D 60002msec read : io=3D4997.6MB, bw=3D85288KB/s, iops=3D21322, runt=3D 60002msec read : io=3D4796.2MB, bw=3D81853KB/s, iops=3D20463, runt=3D 60001msec read : io=3D5062.6MB, bw=3D86400KB/s, iops=3D21599, runt=3D 60001msec read : io=3D4989.6MB, bw=3D85154KB/s, iops=3D21288, runt=3D 60001msec Total read iops: 126,595 ( ~ 21,160 iops/disk) Raw SSDs [ 4k, 100% random reads, 32 Queue Depth] Lenovo N2215 (LSI 3008-8i flashed with LSI IT firmware, without SAS = expander) [numjob=3D1] read : io=3D15032MB, bw=3D256544KB/s, iops=3D64136, runt=3D 60001msec read : io=3D16679MB, bw=3D284656KB/s, iops=3D71163, runt=3D 60001msec read : io=3D15046MB, bw=3D256779KB/s, iops=3D64194, runt=3D 60001msec read : io=3D16667MB, bw=3D284444KB/s, iops=3D71111, runt=3D 60001msec read : io=3D16692MB, bw=3D284867KB/s, iops=3D71216, runt=3D 60001msec read : io=3D15149MB, bw=3D258534KB/s, iops=3D64633, runt=3D 60002msec Total read iops: 406,453 ( ~ 67,742 iops/disk) 321% performance improvement. I chose 4k 32QD because it should deliver the maximum iops and should = clearly show if the I/O is properly configured. I don=E2=80=99t mind testing the embedded m5110e without the SAS = expander because it will be slower for sure.=20 > You might need to increase the number of jobs here. The primary reason = for this parameter is to improve scaling when you=E2=80=99re single = thread CPU bound. With numjob=3D1 FIO will use only a single thread and = there=E2=80=99s only so much a single CPU core can do. The HBA provided slightly better performance without removing the = expander and even more slightly faster after removing the expander, but = then I tried increasing numjob from 1 to 16 (tried also 12, 18, 20, 24 = and 32 but found 16 to get higher iops) and the benchmarks returned = expected results. I guess how this relates with Postgres.. probably = effective_io_concurrency, as suggested by Merlin Moncure, should be the = counterpart of numjob in fio? > I was a little unclear on the disk cache part. It=E2=80=99s a setting, = generally in the RAID controller / HBA. It=E2=80=99s also a filesystem = level option in Linux (hdparm) and Windows (somewhere in device = manager?). The reason to disable the disk cache is that it=E2=80=99s NOT = protected against power loss protection on the MX300. So by disabling it = you can ensure 100% write consistency at the cost of write performance. = (using fully power protected drives will let you keep disk cache = enabled) I always enabled the write cache during my tests. I tried to disable it = but performance were too poor. Those SSD are consumer ones and don=E2=80=99= t have any capacitor :( > > Why 64k and QD=3D4? I thought of 8k and larger QD. Will test as soon = as possible and report here the results :) > =20 > It=E2=80=99s more representative of what you=E2=80=99ll see at the = application level. (If you=E2=80=99ve got a running system, you can just = use IOstat to see what your average QD is. (iostat -x 10, and it=E2=80=99s= the column: avgqu-sz. Change from 10 seconds to whatever interval works = best for your environment) I tried your suggestion (64k, 70/30 random r/w, 4QD) on RAID0 and RAID10 = (mdadm) with the new controller and the results are quite good if we = think that the underlying SSDs are consumer with original firmware = (overprovisioned at 25%). RAID10 is about 22% slower in both reads and writes compared to RAID0, = at least on a 1 minute run. The totals and averages were calculated from = the whole fio log output using the single jobs iops. These are the results: = ##########################################################################= ## mdadm RAID0 [ 64k, 70% random reads, 30% random writes, 04 Queue Depth] Lenovo N2215 (LSI 3008-8i flashed with LSI IT firmware, without SAS = expander) [numjob=3D16] = ##########################################################################= ## Run status group 0 (all jobs): READ: io=3D75943MB, aggrb=3D1265.7MB/s, minb=3D80445KB/s, = maxb=3D81576KB/s, mint=3D60001msec, maxt=3D60004msec WRITE: io=3D32585MB, aggrb=3D556072KB/s, minb=3D34220KB/s, = maxb=3D35098KB/s, mint=3D60001msec, maxt=3D60004msec Disk stats (read/write): md127: ios=3D1213256/520566, merge=3D0/0, ticks=3D0/0, in_queue=3D0, = util=3D0.00%, aggrios=3D202541/86892, aggrmerge=3D0/0, = aggrticks=3D490418/137398, aggrin_queue=3D628566, aggrutil=3D99.20% sdf: ios=3D202557/86818, merge=3D0/0, ticks=3D450384/131512, = in_queue=3D582528, util=3D98.58% sdb: ios=3D202626/87184, merge=3D0/0, ticks=3D573448/177336, = in_queue=3D751784, util=3D99.20% sdg: ios=3D202391/86810, merge=3D0/0, ticks=3D463644/137084, = in_queue=3D601272, util=3D98.46% sde: ios=3D202462/86551, merge=3D0/0, ticks=3D470028/121424, = in_queue=3D592500, util=3D98.79% sda: ios=3D202287/86697, merge=3D0/0, ticks=3D473312/121192, = in_queue=3D595044, util=3D98.95% sdh: ios=3D202928/87293, merge=3D0/0, ticks=3D511696/135840, = in_queue=3D648272, util=3D99.14% Total read iops: 20,242 ( ~ 3,374 iops/disk) Total write iops: 8,679 ( ~ 1,447 iops/disk) = ##########################################################################= ## mdadm RAID10 [ 64k, 70% random reads, 30% random writes, 04 Queue Depth] Lenovo N2215 (LSI 3008-8i flashed with LSI IT firmware, without SAS = expander) [numjob=3D16] = ##########################################################################= ## Run status group 0 (all jobs): READ: io=3D58624MB, aggrb=3D976.11MB/s, minb=3D62125KB/s, = maxb=3D62814KB/s, mint=3D60001msec, maxt=3D60005msec WRITE: io=3D25190MB, aggrb=3D429874KB/s, minb=3D26446KB/s, = maxb=3D27075KB/s, mint=3D60001msec, maxt=3D60005msec Disk stats (read/write): md127: ios=3D936349/402381, merge=3D0/0, ticks=3D0/0, in_queue=3D0, = util=3D0.00%, aggrios=3D156357/134348, aggrmerge=3D0/0, = aggrticks=3D433286/262226, aggrin_queue=3D696052, aggrutil=3D99.41% sdf: ios=3D150239/134315, merge=3D0/0, ticks=3D298268/168472, = in_queue=3D466852, util=3D95.31% sdb: ios=3D153088/133664, merge=3D0/0, ticks=3D329160/188060, = in_queue=3D517432, util=3D96.81% sdg: ios=3D157361/135065, merge=3D0/0, ticks=3D658208/459168, = in_queue=3D1118588, util=3D99.16% sde: ios=3D161361/134315, merge=3D0/0, ticks=3D476388/278628, = in_queue=3D756056, util=3D97.61% sda: ios=3D160431/133664, merge=3D0/0, ticks=3D548620/329708, = in_queue=3D878708, util=3D99.41% sdh: ios=3D155667/135065, merge=3D0/0, ticks=3D289072/149324, = in_queue=3D438680, util=3D96.71% Total read iops: 15,625 ( ~ 2,604 iops/disk) Total write iops: 6,709 ( ~ 1,118 iops/disk) > > Do you have some HBA card to suggest? What do you think of LSI = SAS3008? I think it=E2=80=99s the same as the 3108 without RAID On Chip = feature. Probably I will buy a Lenovo HBA card with that chip. It seems = blazing fast (1mln IOPS) compared to the actual embedded RAID controller = (LSI 2008). > =20 > I=E2=80=99ve been able to consistently get the same performance out of = any of the LSI based cards. The 3008 and 3108 both work great, = regardless of vendor. Just test or read up on the different = configuration parameters (read ahead, write back vs write through, disk = cache) Do you have any suggestion for fine tuning this controller? I=E2=80=99m = referring to parameters like nr_requests, queue_depth, etc. Also, any way to optimize the various mdadm parameters available at = /sys/block/mdX/ ? I disabled the internal bitmap and write performance = improved. Thank you Pietro Pugni --Apple-Mail=_9138C8B3-C857-4027-88B6-730C64FE46BF Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
I just mounted and configured my brand new LSI 3008-8i. = This server had 1 SAS expander connected to 2 backplanes (8 disks to the = first backplane and no disks connected to the second backplane). After = some testing I found the SAS expander was a bottleneck, so I removed it = and connected the first backplane directly to the = controller. 

The following = results are from 4k 100% random reads (32QD) run in parallel on each = single SSD:

Raw SSDs [ = 4k, 100% random reads, 32 Queue Depth]
ServeRaid m5110e (with SAS expander) = [numjob=3D1]
  read : io=3D5111.2MB, = bw=3D87227KB/s, iops=3D21806, runt=3D = 60002msec
  read : io=3D4800.6MB, bw=3D81927KB/s, iops=3D20481, runt=3D 60002msec
  read : = io=3D4997.6MB, bw=3D85288KB/s, iops=3D21322, runt=3D = 60002msec
  read : io=3D4796.2MB, bw=3D81853KB/s, iops=3D20463, runt=3D 60001msec
  read : = io=3D5062.6MB, bw=3D86400KB/s, iops=3D21599, runt=3D = 60001msec
  read : io=3D4989.6MB, bw=3D85154KB/s, iops=3D21288, runt=3D 60001msec
Total read = iops: 126,595 ( ~ 21,160 iops/disk)


Raw SSDs [ 4k, 100% random reads, 32 = Queue Depth]
Lenovo N2215 (LSI = 3008-8i flashed with LSI IT firmware, without SAS expander) = [numjob=3D1]
  read : = io=3D15032MB, bw=3D256544KB/s, iops=3D64136, runt=3D = 60001msec
  read : io=3D16679MB, = bw=3D284656KB/s, iops=3D71163, runt=3D = 60001msec
  read : io=3D15046MB, = bw=3D256779KB/s, iops=3D64194, runt=3D = 60001msec
  read : io=3D16667MB, = bw=3D284444KB/s, iops=3D71111, runt=3D = 60001msec
  read : io=3D16692MB, = bw=3D284867KB/s, iops=3D71216, runt=3D = 60001msec
  read : io=3D15149MB, = bw=3D258534KB/s, iops=3D64633, runt=3D = 60002msec
Total read iops: 406,453 ( ~ 67,742 = iops/disk)


321% = performance improvement.
I chose 4k 32QD because it = should deliver the maximum iops and should clearly show if the I/O is = properly configured.
I don=E2=80=99t mind testing = the embedded m5110e without the SAS expander because it will be slower = for sure. 


You might need to increase the = number of jobs here. The primary reason for this parameter is to improve = scaling when you=E2=80=99re single thread CPU bound. With numjob=3D1 FIO = will use only a single thread and there=E2=80=99s only so much a single = CPU core can do.

The HBA provided slightly better performance = without removing the expander and even more slightly faster after = removing the expander, but then I tried increasing numjob from 1 to 16 = (tried also 12, 18, 20, 24 and 32 but found 16 to get higher iops) and = the benchmarks returned expected results. I guess how this relates with = Postgres.. probably effective_io_concurrency, as = suggested by Merlin Moncure, should be the counterpart of numjob in = fio?


I was a little unclear on the disk cache part. It=E2=80=99= s a setting, generally in the RAID controller / HBA. It=E2=80=99s also a = filesystem level option in Linux (hdparm) and Windows (somewhere in = device manager?). The reason to disable the disk cache is that it=E2=80=99= s NOT protected against power loss protection on the MX300. So by = disabling it you can ensure 100% write consistency at the cost of write = performance. (using fully power protected drives will let you keep disk = cache enabled)

I = always enabled the write cache during my tests. I tried to disable it = but performance were too poor. Those SSD are consumer ones and don=E2=80=99= t have any capacitor :(


> Why 64k and QD=3D4? I thought of 8k and larger QD. Will test = as soon as possible and report here the results :)
 
It=E2=80=99s more representative of what = you=E2=80=99ll see at the application level. (If you=E2=80=99ve got a = running system, you can just use IOstat to see what your average QD is. = (iostat -x 10, and it=E2=80=99s the column: avgqu-sz. Change from 10 = seconds to whatever interval works best for your environment)

I = tried your suggestion (64k, 70/30 random r/w, 4QD) on RAID0 and RAID10 = (mdadm) with the new controller and the results are quite good if we = think that the underlying SSDs are consumer with original firmware = (overprovisioned at 25%).

RAID10 is = about 22% slower in both reads and writes compared to RAID0, at least on = a 1 minute run. The totals and averages were calculated from the whole = fio log output using the single jobs iops.

These are the results:


####################################################= ########################
mdadm RAID0 [ 64k, 70% random reads, 30% random writes, 04 = Queue Depth]
Lenovo N2215 (LSI = 3008-8i flashed with LSI IT firmware, without SAS expander) = [numjob=3D16]
####################################################= ########################
Run status group 0 (all = jobs):
   READ: io=3D75943MB, aggrb=3D1265.7MB/s, = minb=3D80445KB/s, maxb=3D81576KB/s, mint=3D60001msec, = maxt=3D60004msec
  WRITE: io=3D32585MB, aggrb=3D556072KB/s,= minb=3D34220KB/s, maxb=3D35098KB/s, mint=3D60001msec, = maxt=3D60004msec

Disk stats = (read/write):
    md127: ios=3D1213256/520566, = merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.00%, = aggrios=3D202541/86892, aggrmerge=3D0/0, aggrticks=3D490418/137398, = aggrin_queue=3D628566, aggrutil=3D99.20%
  sdf: = ios=3D202557/86818, merge=3D0/0, ticks=3D450384/131512, in_queue=3D582528,= util=3D98.58%
  sdb: ios=3D202626/87184, merge=3D0/0, = ticks=3D573448/177336, in_queue=3D751784, util=3D99.20%
  = sdg: ios=3D202391/86810, merge=3D0/0, ticks=3D463644/137084, = in_queue=3D601272, util=3D98.46%
  sde: ios=3D202462/86551,= merge=3D0/0, ticks=3D470028/121424, in_queue=3D592500, = util=3D98.79%
  sda: ios=3D202287/86697, merge=3D0/0, = ticks=3D473312/121192, in_queue=3D595044, util=3D98.95%
  = sdh: ios=3D202928/87293, merge=3D0/0, ticks=3D511696/135840, = in_queue=3D648272, util=3D99.14%

Total read iops: 20,242 ( ~ 3,374 = iops/disk)
Total write iops: 8,679 ( ~ 1,447 = iops/disk)



###############################################= #############################
mdadm RAID10 [ 64k, 70% random reads, 30% random writes, 04 = Queue Depth]
Lenovo N2215 (LSI = 3008-8i flashed with LSI IT firmware, without SAS expander) = [numjob=3D16]
#########################################= ###################################
Run status group 0 (all = jobs):
   READ: io=3D58624MB, aggrb=3D976.11MB/s, = minb=3D62125KB/s, maxb=3D62814KB/s, mint=3D60001msec, = maxt=3D60005msec
  WRITE: io=3D25190MB, aggrb=3D429874KB/s,= minb=3D26446KB/s, maxb=3D27075KB/s, mint=3D60001msec, = maxt=3D60005msec

Disk stats = (read/write):
    md127: ios=3D936349/402381, = merge=3D0/0, ticks=3D0/0, in_queue=3D0, util=3D0.00%, = aggrios=3D156357/134348, aggrmerge=3D0/0, aggrticks=3D433286/262226, = aggrin_queue=3D696052, aggrutil=3D99.41%
  sdf: = ios=3D150239/134315, merge=3D0/0, ticks=3D298268/168472, = in_queue=3D466852, util=3D95.31%
  sdb: = ios=3D153088/133664, merge=3D0/0, ticks=3D329160/188060, = in_queue=3D517432, util=3D96.81%
  sdg: = ios=3D157361/135065, merge=3D0/0, ticks=3D658208/459168, = in_queue=3D1118588, util=3D99.16%
  sde: = ios=3D161361/134315, merge=3D0/0, ticks=3D476388/278628, = in_queue=3D756056, util=3D97.61%
  sda: = ios=3D160431/133664, merge=3D0/0, ticks=3D548620/329708, = in_queue=3D878708, util=3D99.41%
  sdh: = ios=3D155667/135065, merge=3D0/0, ticks=3D289072/149324, = in_queue=3D438680, util=3D96.71%

Total read = iops: 15,625 ( ~ 2,604 iops/disk)
Total write iops: 6,709 ( ~ 1,118 = iops/disk)



> Do you have some HBA card to = suggest? What do you think of LSI SAS3008? I think it=E2=80=99s the same = as the 3108 without RAID On Chip feature. Probably I will buy a Lenovo = HBA card with that chip. It seems blazing fast (1mln IOPS) compared to = the actual embedded RAID controller (LSI 2008).
 
I=E2=80=99ve been able to consistently = get the same performance out of any of the LSI based cards. The 3008 and = 3108 both work great, regardless of vendor. Just test or read up on the = different configuration parameters (read ahead, write back vs write = through, disk cache)

Do = you have any suggestion for fine tuning this controller? I=E2=80=99m = referring to parameters like nr_requests, queue_depth, = etc.
Also, any way to optimize the various mdadm parameters = available at /sys/block/mdX/ ? I disabled the internal bitmap and write = performance improved.



Thank = you
 Pietro Pugni



= --Apple-Mail=_9138C8B3-C857-4027-88B6-730C64FE46BF--