Date: Mon, 12 Aug 2013 11:01:09 -0500 (CDT)
From: Scott Whitney <scott@journyx.com>
Reply-To: Scott Whitney <swhitney@journyx.com>
To: Craig James <cjames@emolecules.com>
Cc: pgsql-admin@postgresql.org, "David F. Skoll" <dfs@roaringpenguin.com>
Message-ID: <14817358.266767.1376323269109.JavaMail.root@mail.int.journyx.com>
In-Reply-To: 
 <CAFwQ8rd7cg1QpuQnSdtcL0mJYuC9eDJ26wUzjWa4_NFJxD+R=A@mail.gmail.com>
Subject: Re: Opinions on SSDs
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="----=_Part_266766_3626445.1376323269106"
Precedence: bulk
Sender: pgsql-admin-owner@postgresql.org

------=_Part_266766_3626445.1376323269106
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

When you say "16 10K drives," do you mean: 

a) RAID 0 with 16 drives? 
b) RAID 1 with 8+8 drives? 
c) RAID 5 with 12 drives? 
d) RAID 1 with 7+7 drives and 2 hotspares? 

We moved from a 14 FC drive (15k RPM) array (6+6 with 2 hotspares) to a 6 SSD array (2+2 with 2 hotspares) because our iops would max out regularly on the spinning drives. The SSD solution I put in has show significant speed improvements, to say the very least. 

The short answer is unless you're going with option a (which has no redundancy), you're going to be have some I/O wait at 5k tps. 

Now, there is a LOT to understand about drive iops. You could start here, if you would like to read a bit about it: 
http://www.techrepublic.com/blog/the-enterprise-cloud/calculate-iops-in-a-storage-array/ 

Basically, just assume that you're getting 130 iops per drive. Well, 16 drives in a RAID 0 is going to max you out at 2100ish iops, which is low for your stated peak usage (but probably within range for your average usage). However, you start looking at 8+8 or 7+7 with HSP, you're looking at cutting that in half, and you're going to see I/O wait, period. Of course if you were to use a non-recommended RAID5, you'd be taking an even bigger hit on writes. 

Now, I do _not_ want to open a can of worms here and start a war about SSD versus spindles and "perceived performance vs real," or any such thing. I attended Greg Smith's (excellent) talk in Austin at PG Day wrt "Seeking Postgres," and I had also personally amassed quite a bit of data on such comparisons myself. Unfortunately, some of that talk not compare apples to apples (3-disk RAID 0 versus singe Intel 520SSD), and I quite simply find that the benchmarks do not really reflect real world usage. 

Source: months and months of real-world stress-testing specifically 10k drives (SAS) against SSD (SATA) drives in the same configuration on the same machine using the same tests plus over a year (total) of production deployment among 3 servers thusly configured. 

So far as personal experience with the Intel drives, I don't have that, personally. I'm using Crucial, and I'm pretty happy with those. The _problem_ with SSD is there is no "put it in the freezer" magic bullet. When they fail, they fail, and they're gone. So, IMO (and there are MANY MANY valid opinions on this), use slightly cheaper drives and proactively replace them every 9 months or year. 

----- Original Message -----

> On Mon, Aug 12, 2013 at 8:28 AM, David F. Skoll <
> dfs@roaringpenguin.com > wrote:

> > 3) Our current workload peaks at about 5000 transactions per
> > second;
> 
> > you can assume about one-third to one-half of those are writes. Do
> 
> > you think we can get away with 16 10Krpm SATA drives instead of the
> 
> > SSDs?
> 

> pgbench peaks out at 5K-7K transactions per second on my server which
> uses just 10ea. of 7Krpm SATA drives:

> WAL: RAID1 (2 disks)
> Data: RAID10 (8 disks)
> 3Ware RAID controller with BBU
> 2x4 core Intel CPUs
> 12 GB memory

> I don't know how pgbench compares to your workload. But suspect 16
> 10K SATA drives would be pretty fast if you combine them with a BBU
> RAID controller.

> On the other hand, I swore this would be the last server I buy with
> spinning storage.

> Craig

------=_Part_266766_3626445.1376323269106
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><head><style type=3D'text/css'>p { margin: 0; }</style></head><body><=
div style=3D'font-family: times new roman,new york,times,serif; font-size: =
12pt; color: #000000'>When you say "16 10K drives," do you mean:<div><br></=
div><div>a) RAID 0 with 16 drives?</div><div>b) RAID 1 with 8+8 drives?</di=
v><div>c) RAID 5 with 12 drives?</div><div>d) RAID 1 with 7+7 drives and 2 =
hotspares?</div><div><br></div><div>We moved from a 14 FC drive (15k RPM) a=
rray (6+6 with 2 hotspares) to a 6 SSD array (2+2 with 2 hotspares) because=
 our iops would max out regularly on the spinning drives. The SSD solution =
I put in has show significant speed improvements, to say the very least.</d=
iv><div><br></div><div>The short answer is unless you're going with option =
a (which has no redundancy), you're going to be have some I/O wait at 5k tp=
s.</div><div><br></div><div>Now, there is a LOT to understand about drive i=
ops. You could start here, if you would like to read a bit about it:</div><=
div><a href=3D"http://www.techrepublic.com/blog/the-enterprise-cloud/calcul=
ate-iops-in-a-storage-array/">http://www.techrepublic.com/blog/the-enterpri=
se-cloud/calculate-iops-in-a-storage-array/</a></div><div><br></div><div>Ba=
sically, just assume that you're getting 130 iops per drive. Well, 16 drive=
s in a RAID 0 is going to max you out at 2100ish iops, which is low for you=
r stated peak usage (but probably within range for your average usage). How=
ever, you start looking at 8+8 or 7+7 with HSP, you're looking at cutting t=
hat in half, and you're going to see I/O wait, period. Of course if you wer=
e to use a non-recommended RAID5, you'd be taking an even bigger hit on wri=
tes.</div><div><br></div><div>Now, I do _not_ want to open a can of worms h=
ere and start a war about SSD versus spindles and "perceived performance vs=
 real," or any such thing. I attended Greg Smith's &nbsp;(excellent) talk i=
n Austin at PG Day wrt "Seeking Postgres," and I had also personally amasse=
d quite a bit of data on such comparisons myself. &nbsp;Unfortunately, some=
 of that talk not compare apples to apples (3-disk RAID 0 versus singe Inte=
l 520SSD), and I quite simply find that the benchmarks do not really reflec=
t real world usage.</div><div></div><div><br></div><div>Source: months and =
months of real-world stress-testing specifically 10k drives (SAS) against S=
SD (SATA) drives in the same configuration on the same machine using the sa=
me tests plus over a year (total) of production deployment among 3 servers =
thusly configured.<br><br>So far as personal experience with the Intel driv=
es, I don't have that, personally. I'm using Crucial, and I'm pretty happy =
with those. The _problem_ with SSD is there is no "put it in the freezer" m=
agic bullet. When they fail, they fail, and they're gone. So, IMO (and ther=
e are MANY MANY valid opinions on this), use slightly cheaper drives and pr=
oactively replace them every 9 months or year.</div><div><br></div><div><br=
><br><hr id=3D"zwchr"><blockquote style=3D"border-left:2px solid #1010FF;ma=
rgin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:nor=
mal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:1=
2pt;"><div class=3D"gmail_quote">On Mon, Aug 12, 2013 at 8:28 AM, David F. =
Skoll <span dir=3D"ltr">&lt;<a href=3D"mailto:dfs@roaringpenguin.com" targe=
t=3D"_blank">dfs@roaringpenguin.com</a>&gt;</span> wrote:<br><blockquote cl=
ass=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;p=
adding-left:1ex">
<br>
3) Our current workload peaks at about 5000 transactions per second;<br>
you can assume about one-third to one-half of those are writes. &nbsp;Do<br=
>
you think we can get away with 16 10Krpm SATA drives instead of the<br>
SSDs?<br></blockquote></div><br>pgbench peaks out at 5K-7K transactions per=
 second on my server which uses just 10ea. of 7Krpm SATA drives:<br><br>&nb=
sp; WAL: RAID1 (2 disks)<br>&nbsp; Data: RAID10 (8 disks)<br>&nbsp; 3Ware R=
AID controller with BBU<br>
&nbsp; 2x4 core Intel CPUs<br>&nbsp; 12 GB memory<br><br>I don't know how p=
gbench compares to your workload.&nbsp; But suspect 16 10K SATA drives woul=
d be pretty fast if you combine them with a BBU RAID controller.<br><br>On =
the other hand, I swore this would be the last server I buy with spinning s=
torage.<br>
<br>Craig<br><br>
</blockquote><br></div></div></body></html>
------=_Part_266766_3626445.1376323269106--