pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedSAN/NAS options
25+ messages / 12 participants
[nested] [flat]
* SAN/NAS options
@ 2005-12-14 06:43 Charles Sprickman <spork@bway.net>
2005-12-21 05:58 ` Re: SAN/NAS options Charles Sprickman <spork@bway.net>
0 siblings, 1 reply; 25+ messages in thread
From: Charles Sprickman @ 2005-12-14 06:43 UTC (permalink / raw)
To: pgsql-performance
Hello all,
It seems that I'm starting to outgrow our current Postgres setup. We've
been running a handful of machines as standalone db servers. This is all
in a colocation environment, so everything is stuffed into 1U Supermicro
boxes. Our standard build looks like this:
Supermicro 1U w/SCA backplane and 4 bays
2x2.8 GHz Xeons
Adaptec 2015S "zero channel" RAID card
2 or 4 x 73GB Seagate 10K Ultra 320 drives (mirrored+striped)
2GB RAM
FreeBSD 4.11
PGSQL data from 5-10GB per box
Recently I started studying what we were running up against in our nightly
runs that do a ton of updates/inserts to prep things for the tasks the db
does during the business day (light mix of selects/inserts/updates).
While we have plenty of disk bandwidth (according to bonnie), we are
really dying on IOPS. I'm guessing this is a mix of a rather anemic RAID
controller (ever notice how adaptec doesn't publish any real
performance specs on raid cards?) and having only two or four spindles
(effectively 1 or 2 on writes).
So that's where we are...
I'm new to the whole SAN thing, but did recently pick up a few used NetApp
shelves and a Fibre Channel RAID HBA (Mylex ExtremeRAID 3000, also used)
to toy with. I started wondering if I could put something together to
both get our storage on one set of boxes and allow me to get data striped
across more drives. Our budget is not huge and we are not adverse to
getting used gear where appropriate.
What do you folks recommend? I'm just starting to look at what's out
there for SANs and NAS, and from what I've seen, our options are:
NetApp Filers - the pluses with these are that if we use NFS, we don't
have to worry about either large filesystem support in FreeBSD (2TB
practical limit), or limitation on "growing" partitions as the NetApp just
deals with that. I also understand these make backups a bit simpler. I
have a great, trusted, spare-stocking source for these.
Apple X-Serve RAID - well, it's pretty cheap. Honestly, that's all I know
about it - they don't talk about IOPS numbers, and I have no idea what
lurks in that box as a RAID controller.
SAN box w/integrated RAID - it seems like this might not be a good choice
since the RAID hardware in the box may be where I hit any limits. I also
imagine I'm probably overpaying for some OEM RAID controller integrated
into the box. No idea where to look for used gear.
SAN box, JBOD - this seems like it might be affordable as well. A few big
shelves full of drives a SAN "switch" to plug all the shelves and hosts
into and a FC RAID card in each host. No idea where to look for used gear
here either.
You'll note that I'm being somewhat driven by my OS of choice, FreeBSD.
Unlike Solaris or other commercial offerings, there is no nice volume
management available. While I'd love to keep managing a dozen or so
FreeBSD boxes, I could be persuaded to go to Solaris x86 if the volume
management really shines and Postgres performs well on it.
Lastly, one thing that I'm not yet finding in trying to educate myself on
SANs is a good overview of what's come out in the past few years that's
more affordable than the old big-iron stuff. For example I saw some brief
info on this list's archives about the Dell/EMC offerings. Anything else
in that vein to look at?
I hope this isn't too far off topic for this list. Postgres is the
main application that I'm looking to accomodate. Anything else I can do
with whatever solution we find is just gravy...
Thanks!
Charles
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: SAN/NAS options
2005-12-14 06:43 SAN/NAS options Charles Sprickman <spork@bway.net>
@ 2005-12-21 05:58 ` Charles Sprickman <spork@bway.net>
2005-12-21 10:17 ` Re: SAN/NAS options Alex Stapleton <alexs@advfn.com>
2005-12-21 10:46 ` Re: SAN/NAS options Luke Lonergan <llonergan@greenplum.com>
0 siblings, 2 replies; 25+ messages in thread
From: Charles Sprickman @ 2005-12-21 05:58 UTC (permalink / raw)
To: pgsql-performance
On Wed, 14 Dec 2005, Charles Sprickman wrote:
[big snip]
The list server seems to be regurgitating old stuff, and in doing so it
reminded me to thank everyone for their input. I was kind of waiting to
see if anyone who was very pro-NAS/SAN was going to pipe up, but it looks
like most people are content with per-host storage.
You've given me a lot to go on... Now I'm going to have to do some
research as to real-world RAID controller performance. It's vexing (to
say the least) that most vendors don't supply any raw throughput or TPS
stats on this stuff...
Anyhow, thanks again. You'll probably see me back here in the coming
months as I try to shake some mysql info out of my brain as our pgsql DBA
gets me up to speed on pgsql and what specifically he's doing to stress
things.
Charles
> I hope this isn't too far off topic for this list. Postgres is the main
> application that I'm looking to accomodate. Anything else I can do with
> whatever solution we find is just gravy...
>
> Thanks!
>
> Charles
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster
>
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: SAN/NAS options
2005-12-14 06:43 SAN/NAS options Charles Sprickman <spork@bway.net>
2005-12-21 05:58 ` Re: SAN/NAS options Charles Sprickman <spork@bway.net>
@ 2005-12-21 10:17 ` Alex Stapleton <alexs@advfn.com>
1 sibling, 0 replies; 25+ messages in thread
From: Alex Stapleton @ 2005-12-21 10:17 UTC (permalink / raw)
To: Charles Sprickman <spork@bway.net>; +Cc: pgsql-performance
>
>> I hope this isn't too far off topic for this list. Postgres is
>> the main application that I'm looking to accomodate. Anything
>> else I can do with whatever solution we find is just gravy...
> You've given me a lot to go on... Now I'm going to have to do some
> research as to real-world RAID controller performance. It's vexing
> (to say the least) that most vendors don't supply any raw
> throughput or TPS stats on this stuff...
One word of advice. Stay away from Dell kit. The PERC 4 controllers
they use don't implement RAID 10 properly. It's RAID 1 + JBOD array.
It also has generally dismal IOPS performance too. You might get away
with running software RAID, either in conjunction with, or entirely
avoiding the card.
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: SAN/NAS options
2005-12-14 06:43 SAN/NAS options Charles Sprickman <spork@bway.net>
2005-12-21 05:58 ` Re: SAN/NAS options Charles Sprickman <spork@bway.net>
@ 2005-12-21 10:46 ` Luke Lonergan <llonergan@greenplum.com>
1 sibling, 0 replies; 25+ messages in thread
From: Luke Lonergan @ 2005-12-21 10:46 UTC (permalink / raw)
To: Charles Sprickman <spork@bway.net>; pgsql-performance
Charles,
On 12/20/05 9:58 PM, "Charles Sprickman" <spork@bway.net> wrote:
> You've given me a lot to go on... Now I'm going to have to do some
> research as to real-world RAID controller performance. It's vexing (to
> say the least) that most vendors don't supply any raw throughput or TPS
> stats on this stuff...
Take a look at this:
http://www.wlug.org.nz/HarddiskBenchmarks
> Anyhow, thanks again. You'll probably see me back here in the coming
> months as I try to shake some mysql info out of my brain as our pgsql DBA
> gets me up to speed on pgsql and what specifically he's doing to stress
> things.
Cool!
BTW - based on the above benchmark page, I just immediately ordered 2 x of
the Areca 1220 SATA controllers (
http://www.areca.com.tw/products/html/pcie-sata.htm) so that we can compare
them to the 3Ware 9550SX that we've been using. The 3Ware controllers have
been super fast on sequential access, but I'm concerned about their random
IOPs. The Areca's aren't as popular, and there's consequently less volume
of them, but people who use them rave about them.
- Luke
^ permalink raw reply [nested|flat] 25+ messages in thread
* SAN/NAS options
@ 2005-12-14 06:56 Charles Sprickman <spork@bway.net>
2005-12-14 07:28 ` Re: SAN/NAS options Mark Kirkwood <markir@paradise.net.nz>
2005-12-14 16:53 ` Re: SAN/NAS options Andrew Rawnsley <ronz@investoranalytics.com>
2005-12-16 22:19 ` Re: SAN/NAS options Jim C. Nasby <jnasby@pervasive.com>
2006-01-15 02:37 ` Re: SAN/NAS options Charles Sprickman <spork@bway.net>
0 siblings, 4 replies; 25+ messages in thread
From: Charles Sprickman @ 2005-12-14 06:56 UTC (permalink / raw)
To: pgsql-performance
Hello all,
It seems that I'm starting to outgrow our current Postgres setup. We've been
running a handful of machines as standalone db servers. This is all in a
colocation environment, so everything is stuffed into 1U Supermicro boxes. Our
standard build looks like this:
Supermicro 1U w/SCA backplane and 4 bays
2x2.8 GHz Xeons
Adaptec 2015S "zero channel" RAID card
2 or 4 x 73GB Seagate 10K Ultra 320 drives (mirrored+striped)
2GB RAM
FreeBSD 4.11
PGSQL data from 5-10GB per box
Recently I started studying what we were running up against in our nightly runs
that do a ton of updates/inserts to prep things for the tasks the db does
during the business day (light mix of selects/inserts/updates). While we have
plenty of disk bandwidth (according to bonnie), we are really dying on IOPS.
I'm guessing this is a mix of a rather anemic RAID controller (ever notice how
adaptec doesn't publish any real performance specs on raid cards?) and having
only two or four spindles (effectively 1 or 2 on writes).
So that's where we are...
I'm new to the whole SAN thing, but did recently pick up a few used NetApp
shelves and a Fibre Channel RAID HBA (Mylex ExtremeRAID 3000, also used) to toy
with. I started wondering if I could put something together to both get our
storage on one set of boxes and allow me to get data striped across more
drives. Our budget is not huge and we are not adverse to getting used gear
where appropriate.
What do you folks recommend? I'm just starting to look at what's out there for
SANs and NAS, and from what I've seen, our options are:
NetApp Filers - the pluses with these are that if we use NFS, we don't have to
worry about either large filesystem support in FreeBSD (2TB practical limit),
or limitation on "growing" partitions as the NetApp just deals with that. I
also understand these make backups a bit simpler. I have a great, trusted,
spare-stocking source for these.
Apple X-Serve RAID - well, it's pretty cheap. Honestly, that's all I know
about it - they don't talk about IOPS numbers, and I have no idea what lurks in
that box as a RAID controller.
SAN box w/integrated RAID - it seems like this might not be a good choice since
the RAID hardware in the box may be where I hit any limits. I also imagine I'm
probably overpaying for some OEM RAID controller integrated into the box. No
idea where to look for used gear.
SAN box, JBOD - this seems like it might be affordable as well. A few big
shelves full of drives a SAN "switch" to plug all the shelves and hosts into
and a FC RAID card in each host. No idea where to look for used gear here
either.
You'll note that I'm being somewhat driven by my OS of choice, FreeBSD. Unlike
Solaris or other commercial offerings, there is no nice volume management
available. While I'd love to keep managing a dozen or so FreeBSD boxes, I
could be persuaded to go to Solaris x86 if the volume management really shines
and Postgres performs well on it.
Lastly, one thing that I'm not yet finding in trying to educate myself on SANs
is a good overview of what's come out in the past few years that's more
affordable than the old big-iron stuff. For example I saw some brief info on
this list's archives about the Dell/EMC offerings. Anything else in that vein
to look at?
I hope this isn't too far off topic for this list. Postgres is the main
application that I'm looking to accomodate. Anything else I can do with
whatever solution we find is just gravy...
Thanks!
Charles
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: SAN/NAS options
2005-12-14 06:56 SAN/NAS options Charles Sprickman <spork@bway.net>
@ 2005-12-14 07:28 ` Mark Kirkwood <markir@paradise.net.nz>
2005-12-16 22:18 ` Re: SAN/NAS options Jim C. Nasby <jnasby@pervasive.com>
3 siblings, 1 reply; 25+ messages in thread
From: Mark Kirkwood @ 2005-12-14 07:28 UTC (permalink / raw)
To: Charles Sprickman <spork@bway.net>; +Cc: pgsql-performance
Charles Sprickman wrote:
> Hello all,
>
> It seems that I'm starting to outgrow our current Postgres setup. We've
> been running a handful of machines as standalone db servers. This is
> all in a colocation environment, so everything is stuffed into 1U
> Supermicro boxes. Our standard build looks like this:
>
> Supermicro 1U w/SCA backplane and 4 bays
> 2x2.8 GHz Xeons
> Adaptec 2015S "zero channel" RAID card
> 2 or 4 x 73GB Seagate 10K Ultra 320 drives (mirrored+striped)
> 2GB RAM
> FreeBSD 4.11
> PGSQL data from 5-10GB per box
>
> Recently I started studying what we were running up against in our
> nightly runs that do a ton of updates/inserts to prep things for the
> tasks the db does during the business day (light mix of
> selects/inserts/updates). While we have plenty of disk bandwidth
> (according to bonnie), we are really dying on IOPS. I'm guessing this is
> a mix of a rather anemic RAID controller (ever notice how adaptec
> doesn't publish any real performance specs on raid cards?) and having
> only two or four spindles (effectively 1 or 2 on writes).
>
> So that's where we are...
>
> I'm new to the whole SAN thing, but did recently pick up a few used
> NetApp shelves and a Fibre Channel RAID HBA (Mylex ExtremeRAID 3000,
> also used) to toy with. I started wondering if I could put something
> together to both get our storage on one set of boxes and allow me to get
> data striped across more drives. Our budget is not huge and we are not
> adverse to getting used gear where appropriate.
>
> What do you folks recommend? I'm just starting to look at what's out
> there for SANs and NAS, and from what I've seen, our options are:
>
Leaving the whole SAN issue for a moment:
It would be interesting to see if moving to FreeBSD 6.0 would help you -
the vfs layer is no longer throttled by the (SMP) GIANT lock in this
version, and that may make quite a difference (given you have SMP boxes).
Another interesting thing to try is rebuilding the database ufs
filesystem(s) with 32K blocks and 4K frags (as opposed to 8K/1K or
16K/2K - can't recall the default on 4.x). I found this to give a factor
of 2 speedup on random disk access (specifically queries doing indexed
joins).
Is it mainly your 2 disk machines that are IOPS bound? if so, a cheap
option may be to buy 2 more cheetahs for them! If it's the 4's, well how
about a 2U U320 diskpack from whomever supplies you the Supermicro boxes?
I have just noticed Luke's posting - I would second the advice to avoid
SAN - in my experience it's an expensive way to buy storage.
best wishes
Mark
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: SAN/NAS options
2005-12-14 06:56 SAN/NAS options Charles Sprickman <spork@bway.net>
2005-12-14 07:28 ` Re: SAN/NAS options Mark Kirkwood <markir@paradise.net.nz>
@ 2005-12-16 22:18 ` Jim C. Nasby <jnasby@pervasive.com>
2005-12-16 22:49 ` Re: SAN/NAS options Mark Kirkwood <markir@paradise.net.nz>
2005-12-16 22:51 ` Re: SAN/NAS options Michael Stone <mstone+postgres@mathom.us>
0 siblings, 2 replies; 25+ messages in thread
From: Jim C. Nasby @ 2005-12-16 22:18 UTC (permalink / raw)
To: Mark Kirkwood <markir@paradise.net.nz>; +Cc: Charles Sprickman <spork@bway.net>; pgsql-performance
On Wed, Dec 14, 2005 at 08:28:56PM +1300, Mark Kirkwood wrote:
> Another interesting thing to try is rebuilding the database ufs
> filesystem(s) with 32K blocks and 4K frags (as opposed to 8K/1K or
> 16K/2K - can't recall the default on 4.x). I found this to give a factor
> of 2 speedup on random disk access (specifically queries doing indexed
> joins).
Even if you're doing a lot of random IO? I would think that random IO
would perform better if you use smaller (8K) blocks, since there's less
data being read in and then just thrown away that way.
> Is it mainly your 2 disk machines that are IOPS bound? if so, a cheap
> option may be to buy 2 more cheetahs for them! If it's the 4's, well how
> about a 2U U320 diskpack from whomever supplies you the Supermicro boxes?
Also, on the 4 drive machines if you can spare the room you might see a
big gain by putting the tables on one mirror and the OS and transaction
logs on the other.
--
Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: SAN/NAS options
2005-12-14 06:56 SAN/NAS options Charles Sprickman <spork@bway.net>
2005-12-14 07:28 ` Re: SAN/NAS options Mark Kirkwood <markir@paradise.net.nz>
2005-12-16 22:18 ` Re: SAN/NAS options Jim C. Nasby <jnasby@pervasive.com>
@ 2005-12-16 22:49 ` Mark Kirkwood <markir@paradise.net.nz>
1 sibling, 0 replies; 25+ messages in thread
From: Mark Kirkwood @ 2005-12-16 22:49 UTC (permalink / raw)
To: Jim C. Nasby <jnasby@pervasive.com>; +Cc: Charles Sprickman <spork@bway.net>; pgsql-performance
Jim C. Nasby wrote:
> On Wed, Dec 14, 2005 at 08:28:56PM +1300, Mark Kirkwood wrote:
>
>>Another interesting thing to try is rebuilding the database ufs
>>filesystem(s) with 32K blocks and 4K frags (as opposed to 8K/1K or
>>16K/2K - can't recall the default on 4.x). I found this to give a factor
>>of 2 speedup on random disk access (specifically queries doing indexed
>>joins).
>
>
> Even if you're doing a lot of random IO? I would think that random IO
> would perform better if you use smaller (8K) blocks, since there's less
> data being read in and then just thrown away that way.
>
>
Yeah, that's what I would have expected too! but the particular queries
I tested do a ton of random IO (correlation of 0.013 on the join column
for the big table). I did wonder if the gain has something to do with
the underlying RAID stripe size (64K or 256K in my case), as I have only
tested the 32K vs 8K/16K on RAIDed systems.
I guess for a system where the number of concurrent users give rise to
memory pressure, it will cause more thrashing of the file buffer cache,
much could be a net loss.
Still worth trying out I think, you will know soon enough if it is a win
or lose!
Note that I did *not* alter Postgres page/block size (BLCKSZ) from 8K,
so no dump/reload is required to test this out.
cheers
Mark
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: SAN/NAS options
2005-12-14 06:56 SAN/NAS options Charles Sprickman <spork@bway.net>
2005-12-14 07:28 ` Re: SAN/NAS options Mark Kirkwood <markir@paradise.net.nz>
2005-12-16 22:18 ` Re: SAN/NAS options Jim C. Nasby <jnasby@pervasive.com>
@ 2005-12-16 22:51 ` Michael Stone <mstone+postgres@mathom.us>
2005-12-17 00:25 ` Re: SAN/NAS options Jim C. Nasby <jnasby@pervasive.com>
1 sibling, 1 reply; 25+ messages in thread
From: Michael Stone @ 2005-12-16 22:51 UTC (permalink / raw)
To: pgsql-performance
On Fri, Dec 16, 2005 at 04:18:01PM -0600, Jim C. Nasby wrote:
>Even if you're doing a lot of random IO? I would think that random IO
>would perform better if you use smaller (8K) blocks, since there's less
>data being read in and then just thrown away that way.
The overhead of reading an 8k block instead of a 32k block is too small
to measure on modern hardware. The seek is what dominates; leaving the
read head on a little longer and then transmitting a little more over a
200 megabyte channel is statistical fuzz.
Mike Stone
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: SAN/NAS options
2005-12-14 06:56 SAN/NAS options Charles Sprickman <spork@bway.net>
2005-12-14 07:28 ` Re: SAN/NAS options Mark Kirkwood <markir@paradise.net.nz>
2005-12-16 22:18 ` Re: SAN/NAS options Jim C. Nasby <jnasby@pervasive.com>
2005-12-16 22:51 ` Re: SAN/NAS options Michael Stone <mstone+postgres@mathom.us>
@ 2005-12-17 00:25 ` Jim C. Nasby <jnasby@pervasive.com>
2005-12-17 00:48 ` Re: SAN/NAS options Michael Stone <mstone+postgres@mathom.us>
0 siblings, 1 reply; 25+ messages in thread
From: Jim C. Nasby @ 2005-12-17 00:25 UTC (permalink / raw)
To: pgsql-performance
On Fri, Dec 16, 2005 at 05:51:03PM -0500, Michael Stone wrote:
> On Fri, Dec 16, 2005 at 04:18:01PM -0600, Jim C. Nasby wrote:
> >Even if you're doing a lot of random IO? I would think that random IO
> >would perform better if you use smaller (8K) blocks, since there's less
> >data being read in and then just thrown away that way.
>
> The overhead of reading an 8k block instead of a 32k block is too small
> to measure on modern hardware. The seek is what dominates; leaving the
> read head on a little longer and then transmitting a little more over a
> 200 megabyte channel is statistical fuzz.
True, but now you've got 4x the amount of data in your cache that you
probably don't need.
Looks like time to do some benchmarking...
--
Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: SAN/NAS options
2005-12-14 06:56 SAN/NAS options Charles Sprickman <spork@bway.net>
2005-12-14 07:28 ` Re: SAN/NAS options Mark Kirkwood <markir@paradise.net.nz>
2005-12-16 22:18 ` Re: SAN/NAS options Jim C. Nasby <jnasby@pervasive.com>
2005-12-16 22:51 ` Re: SAN/NAS options Michael Stone <mstone+postgres@mathom.us>
2005-12-17 00:25 ` Re: SAN/NAS options Jim C. Nasby <jnasby@pervasive.com>
@ 2005-12-17 00:48 ` Michael Stone <mstone+postgres@mathom.us>
0 siblings, 0 replies; 25+ messages in thread
From: Michael Stone @ 2005-12-17 00:48 UTC (permalink / raw)
To: pgsql-performance
On Fri, Dec 16, 2005 at 06:25:25PM -0600, Jim C. Nasby wrote:
>True, but now you've got 4x the amount of data in your cache that you
>probably don't need.
Or you might be 4x more likely to have data cached that's needed later.
If you're hitting disk either way, that's probably more likely than the
extra IO pushing something critical out--if *all* the important stuff
were cached you wouldn't be doing the seeks in the first place. This
will obviously be heavily dependent on the amount of ram you've got and
your workload, so (as always) you'll have to benchmark it to get past
the hand-waving stage.
Mike Stone
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: SAN/NAS options
2005-12-14 06:56 SAN/NAS options Charles Sprickman <spork@bway.net>
@ 2005-12-14 16:53 ` Andrew Rawnsley <ronz@investoranalytics.com>
2005-12-14 18:32 ` Re: SAN/NAS options Michael Stone <mstone+postgres@mathom.us>
3 siblings, 1 reply; 25+ messages in thread
From: Andrew Rawnsley @ 2005-12-14 16:53 UTC (permalink / raw)
To: Charles Sprickman <spork@bway.net>; pgsql-performance
The Apple is, as you say, cheap (except, the Apple markup on the disks
fuzzes that a bit). Its easy to set up, and has been quite reliable for me,
but do not expect anything resembling good DB performance out of it (I gave
up running anything but backup DBs on it). From the mouth of Apple guys, it
(and Xsan) are heavily optimized for sequential access. They want to sell
piles of these to the music/film industry, where they have some cred. Oracle
has apparently gotten some performance gains through raw device pixie dust
and voodoo, but even as a (reluctant, kicking-and-screaming) Oracle guy I
wouldn't go there.
Other goofy things about it: it isn't 1 device with 14 disks and redundant
controllers. Its 2 7 disk arrays with non-redundant controllers. It doesn't
do RAID10.
If you want a gob-o-space with no performance requirements, its fine.
Otherwise...
On 12/14/05 1:56 AM, "Charles Sprickman" <spork@bway.net> wrote:
> Hello all,
>
> It seems that I'm starting to outgrow our current Postgres setup. We've been
> running a handful of machines as standalone db servers. This is all in a
> colocation environment, so everything is stuffed into 1U Supermicro boxes.
> Our
> standard build looks like this:
>
> Supermicro 1U w/SCA backplane and 4 bays
> 2x2.8 GHz Xeons
> Adaptec 2015S "zero channel" RAID card
> 2 or 4 x 73GB Seagate 10K Ultra 320 drives (mirrored+striped)
> 2GB RAM
> FreeBSD 4.11
> PGSQL data from 5-10GB per box
>
> Recently I started studying what we were running up against in our nightly
> runs
> that do a ton of updates/inserts to prep things for the tasks the db does
> during the business day (light mix of selects/inserts/updates). While we have
> plenty of disk bandwidth (according to bonnie), we are really dying on IOPS.
> I'm guessing this is a mix of a rather anemic RAID controller (ever notice how
> adaptec doesn't publish any real performance specs on raid cards?) and having
> only two or four spindles (effectively 1 or 2 on writes).
>
> So that's where we are...
>
> I'm new to the whole SAN thing, but did recently pick up a few used NetApp
> shelves and a Fibre Channel RAID HBA (Mylex ExtremeRAID 3000, also used) to
> toy
> with. I started wondering if I could put something together to both get our
> storage on one set of boxes and allow me to get data striped across more
> drives. Our budget is not huge and we are not adverse to getting used gear
> where appropriate.
>
> What do you folks recommend? I'm just starting to look at what's out there
> for
> SANs and NAS, and from what I've seen, our options are:
>
> NetApp Filers - the pluses with these are that if we use NFS, we don't have to
> worry about either large filesystem support in FreeBSD (2TB practical limit),
> or limitation on "growing" partitions as the NetApp just deals with that. I
> also understand these make backups a bit simpler. I have a great, trusted,
> spare-stocking source for these.
>
> Apple X-Serve RAID - well, it's pretty cheap. Honestly, that's all I know
> about it - they don't talk about IOPS numbers, and I have no idea what lurks
> in
> that box as a RAID controller.
>
> SAN box w/integrated RAID - it seems like this might not be a good choice
> since
> the RAID hardware in the box may be where I hit any limits. I also imagine
> I'm
> probably overpaying for some OEM RAID controller integrated into the box. No
> idea where to look for used gear.
>
> SAN box, JBOD - this seems like it might be affordable as well. A few big
> shelves full of drives a SAN "switch" to plug all the shelves and hosts into
> and a FC RAID card in each host. No idea where to look for used gear here
> either.
>
> You'll note that I'm being somewhat driven by my OS of choice, FreeBSD. Unlike
> Solaris or other commercial offerings, there is no nice volume management
> available. While I'd love to keep managing a dozen or so FreeBSD boxes, I
> could be persuaded to go to Solaris x86 if the volume management really shines
> and Postgres performs well on it.
>
> Lastly, one thing that I'm not yet finding in trying to educate myself on SANs
> is a good overview of what's come out in the past few years that's more
> affordable than the old big-iron stuff. For example I saw some brief info on
> this list's archives about the Dell/EMC offerings. Anything else in that vein
> to look at?
>
> I hope this isn't too far off topic for this list. Postgres is the main
> application that I'm looking to accomodate. Anything else I can do with
> whatever solution we find is just gravy...
>
> Thanks!
>
> Charles
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: don't forget to increase your free space map settings
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: SAN/NAS options
2005-12-14 06:56 SAN/NAS options Charles Sprickman <spork@bway.net>
2005-12-14 16:53 ` Re: SAN/NAS options Andrew Rawnsley <ronz@investoranalytics.com>
@ 2005-12-14 18:32 ` Michael Stone <mstone+postgres@mathom.us>
0 siblings, 0 replies; 25+ messages in thread
From: Michael Stone @ 2005-12-14 18:32 UTC (permalink / raw)
To: pgsql-performance
On Wed, Dec 14, 2005 at 11:53:52AM -0500, Andrew Rawnsley wrote:
>Other goofy things about it: it isn't 1 device with 14 disks and redundant
>controllers. Its 2 7 disk arrays with non-redundant controllers. It doesn't
>do RAID10.
And if you want hot spares you need *two* per tray (one for each
controller). That definately changes the cost curve. :)
Mike Stone
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: SAN/NAS options
2005-12-14 06:56 SAN/NAS options Charles Sprickman <spork@bway.net>
@ 2005-12-16 22:19 ` Jim C. Nasby <jnasby@pervasive.com>
2005-12-20 00:41 ` Re: SAN/NAS options Matthew Schumacher <matt.s@aptalaska.net>
3 siblings, 1 reply; 25+ messages in thread
From: Jim C. Nasby @ 2005-12-16 22:19 UTC (permalink / raw)
To: Charles Sprickman <spork@bway.net>; +Cc: pgsql-performance
On Wed, Dec 14, 2005 at 01:56:10AM -0500, Charles Sprickman wrote:
You'll note that I'm being somewhat driven by my OS of choice, FreeBSD.
> Unlike Solaris or other commercial offerings, there is no nice volume
> management available. While I'd love to keep managing a dozen or so
> FreeBSD boxes, I could be persuaded to go to Solaris x86 if the volume
> management really shines and Postgres performs well on it.
Have you looked at vinum? It might not qualify as a true volume manager,
but it's still pretty handy.
--
Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: SAN/NAS options
2005-12-14 06:56 SAN/NAS options Charles Sprickman <spork@bway.net>
2005-12-16 22:19 ` Re: SAN/NAS options Jim C. Nasby <jnasby@pervasive.com>
@ 2005-12-20 00:41 ` Matthew Schumacher <matt.s@aptalaska.net>
0 siblings, 0 replies; 25+ messages in thread
From: Matthew Schumacher @ 2005-12-20 00:41 UTC (permalink / raw)
To: pgsql-performance
Jim C. Nasby wrote:
> On Wed, Dec 14, 2005 at 01:56:10AM -0500, Charles Sprickman wrote:
> You'll note that I'm being somewhat driven by my OS of choice, FreeBSD.
>
>>Unlike Solaris or other commercial offerings, there is no nice volume
>>management available. While I'd love to keep managing a dozen or so
>>FreeBSD boxes, I could be persuaded to go to Solaris x86 if the volume
>>management really shines and Postgres performs well on it.
>
>
> Have you looked at vinum? It might not qualify as a true volume manager,
> but it's still pretty handy.
I am looking very closely at purchasing a SANRAD Vswitch 2000, a Nexsan
SATABoy with SATA disks, and the Qlogic iscsi controller cards.
Nexsan claims up to 370MB/s sustained per controller and 44,500 IOPS but
I'm not sure if that is good or bad. It's certainly faster than the LSI
megaraid controller I'm using now with a raid 1 mirror.
The sanrad box looks like it saves money in that you don't have to by
controller cards for everything, but for I/O intensive servers such as
the database server, I would end up buying an iscsi controller card anyway.
At this point I'm not sure what the best solution is. I like the idea
of having logical disks available though iscsi because of how flexible
it is, but I really don't want to spend $20k (10 for the nexsan and 10
for the sanrad) and end up with poor performance.
On other advantage to iscsi is that I can go completely diskless on my
servers and boot from iscsi which means that I don't have to have spare
disks for each host, now I just have spare disks for the nexsan chassis.
So the question becomes: has anyone put postgres on an iscsi san, and if
so how did it perform?
schu
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: SAN/NAS options
2005-12-14 06:56 SAN/NAS options Charles Sprickman <spork@bway.net>
@ 2006-01-15 02:37 ` Charles Sprickman <spork@bway.net>
2006-01-15 03:02 ` Re: SAN/NAS options Luke Lonergan <llonergan@greenplum.com>
2006-01-15 04:04 ` Re: SAN/NAS options Christopher Browne <cbbrowne@acm.org>
2006-01-18 18:54 ` Re: SAN/NAS options Michael Adler <adler@pobox.com>
3 siblings, 3 replies; 25+ messages in thread
From: Charles Sprickman @ 2006-01-15 02:37 UTC (permalink / raw)
To: pgsql-performance
Following up to myself again...
On Wed, 14 Dec 2005, Charles Sprickman wrote:
> Hello all,
>
> Supermicro 1U w/SCA backplane and 4 bays
> 2x2.8 GHz Xeons
> Adaptec 2015S "zero channel" RAID card
I don't want to throw away the four machines like that that we have. I do
want to throw away the ZCR cards... :) If I ditch those I still have a 1U
box with a U320 scsi plug on the back.
I'm vaguely considering pairing these two devices:
http://www.areca.us/products/html/products.htm
That's an Areca 16 channel SATA II (I haven't even read up on what's new
in SATA II) RAID controller with an optional U320 SCSI daughter card to
connect to the host(s).
http://www.chenbro.com.tw/Chenbro_Special/RM321.php
How can I turn that box down? Those people in the picture look very
excited about it? Seriously though, it looks like an interesting and
economical pairing that gives me most of what I'm looking for:
-a modern RAID engine
-small form factor
-remote management of the array
-ability to reuse my current db hosts that are disk-bound
Disadvantages:
-only 1 or 2 hosts per box
-more difficult to move storage from host to host (compared to a SAN or
NAS system)
-no fancy NetApp features like snapshots
-I have no experience with Areca SATA->SCSI RAID controllers
Any thoughts on this? The controller looks to be about $1500, the
enclosure about $400, and the drives are no great mystery, cost would
depend on what total capacity I'm looking for.
Our initial plan is to set one up for storage for a mail archive project,
and to also have a host use this storage to host replicated copies of all
Postgres databases. If things look good, we'd start moving our main PG
hosts to use a similar RAID box.
Thanks,
Charles
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: SAN/NAS options
2005-12-14 06:56 SAN/NAS options Charles Sprickman <spork@bway.net>
2006-01-15 02:37 ` Re: SAN/NAS options Charles Sprickman <spork@bway.net>
@ 2006-01-15 03:02 ` Luke Lonergan <llonergan@greenplum.com>
2006-01-15 03:23 ` Re: SAN/NAS options Charles Sprickman <spork@bway.net>
2 siblings, 1 reply; 25+ messages in thread
From: Luke Lonergan @ 2006-01-15 03:02 UTC (permalink / raw)
To: Charles Sprickman <spork@bway.net>; pgsql-performance
Charles,
On 1/14/06 6:37 PM, "Charles Sprickman" <spork@bway.net> wrote:
> I'm vaguely considering pairing these two devices:
>
> http://www.areca.us/products/html/products.htm
>
> That's an Areca 16 channel SATA II (I haven't even read up on what's new
> in SATA II) RAID controller with an optional U320 SCSI daughter card to
> connect to the host(s).
I'm confused - SATA with a SCSI daughter card? Where does the SCSI go?
The Areca has a number (8,12,16) of single drive attach SATA ports coming
out of it, each of which will go to a disk drive connection on the
backplane.
> http://www.chenbro.com.tw/Chenbro_Special/RM321.php
>
> How can I turn that box down? Those people in the picture look very
> excited about it? Seriously though, it looks like an interesting and
> economical pairing that gives me most of what I'm looking for:
What a picture! I'm totally enthusiastic all of a sudden! I'm putting !!!
at the end of every sentence!
We just bought 4 very similar systems that use the chassis from California
Design - our latest favorite source:
http://www.asacomputers.com/
They did an excellent job of setting the systems up, with proper labeling
and Quality Control. They also installed Fedora Core 4 and set up the
filesystems, the only mistake they made was that they didn't enable 2TB
clipping so that we had to rebuild the RAIDs (and install CentOS with the
xfs filesystem).
We paid $10.4K each for 16x 400GB WD RE2 SATA II drives, 16GB RAM and two
Opteron 250s. We also put a single 200GB SATA system drive into each. RAID
card is the 3Ware 9550SX.
Performance has been stunning - we're getting 800MB/s sustained I/O
throughput using the two 9550SX controllers in parallel.
> Any thoughts on this? The controller looks to be about $1500, the
> enclosure about $400, and the drives are no great mystery, cost would
> depend on what total capacity I'm looking for.
I'd get ASA to build it for you - use the Tyan 2882 series motherboard for
greatest stablity. They may try to sell you hard on the SuperMicro boards,
we've had less luck with them.
> Our initial plan is to set one up for storage for a mail archive project,
> and to also have a host use this storage to host replicated copies of all
> Postgres databases. If things look good, we'd start moving our main PG
> hosts to use a similar RAID box.
Good approach.
I'm personally spending as much time using these machines as I can - they
are the fastest I've been on in a *long* time.
- Luke
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: SAN/NAS options
2005-12-14 06:56 SAN/NAS options Charles Sprickman <spork@bway.net>
2006-01-15 02:37 ` Re: SAN/NAS options Charles Sprickman <spork@bway.net>
2006-01-15 03:02 ` Re: SAN/NAS options Luke Lonergan <llonergan@greenplum.com>
@ 2006-01-15 03:23 ` Charles Sprickman <spork@bway.net>
2006-01-15 17:21 ` Re: SAN/NAS options Luke Lonergan <llonergan@greenplum.com>
0 siblings, 1 reply; 25+ messages in thread
From: Charles Sprickman @ 2006-01-15 03:23 UTC (permalink / raw)
To: Luke Lonergan <llonergan@greenplum.com>; +Cc: pgsql-performance
On Sat, 14 Jan 2006, Luke Lonergan wrote:
> Charles,
>
> On 1/14/06 6:37 PM, "Charles Sprickman" <spork@bway.net> wrote:
>
>> I'm vaguely considering pairing these two devices:
>>
>> http://www.areca.us/products/html/products.htm
>>
>> That's an Areca 16 channel SATA II (I haven't even read up on what's new
>> in SATA II) RAID controller with an optional U320 SCSI daughter card to
>> connect to the host(s).
>
> I'm confused - SATA with a SCSI daughter card? Where does the SCSI go?
Bad ASCII diagram follows (D=disk, C=controller H=host):
SATA ____
D -------| | SCSI ________
D -------| C |--------| H |
D -------| | |________|
D -------|____|
(etc. up
to 16
drives)
The drives and the controller go in the Chenbro case. U320 SCSI from the
RAID controller in the Chenbro case to the 1U server.
C
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: SAN/NAS options
2005-12-14 06:56 SAN/NAS options Charles Sprickman <spork@bway.net>
2006-01-15 02:37 ` Re: SAN/NAS options Charles Sprickman <spork@bway.net>
2006-01-15 03:02 ` Re: SAN/NAS options Luke Lonergan <llonergan@greenplum.com>
2006-01-15 03:23 ` Re: SAN/NAS options Charles Sprickman <spork@bway.net>
@ 2006-01-15 17:21 ` Luke Lonergan <llonergan@greenplum.com>
0 siblings, 0 replies; 25+ messages in thread
From: Luke Lonergan @ 2006-01-15 17:21 UTC (permalink / raw)
To: Charles Sprickman <spork@bway.net>; +Cc: pgsql-performance
Charles,
On 1/14/06 7:23 PM, "Charles Sprickman" <spork@bway.net> wrote:
> The drives and the controller go in the Chenbro case. U320 SCSI from the
> RAID controller in the Chenbro case to the 1U server.
Thanks for the explanation - I didn't click on your Areca link until now,
thinking it was a generic link to their products page.
Looks great - I think this might do better than the SATA -> FC products
because of the use of faster processors, but I'd keep my expectations low
until we see some performance data on it.
We've had some very poor experiences with Fibre Channel attach SATA disk
controllers. A large vendor of same ultimately concluded that they will no
longer recommend them for database use because of the terrible performance
of their unit. We ended up with a 110MB/s bottleneck on the controller when
using 200MB/s FC connections.
With the dual U320 attach and 16 drives, you should be able to saturate the
SCSI busses at about 600MB/s. It would be great if you could post your I/O
results here!
- Luke
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: SAN/NAS options
2005-12-14 06:56 SAN/NAS options Charles Sprickman <spork@bway.net>
2006-01-15 02:37 ` Re: SAN/NAS options Charles Sprickman <spork@bway.net>
@ 2006-01-15 04:04 ` Christopher Browne <cbbrowne@acm.org>
2 siblings, 0 replies; 25+ messages in thread
From: Christopher Browne @ 2006-01-15 04:04 UTC (permalink / raw)
To: pgsql-performance
> Following up to myself again...
>
> On Wed, 14 Dec 2005, Charles Sprickman wrote:
>
>> Hello all,
>>
>> Supermicro 1U w/SCA backplane and 4 bays
>> 2x2.8 GHz Xeons
>> Adaptec 2015S "zero channel" RAID card
>
> I don't want to throw away the four machines like that that we have.
> I do want to throw away the ZCR cards... :) If I ditch those I still
> have a 1U box with a U320 scsi plug on the back.
>
> I'm vaguely considering pairing these two devices:
> http://www.areca.us/products/html/products.htm
> http://www.chenbro.com.tw/Chenbro_Special/RM321.php
> How can I turn that box down? Those people in the picture look very
> excited about it? Seriously though, it looks like an interesting and
> economical pairing that gives me most of what I'm looking for:
The combination definitely looks attractive. I have only been hearing
positive things about the Areca cards; the overall combination sounds
pretty attractive.
> Disadvantages:
>
> -only 1 or 2 hosts per box
> -more difficult to move storage from host to host (compared to a SAN
> or NAS system)
> -no fancy NetApp features like snapshots
> -I have no experience with Areca SATA->SCSI RAID controllers
>
> Any thoughts on this? The controller looks to be about $1500, the
> enclosure about $400, and the drives are no great mystery, cost would
> depend on what total capacity I'm looking for.
Another "usage model" that could be appropriate would be
ATA-over-Ethernet...
<http://en.wikipedia.org/wiki/ATA-over-Ethernet;
> Our initial plan is to set one up for storage for a mail archive
> project, and to also have a host use this storage to host replicated
> copies of all Postgres databases. If things look good, we'd start
> moving our main PG hosts to use a similar RAID box.
We're thinking about some stuff like this to host things that require
bulky amounts of disk that are otherwise "not high TPC" sorts of apps.
This is definitely not a "gold plated" answer, compared to the NetApp
and EMC boxes of the world, but can be useful in contexts where they
are too expensive.
--
let name="cbbrowne" and tld="gmail.com" in String.concat "@" [name;tld];;
http://linuxdatabases.info/info/x.html
It is usually a good idea to put a capacitor of a few microfarads
across the output, as shown.
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: SAN/NAS options
2005-12-14 06:56 SAN/NAS options Charles Sprickman <spork@bway.net>
2006-01-15 02:37 ` Re: SAN/NAS options Charles Sprickman <spork@bway.net>
@ 2006-01-18 18:54 ` Michael Adler <adler@pobox.com>
2 siblings, 0 replies; 25+ messages in thread
From: Michael Adler @ 2006-01-18 18:54 UTC (permalink / raw)
To: Charles Sprickman <spork@bway.net>; +Cc: pgsql-performance
On Sat, Jan 14, 2006 at 09:37:01PM -0500, Charles Sprickman wrote:
> Following up to myself again...
>
> On Wed, 14 Dec 2005, Charles Sprickman wrote:
>
> >Hello all,
> >
> >Supermicro 1U w/SCA backplane and 4 bays
> >2x2.8 GHz Xeons
> >Adaptec 2015S "zero channel" RAID card
>
> I don't want to throw away the four machines like that that we have. I do
> want to throw away the ZCR cards... :) If I ditch those I still have a 1U
> box with a U320 scsi plug on the back.
>
> I'm vaguely considering pairing these two devices:
>
> http://www.areca.us/products/html/products.htm
>
> That's an Areca 16 channel SATA II (I haven't even read up on what's new
> in SATA II) RAID controller with an optional U320 SCSI daughter card to
> connect to the host(s).
>
> http://www.chenbro.com.tw/Chenbro_Special/RM321.php
Not sure how significant, but the RM321 backplane claims to support
SATA 150 (aka SATA I) only.
-Mike
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: SAN/NAS options
@ 2005-12-14 07:10 Luke Lonergan <LLonergan@greenplum.com>
0 siblings, 0 replies; 25+ messages in thread
From: Luke Lonergan @ 2005-12-14 07:10 UTC (permalink / raw)
To: Charles Sprickman <spork@bway.net>; pgsql-performance
Charles,
> Lastly, one thing that I'm not yet finding in trying to
> educate myself on SANs is a good overview of what's come out
> in the past few years that's more affordable than the old
> big-iron stuff. For example I saw some brief info on this
> list's archives about the Dell/EMC offerings. Anything else
> in that vein to look at?
My two cents: SAN is a bad investment, go for big internal storage.
The 3Ware or Areca SATA RAID adapters kick butt and if you look in the
newest colos (I was just in ours "365main.net" today), you will see rack
on rack of machines with from 4 to 16 internal SATA drives. Are they
all DB servers? Not necessarily, but that's where things are headed.
You can get a 3U server with dual opteron 250s, 16GB RAM and 16x 400GB
SATAII drives with the 3Ware 9550SX controller for $10K - we just
ordered 4 of them. I don't think you can buy an external disk chassis
and a Fibre channel NIC for that.
Performance? 800MB/s RAID5 reads, 400MB/s RAID5 writes. Random IOs are
also very high for RAID10, but we don't use it so YMMV - look at Areca
and 3Ware.
Managability? Good web management interfaces with 6+ years of
development from 3Ware, e-mail, online rebuild options, all the goodies.
No "snapshot" or offline backup features like the high-end SANs, but do
you really need it?
Need more power or storage over time? Run a parallel DB like Bizgres
MPP, you can add more servers with internal storage and increase your
I/O, CPU and memory.
- Luke
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: SAN/NAS options
@ 2005-12-15 21:13 Anjan Dave <adave@vantage.com>
0 siblings, 0 replies; 25+ messages in thread
From: Anjan Dave @ 2005-12-15 21:13 UTC (permalink / raw)
To: Luke Lonergan <LLonergan@greenplum.com>; Charles Sprickman <spork@bway.net>; pgsql-performance
Luke,
How did you measure 800MB/sec, is it cached, or physical I/O?
-anjan
-----Original Message-----
From: Luke Lonergan [mailto:LLonergan@greenplum.com]
Sent: Wednesday, December 14, 2005 2:10 AM
To: Charles Sprickman; pgsql-performance@postgresql.org
Subject: Re: [PERFORM] SAN/NAS options
Charles,
> Lastly, one thing that I'm not yet finding in trying to
> educate myself on SANs is a good overview of what's come out
> in the past few years that's more affordable than the old
> big-iron stuff. For example I saw some brief info on this
> list's archives about the Dell/EMC offerings. Anything else
> in that vein to look at?
My two cents: SAN is a bad investment, go for big internal storage.
The 3Ware or Areca SATA RAID adapters kick butt and if you look in the
newest colos (I was just in ours "365main.net" today), you will see rack
on rack of machines with from 4 to 16 internal SATA drives. Are they
all DB servers? Not necessarily, but that's where things are headed.
You can get a 3U server with dual opteron 250s, 16GB RAM and 16x 400GB
SATAII drives with the 3Ware 9550SX controller for $10K - we just
ordered 4 of them. I don't think you can buy an external disk chassis
and a Fibre channel NIC for that.
Performance? 800MB/s RAID5 reads, 400MB/s RAID5 writes. Random IOs are
also very high for RAID10, but we don't use it so YMMV - look at Areca
and 3Ware.
Managability? Good web management interfaces with 6+ years of
development from 3Ware, e-mail, online rebuild options, all the goodies.
No "snapshot" or offline backup features like the high-end SANs, but do
you really need it?
Need more power or storage over time? Run a parallel DB like Bizgres
MPP, you can add more servers with internal storage and increase your
I/O, CPU and memory.
- Luke
---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: SAN/NAS options
@ 2005-12-15 21:40 Luke Lonergan <LLonergan@greenplum.com>
0 siblings, 0 replies; 25+ messages in thread
From: Luke Lonergan @ 2005-12-15 21:40 UTC (permalink / raw)
To: adave@vantage.com; spork@bway.net; pgsql-performance
Physical using xfs on Linux.
- Luke
--------------------------
Sent from my BlackBerry Wireless Device
-----Original Message-----
From: Anjan Dave <adave@vantage.com>
To: Luke Lonergan <LLonergan@greenplum.com>; Charles Sprickman <spork@bway.net>; pgsql-performance@postgresql.org <pgsql-performance@postgresql.org>
Sent: Thu Dec 15 16:13:04 2005
Subject: RE: [PERFORM] SAN/NAS options
Luke,
How did you measure 800MB/sec, is it cached, or physical I/O?
-anjan
-----Original Message-----
From: Luke Lonergan [mailto:LLonergan@greenplum.com]
Sent: Wednesday, December 14, 2005 2:10 AM
To: Charles Sprickman; pgsql-performance@postgresql.org
Subject: Re: [PERFORM] SAN/NAS options
Charles,
> Lastly, one thing that I'm not yet finding in trying to
> educate myself on SANs is a good overview of what's come out
> in the past few years that's more affordable than the old
> big-iron stuff. For example I saw some brief info on this
> list's archives about the Dell/EMC offerings. Anything else
> in that vein to look at?
My two cents: SAN is a bad investment, go for big internal storage.
The 3Ware or Areca SATA RAID adapters kick butt and if you look in the
newest colos (I was just in ours "365main.net" today), you will see rack
on rack of machines with from 4 to 16 internal SATA drives. Are they
all DB servers? Not necessarily, but that's where things are headed.
You can get a 3U server with dual opteron 250s, 16GB RAM and 16x 400GB
SATAII drives with the 3Ware 9550SX controller for $10K - we just
ordered 4 of them. I don't think you can buy an external disk chassis
and a Fibre channel NIC for that.
Performance? 800MB/s RAID5 reads, 400MB/s RAID5 writes. Random IOs are
also very high for RAID10, but we don't use it so YMMV - look at Areca
and 3Ware.
Managability? Good web management interfaces with 6+ years of
development from 3Ware, e-mail, online rebuild options, all the goodies.
No "snapshot" or offline backup features like the high-end SANs, but do
you really need it?
Need more power or storage over time? Run a parallel DB like Bizgres
MPP, you can add more servers with internal storage and increase your
I/O, CPU and memory.
- Luke
---------------------------(end of broadcast)---------------------------
TIP 5: don't forget to increase your free space map settings
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: SAN/NAS options
@ 2005-12-20 01:24 Anjan Dave <adave@vantage.com>
0 siblings, 0 replies; 25+ messages in thread
From: Anjan Dave @ 2005-12-20 01:24 UTC (permalink / raw)
To: Matthew Schumacher <matt.s@aptalaska.net>; pgsql-performance
Usually manufacturer's claims are tested in 'ideal' conditions, it may not translate well on bandwidth seen on the host side. A 2Gbps Fiber Channel connection would (ideally) give you about 250MB/sec per HBA. Not sure how it translates for GigE considering scsi protocol overheads, but you may want to confirm from them how they achieved 370MB/sec (hwo many iSCSI controllers, what file system, how many drives, what RAID type, block size, strip size, cache settings, etc), and whether it was physical I/O or cached. In other words, if someone has any benchmark numbers, that would be helpful.
Regarding diskless iscsi boots for future servers, remember that it's a shared storage, if you have a busy server attached to your Nexsan, you may have to think twice on sharing the performance (throughput and IOPS of the storage controller) without impacting the existing hosts, unless you are zizing it now.
And you want to have a pretty clean GigE network, more or less dedicated to this block traffic.
Large internal storage with more memory and AMD CPUs is an option as Luke had originally suggested. Check out Appro as well.
I'd also be curious to know if someone has been using this (SATA/iSCSI/SAS) solution and what are some I/O numbers observed.
Thanks,
Anjan
-----Original Message-----
From: Matthew Schumacher [mailto:matt.s@aptalaska.net]
Sent: Mon 12/19/2005 7:41 PM
To: pgsql-performance@postgresql.org
Cc:
Subject: Re: [PERFORM] SAN/NAS options
Jim C. Nasby wrote:
> On Wed, Dec 14, 2005 at 01:56:10AM -0500, Charles Sprickman wrote:
> You'll note that I'm being somewhat driven by my OS of choice, FreeBSD.
>
>>Unlike Solaris or other commercial offerings, there is no nice volume
>>management available. While I'd love to keep managing a dozen or so
>>FreeBSD boxes, I could be persuaded to go to Solaris x86 if the volume
>>management really shines and Postgres performs well on it.
>
>
> Have you looked at vinum? It might not qualify as a true volume manager,
> but it's still pretty handy.
I am looking very closely at purchasing a SANRAD Vswitch 2000, a Nexsan
SATABoy with SATA disks, and the Qlogic iscsi controller cards.
Nexsan claims up to 370MB/s sustained per controller and 44,500 IOPS but
I'm not sure if that is good or bad. It's certainly faster than the LSI
megaraid controller I'm using now with a raid 1 mirror.
The sanrad box looks like it saves money in that you don't have to by
controller cards for everything, but for I/O intensive servers such as
the database server, I would end up buying an iscsi controller card anyway.
At this point I'm not sure what the best solution is. I like the idea
of having logical disks available though iscsi because of how flexible
it is, but I really don't want to spend $20k (10 for the nexsan and 10
for the sanrad) and end up with poor performance.
On other advantage to iscsi is that I can go completely diskless on my
servers and boot from iscsi which means that I don't have to have spare
disks for each host, now I just have spare disks for the nexsan chassis.
So the question becomes: has anyone put postgres on an iscsi san, and if
so how did it perform?
schu
---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?
http://archives.postgresql.org
^ permalink raw reply [nested|flat] 25+ messages in thread
end of thread, other threads:[~2006-01-18 18:54 UTC | newest]
Thread overview: 25+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2005-12-14 06:43 SAN/NAS options Charles Sprickman <spork@bway.net>
2005-12-21 05:58 ` Charles Sprickman <spork@bway.net>
2005-12-21 10:17 ` Alex Stapleton <alexs@advfn.com>
2005-12-21 10:46 ` Luke Lonergan <llonergan@greenplum.com>
2005-12-14 06:56 SAN/NAS options Charles Sprickman <spork@bway.net>
2005-12-14 07:28 ` Mark Kirkwood <markir@paradise.net.nz>
2005-12-16 22:18 ` Jim C. Nasby <jnasby@pervasive.com>
2005-12-16 22:49 ` Mark Kirkwood <markir@paradise.net.nz>
2005-12-16 22:51 ` Michael Stone <mstone+postgres@mathom.us>
2005-12-17 00:25 ` Jim C. Nasby <jnasby@pervasive.com>
2005-12-17 00:48 ` Michael Stone <mstone+postgres@mathom.us>
2005-12-14 16:53 ` Andrew Rawnsley <ronz@investoranalytics.com>
2005-12-14 18:32 ` Michael Stone <mstone+postgres@mathom.us>
2005-12-16 22:19 ` Jim C. Nasby <jnasby@pervasive.com>
2005-12-20 00:41 ` Matthew Schumacher <matt.s@aptalaska.net>
2006-01-15 02:37 ` Charles Sprickman <spork@bway.net>
2006-01-15 03:02 ` Luke Lonergan <llonergan@greenplum.com>
2006-01-15 03:23 ` Charles Sprickman <spork@bway.net>
2006-01-15 17:21 ` Luke Lonergan <llonergan@greenplum.com>
2006-01-15 04:04 ` Christopher Browne <cbbrowne@acm.org>
2006-01-18 18:54 ` Michael Adler <adler@pobox.com>
2005-12-14 07:10 Re: SAN/NAS options Luke Lonergan <LLonergan@greenplum.com>
2005-12-15 21:13 Re: SAN/NAS options Anjan Dave <adave@vantage.com>
2005-12-15 21:40 Re: SAN/NAS options Luke Lonergan <LLonergan@greenplum.com>
2005-12-20 01:24 Re: SAN/NAS options Anjan Dave <adave@vantage.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox