Date: Wed, 14 Dec 2005 20:28:56 +1300
From: Mark Kirkwood <markir@paradise.net.nz>
Subject: Re: SAN/NAS options
In-reply-to: <Pine.OSX.4.61.0512140155590.282@office-dhcp-18.bway.net>
To: Charles Sprickman <spork@bway.net>
Cc: pgsql-performance@postgresql.org
Message-id: <439FC9B8.8020709@paradise.net.nz>
MIME-version: 1.0
Content-type: text/plain; format=flowed; charset=ISO-8859-1
Content-transfer-encoding: 7bit
User-Agent: Mozilla Thunderbird 1.0.6 (X11/20051106)
References: <Pine.OSX.4.61.0512140155590.282@office-dhcp-18.bway.net>

Charles Sprickman wrote:
> Hello all,
> 
> It seems that I'm starting to outgrow our current Postgres setup.  We've 
> been running a handful of machines as standalone db servers.  This is 
> all in a colocation environment, so everything is stuffed into 1U 
> Supermicro boxes.  Our standard build looks like this:
> 
> Supermicro 1U w/SCA backplane and 4 bays
> 2x2.8 GHz Xeons
> Adaptec 2015S "zero channel" RAID card
> 2 or 4 x 73GB Seagate 10K Ultra 320 drives (mirrored+striped)
> 2GB RAM
> FreeBSD 4.11
> PGSQL data from 5-10GB per box
> 
> Recently I started studying what we were running up against in our 
> nightly runs that do a ton of updates/inserts to prep things for the 
> tasks the db does during the business day (light mix of 
> selects/inserts/updates). While we have plenty of disk bandwidth 
> (according to bonnie), we are really dying on IOPS. I'm guessing this is 
> a mix of a rather anemic RAID controller (ever notice how adaptec 
> doesn't publish any real performance specs on raid cards?) and having 
> only two or four spindles (effectively 1 or 2 on writes).
> 
> So that's where we are...
> 
> I'm new to the whole SAN thing, but did recently pick up a few used 
> NetApp shelves and a Fibre Channel RAID HBA (Mylex ExtremeRAID 3000, 
> also used) to toy with.  I started wondering if I could put something 
> together to both get our storage on one set of boxes and allow me to get 
> data striped across more drives.  Our budget is not huge and we are not 
> adverse to getting used gear where appropriate.
> 
> What do you folks recommend?  I'm just starting to look at what's out 
> there for SANs and NAS, and from what I've seen, our options are:
> 

Leaving the whole SAN issue for a moment:

It would be interesting to see if moving to FreeBSD 6.0 would help you - 
the vfs layer is no longer throttled by the (SMP) GIANT lock in this 
version, and that may make quite a difference (given you have SMP boxes).

Another interesting thing to try is rebuilding the database ufs 
filesystem(s) with 32K blocks and 4K frags (as opposed to 8K/1K or 
16K/2K - can't recall the default on 4.x). I found this to give a factor 
of 2 speedup on random disk access (specifically queries doing indexed 
joins).

Is it mainly your 2 disk machines that are IOPS bound? if so, a cheap 
option may be to buy 2 more cheetahs for them! If it's the 4's, well how 
about a 2U U320 diskpack from whomever supplies you the Supermicro boxes?

I have just noticed Luke's posting - I would second the advice to avoid 
SAN - in my experience it's an expensive way to buy storage.

best wishes

Mark