Subject: Re: Performance with 2 AMD/Opteron 2.6Ghz and 8gig
From: Mark Lewis <mark.lewis@mir3.com>
To: Jeff Trout <threshar@torgo.978.org>
Cc: Luke Lonergan <LLonergan@greenplum.com>,
	Mikael Carneholm <Mikael.Carneholm@WirelessCar.com>,
	Kjell Tore Fossbakk <kjelltore@gmail.com>, pgsql-performance@postgresql.org
In-Reply-To: <7A073C1C-BB04-42C7-A7B9-D25A18DB27D0@torgo.978.org>
References: <3E37B936B592014B978C4415F90D662D03E6CA59@MI8NYCMAIL06.Mi8.com>
	<7A073C1C-BB04-42C7-A7B9-D25A18DB27D0@torgo.978.org>
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Organization: MIR3, Inc.
Date: Fri, 28 Jul 2006 11:01:54 -0700
Message-Id: <1154109714.1634.609.camel@archimedes>
Mime-Version: 1.0

This isn't all that surprising.  The main weaknesses of RAID-5 are poor
write performance and stupid hardware controllers that make the write
performance even worse than it needs to be.  Your numbers bear that out.
Reads off RAID-5 are usually pretty good.

Your 'dd' test is going to be a little misleading though.  Most DB
access isn't usually purely sequential; while it's easy to see why HW
RAID-5 might outperform HW-RAID-10 in large sequential reads (the RAID
controller would need to be smarter than most to make RAID-10 as fast as
RAID-5), I would expect that HW RAID-5 and RAID-10 random reads would be
about equal or else maybe give a slight edge to RAID-10. 

-- Mark Lewis


On Fri, 2006-07-28 at 13:31 -0400, Jeff Trout wrote:
> I too have a DL385 with a single DC Opteron 270.
> It claims to have a smart array 6i controller and over the last  
> couple of days I've been runnign some tests on it, which have been  
> yielding some suprising results.
> 
> I've got 6 10k U320 disks in it. 2 are in a mirror set.  We'll not  
> pay any attention to them.
> The remaining 4 disks I've been toying with to see what config works  
> best, using hardware raid and software raid.
> 
> system info:
> dl dl385 - 1 opteron 270 - 5GB ram - smart array 6i
> cciss0: HP Smart Array 6i Controller
> Firmware Version: 2.58
> Linux db03 2.6.17-1.2157_FC5 #1 SMP Tue Jul 11 22:53:56 EDT 2006  
> x86_64 x86_64 x86_64 GNU/Linux
> using xfs
> 
> Each drive can sustain 80MB/sec read (dd, straight off device)
> 
> So here are the results I have so far.  (averaged)
> 
> 
> hardware raid 5:
> dd - write 20GB file - 48MB/sec
> dd - read 20GB file - 247MB/sec
> [ didn't do a bonnie run on this yet ]
> pretty terrible write performance. good read.
> 
> hardware raid 10
> dd - write 20GB - 104MB/sec
> dd - read 20GB - 196MB/sec
> bonnie++
> Version  1.03       ------Sequential Output------ --Sequential Input-  
> --Random-
>                      -Per Chr- --Block-- -Rewrite- -Per Chr- -- 
> Block-- --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec % 
> CP  /sec %CP
> db03          9592M 45830  97 129501  31 62981  14 48524  99 185818   
> 19 949.0   1
> 
> software raid 5
> dd - write 20gb - 85MB/sec
> dd - read 20gb - 135MB/sec
> 
> I was very suprised at those results. I was sort of expecting it to  
> smoke the hardware. I repeated the test many times, and kept getting
> these numbers.
> 
> bonnie++:
> Version  1.03       ------Sequential Output------ --Sequential Input-  
> --Random-
>                      -Per Chr- --Block-- -Rewrite- -Per Chr- -- 
> Block-- --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec % 
> CP  /sec %CP
> db03          9592M 44110  97 81481  23 34604  10 44495  95 157063   
> 28 919.3   1
> 
> software 10:
> dd - write - 20GB - 108MB/sec
> dd - read - 20GB - 86MB/sec(!!!! WTF? - this is repeatable!!)
> bonnie++
> Version  1.03       ------Sequential Output------ --Sequential Input-  
> --Random-
>                      -Per Chr- --Block-- -Rewrite- -Per Chr- -- 
> Block-- --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec % 
> CP  /sec %CP
> db03          9592M 44539  98 105444  20 34127   8 39830  83 100374   
> 10  1072   1
> 
> 
> so I'm going to be going with hw r5, which went against what I  
> thought going in - read perf is more important for my usage than write.
> 
> I'm still not sure about that software 10 read number. something is  
> not right there...
> 
> --
> Jeff Trout <jeff@jefftrout.com>
> http://www.dellsmartexitin.com/
> http://www.stuarthamm.net/
> 
> 
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 9: In versions below 8.0, the planner will ignore your desire to
>        choose an index scan if your joining column's datatypes do not
>        match