pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedRAID Controller (HP P400) beat by SW-RAID?
12+ messages / 10 participants
[nested] [flat]
* RAID Controller (HP P400) beat by SW-RAID?
@ 2011-09-11 22:44 Anthony Presley <anthony@resolution.com>
2011-09-11 22:51 ` Re: RAID Controller (HP P400) beat by SW-RAID? Stephen Frost <sfrost@snowman.net>
2011-09-11 23:17 ` Re: RAID Controller (HP P400) beat by SW-RAID? Tomas Vondra <tv@fuzzy.cz>
2011-09-12 02:12 ` Re: RAID Controller (HP P400) beat by SW-RAID? Alan Hodgson <ahodgson@simkin.ca>
2011-09-12 02:25 ` Re: RAID Controller (HP P400) beat by SW-RAID? Scott Marlowe <scott.marlowe@gmail.com>
2011-09-12 03:10 ` Re: RAID Controller (HP P400) beat by SW-RAID? mark <dvlhntr@gmail.com>
2011-09-13 06:22 ` Re: RAID Controller (HP P400) beat by SW-RAID? Arjen van der Meijden <acmmailing@tweakers.net>
2011-09-14 07:22 ` Re: RAID Controller (HP P400) beat by SW-RAID? Greg Smith <greg@2ndQuadrant.com>
0 siblings, 7 replies; 12+ messages in thread
From: Anthony Presley @ 2011-09-11 22:44 UTC (permalink / raw)
To: pgsql-performance
We've currently got PG 8.4.4 running on a whitebox hardware set up, with (2)
5410 Xeon's, and 16GB of RAM. It's also got (4) 7200RPM SATA drives, using
the onboard IDE controller and ext3.
A few weeks back, we purchased two refurb'd HP DL360's G5's, and were hoping
to set them up with PG 9.0.2, running replicated. These machines have (2)
5410 Xeon's, 36GB of RAM, (6) 10k SAS drives, and are using the HP SA P400i
with 512MB of BBWC. PG is running on an ext4 (noatime) partition, and they
drives configured as RAID 1+0 (seems with this controller, I cannot do
JBOD). I've spent a few hours going back and forth benchmarking the new
systems, and have set up the DWC, and the accelerator cache using hpacucli.
I've tried accelerator caches of 25/75, 50/50, and 75/25.
To start with, I've set the "relevant" parameters in postgresql.conf the
same on the new config as the old:
max_connections = 150
shared_buffers = 6400MB (have tried as high as 20GB)
work_mem = 20MB (have tried as high as 100MB)
effective_io_concurrency = 6
fsync = on
synchronous_commit = off
wal_buffers = 16MB
checkpoint_segments = 30 (have tried 200 when I was loading the db)
random_page_cost = 2.5
effective_cache_size = 10240MB (have tried as high as 16GB)
First thing I noticed is that it takes the same amount of time to load the
db (about 40 minutes) on the new hardware as the old hardware. I was really
hoping with the faster, additional drives and a hardware RAID controller,
that this would be faster. The database is only about 9GB with pg_dump
(about 28GB with indexes).
Using pgfouine I've identified about 10 "problematic" SELECT queries that
take anywhere from .1 seconds to 30 seconds on the old hardware. Running
these same queries on the new hardware is giving me results in the .2 to 66
seconds. IE, it's twice as slow.
I've tried increasing the shared_buffers, and some other parameters
(work_mem), but haven't yet seen the new hardware perform even at the same
speed as the old hardware.
I was really hoping that with hardware RAID that something would be faster
(loading times, queries, etc...). What am I doing wrong?
About the only thing left that I know to try is to drop the RAID1+0 and go
to RAID0 in hardware, and do RAID1 in software. Any other thoughts?
Thanks!
--
Anthony
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: RAID Controller (HP P400) beat by SW-RAID?
2011-09-11 22:44 RAID Controller (HP P400) beat by SW-RAID? Anthony Presley <anthony@resolution.com>
@ 2011-09-11 22:51 ` Stephen Frost <sfrost@snowman.net>
6 siblings, 0 replies; 12+ messages in thread
From: Stephen Frost @ 2011-09-11 22:51 UTC (permalink / raw)
To: Anthony Presley <anthony@resolution.com>; +Cc: pgsql-performance
* Anthony Presley (anthony@resolution.com) wrote:
> I was really hoping that with hardware RAID that something would be faster
> (loading times, queries, etc...). What am I doing wrong?
ext3 and ext4 do NOT perform identically out of the box.. You might be
running into the write barriers problem here with ext4 forcing the RAID
controllers to push commits all the way to the hard drive before
returning (thus making the BBWC next to useless).
You might try w/ ext3 on the new system instead.
Also, the p800's are definitely better than the p400's, but I don't know
that it's the controller that's really the issue here..
Thanks,
Stephen
Attachments:
signature.asc (application/pgp-signature, 197B) download
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: RAID Controller (HP P400) beat by SW-RAID?
2011-09-11 22:44 RAID Controller (HP P400) beat by SW-RAID? Anthony Presley <anthony@resolution.com>
@ 2011-09-11 23:17 ` Tomas Vondra <tv@fuzzy.cz>
2011-09-12 16:42 ` Re: RAID Controller (HP P400) beat by SW-RAID? Anthony Presley <anthony@resolution.com>
6 siblings, 1 reply; 12+ messages in thread
From: Tomas Vondra @ 2011-09-11 23:17 UTC (permalink / raw)
To: Anthony Presley <anthony@resolution.com>; +Cc: pgsql-performance
Dne 12.9.2011 00:44, Anthony Presley napsal(a):
> We've currently got PG 8.4.4 running on a whitebox hardware set up,
> with (2) 5410 Xeon's, and 16GB of RAM. It's also got (4) 7200RPM
> SATA drives, using the onboard IDE controller and ext3.
>
> A few weeks back, we purchased two refurb'd HP DL360's G5's, and
> were hoping to set them up with PG 9.0.2, running replicated. These
> machines have (2) 5410 Xeon's, 36GB of RAM, (6) 10k SAS drives, and
> are using the HP SA P400i with 512MB of BBWC. PG is running on an
> ext4 (noatime) partition, and they drives configured as RAID 1+0
> (seems with this controller, I cannot do JBOD). I've spent a few
> hours going back and forth benchmarking the new systems, and have set
> up the DWC, and the accelerator cache using hpacucli. I've tried
> accelerator caches of 25/75, 50/50, and 75/25.
Whas is an 'accelerator cache'? Is that the cache on the controller?
Then give 100% to the write cache - the read cache does not need to be
protected by the battery, the page cache at the OS level can do the same
service.
Provide more details about the ext3/ext4 - there are various data modes
(writeback, ordered, journal), various other settings (barriers, stripe
size, ...) that matter.
According to the benchmark I've done a few days back, the performance
difference between ext3 and ext4 is rather small, when comparing equally
configured file systems (i.e. data=journal vs. data=journal) etc.
With read-only workload (e.g. just SELECT statements), the config does
not matter (e.g. journal is just as fast as writeback).
See for example these comparisons
read-only workload: http://bit.ly/q04Tpg
read-write workload: http://bit.ly/qKgWgn
The ext4 is usually a bit faster than equally configured ext3, but the
difference should not be 100%.
> To start with, I've set the "relevant" parameters in postgresql.conf
> the same on the new config as the old:
>
> max_connections = 150 shared_buffers = 6400MB (have tried as high as
> 20GB) work_mem = 20MB (have tried as high as 100MB)
> effective_io_concurrency = 6 fsync = on synchronous_commit = off
> wal_buffers = 16MB checkpoint_segments = 30 (have tried 200 when I
> was loading the db) random_page_cost = 2.5 effective_cache_size =
> 10240MB (have tried as high as 16GB)
>
> First thing I noticed is that it takes the same amount of time to
> load the db (about 40 minutes) on the new hardware as the old
> hardware. I was really hoping with the faster, additional drives and
> a hardware RAID controller, that this would be faster. The database
> is only about 9GB with pg_dump (about 28GB with indexes).
>
> Using pgfouine I've identified about 10 "problematic" SELECT queries
> that take anywhere from .1 seconds to 30 seconds on the old
> hardware. Running these same queries on the new hardware is giving me
> results in the .2 to 66 seconds. IE, it's twice as slow.
>
> I've tried increasing the shared_buffers, and some other parameters
> (work_mem), but haven't yet seen the new hardware perform even at
> the same speed as the old hardware.
In that case some of the assumptions is wrong. For example the new RAID
is slow for some reason. Bad stripe size, slow controller, ...
Do the basic hw benchmarking, i.e. use bonnie++ to benchmark the disk,
etc. Only if this provides expected results (i.e. the new hw performs
better) it makes sense to mess with the database.
Tomas
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: RAID Controller (HP P400) beat by SW-RAID?
2011-09-11 22:44 RAID Controller (HP P400) beat by SW-RAID? Anthony Presley <anthony@resolution.com>
2011-09-11 23:17 ` Re: RAID Controller (HP P400) beat by SW-RAID? Tomas Vondra <tv@fuzzy.cz>
@ 2011-09-12 16:42 ` Anthony Presley <anthony@resolution.com>
0 siblings, 0 replies; 12+ messages in thread
From: Anthony Presley @ 2011-09-12 16:42 UTC (permalink / raw)
To: Tomas Vondra <tv@fuzzy.cz>; +Cc: pgsql-performance
On Sun, Sep 11, 2011 at 6:17 PM, Tomas Vondra <tv@fuzzy.cz> wrote:
> Dne 12.9.2011 00:44, Anthony Presley napsal(a):
> > We've currently got PG 8.4.4 running on a whitebox hardware set up,
> > with (2) 5410 Xeon's, and 16GB of RAM. It's also got (4) 7200RPM
> > SATA drives, using the onboard IDE controller and ext3.
> >
> > A few weeks back, we purchased two refurb'd HP DL360's G5's, and
> > were hoping to set them up with PG 9.0.2, running replicated. These
> > machines have (2) 5410 Xeon's, 36GB of RAM, (6) 10k SAS drives, and
> > are using the HP SA P400i with 512MB of BBWC. PG is running on an
> > ext4 (noatime) partition, and they drives configured as RAID 1+0
> > (seems with this controller, I cannot do JBOD). I've spent a few
> > hours going back and forth benchmarking the new systems, and have set
> > up the DWC, and the accelerator cache using hpacucli. I've tried
> > accelerator caches of 25/75, 50/50, and 75/25.
>
> Whas is an 'accelerator cache'? Is that the cache on the controller?
> Then give 100% to the write cache - the read cache does not need to be
> protected by the battery, the page cache at the OS level can do the same
> service.
>
It is the cache on the controller. I've tried giving 100% to that cache.
> Provide more details about the ext3/ext4 - there are various data modes
> (writeback, ordered, journal), various other settings (barriers, stripe
> size, ...) that matter.
>
ext3 (on the old server) is using CentOS 5.2 defaults for mounting.
ext4 (on the new server) is using noatime,barrier=0
> According to the benchmark I've done a few days back, the performance
> difference between ext3 and ext4 is rather small, when comparing equally
> configured file systems (i.e. data=journal vs. data=journal) etc.
>
> With read-only workload (e.g. just SELECT statements), the config does
> not matter (e.g. journal is just as fast as writeback).
>
> See for example these comparisons
>
> read-only workload: http://bit.ly/q04Tpg
> read-write workload: http://bit.ly/qKgWgn
>
> The ext4 is usually a bit faster than equally configured ext3, but the
> difference should not be 100%.
>
Yes - it's very strange.
> > To start with, I've set the "relevant" parameters in postgresql.conf
> > the same on the new config as the old:
> >
> > max_connections = 150 shared_buffers = 6400MB (have tried as high as
> > 20GB) work_mem = 20MB (have tried as high as 100MB)
> > effective_io_concurrency = 6 fsync = on synchronous_commit = off
> > wal_buffers = 16MB checkpoint_segments = 30 (have tried 200 when I
> > was loading the db) random_page_cost = 2.5 effective_cache_size =
> > 10240MB (have tried as high as 16GB)
> >
> > First thing I noticed is that it takes the same amount of time to
> > load the db (about 40 minutes) on the new hardware as the old
> > hardware. I was really hoping with the faster, additional drives and
> > a hardware RAID controller, that this would be faster. The database
> > is only about 9GB with pg_dump (about 28GB with indexes).
> >
> > Using pgfouine I've identified about 10 "problematic" SELECT queries
> > that take anywhere from .1 seconds to 30 seconds on the old
> > hardware. Running these same queries on the new hardware is giving me
> > results in the .2 to 66 seconds. IE, it's twice as slow.
> >
> > I've tried increasing the shared_buffers, and some other parameters
> > (work_mem), but haven't yet seen the new hardware perform even at
> > the same speed as the old hardware.
>
> In that case some of the assumptions is wrong. For example the new RAID
> is slow for some reason. Bad stripe size, slow controller, ...
>
> Do the basic hw benchmarking, i.e. use bonnie++ to benchmark the disk,
> etc. Only if this provides expected results (i.e. the new hw performs
> better) it makes sense to mess with the database.
>
> Tomas
>
--
Anthony Presley
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: RAID Controller (HP P400) beat by SW-RAID?
2011-09-11 22:44 RAID Controller (HP P400) beat by SW-RAID? Anthony Presley <anthony@resolution.com>
@ 2011-09-12 02:12 ` Alan Hodgson <ahodgson@simkin.ca>
6 siblings, 0 replies; 12+ messages in thread
From: Alan Hodgson @ 2011-09-12 02:12 UTC (permalink / raw)
To: pgsql-performance
On September 11, 2011 03:44:34 PM Anthony Presley wrote:
> First thing I noticed is that it takes the same amount of time to load the
> db (about 40 minutes) on the new hardware as the old hardware. I was
> really hoping with the faster, additional drives and a hardware RAID
> controller, that this would be faster. The database is only about 9GB
> with pg_dump (about 28GB with indexes).
Loading the DB is going to be CPU-bound (on a single) core, unless your disks
really suck, which they don't. Most of the time will be spent building
indexes.
I don't know offhand why the queries are slower, though, unless you're not
getting as much cached before testing as on the older box.
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: RAID Controller (HP P400) beat by SW-RAID?
2011-09-11 22:44 RAID Controller (HP P400) beat by SW-RAID? Anthony Presley <anthony@resolution.com>
@ 2011-09-12 02:25 ` Scott Marlowe <scott.marlowe@gmail.com>
6 siblings, 0 replies; 12+ messages in thread
From: Scott Marlowe @ 2011-09-12 02:25 UTC (permalink / raw)
To: Anthony Presley <anthony@resolution.com>; +Cc: pgsql-performance
On Sun, Sep 11, 2011 at 4:44 PM, Anthony Presley <anthony@resolution.com> wrote:
> We've currently got PG 8.4.4 running on a whitebox hardware set up, with (2)
> 5410 Xeon's, and 16GB of RAM. It's also got (4) 7200RPM SATA drives, using
> the onboard IDE controller and ext3.
> A few weeks back, we purchased two refurb'd HP DL360's G5's, and were hoping
> to set them up with PG 9.0.2, running replicated. These machines have (2)
> 5410 Xeon's, 36GB of RAM, (6) 10k SAS drives, and are using the HP SA P400i
> with 512MB of BBWC. PG is running on an ext4 (noatime) partition, and they
Two issues here. One is that the onboard controller and disks on the
old machine might not be obeying fsync properly, giving a speed boost
at the expense of crash safeness. Two is that the P400 has gotten
pretty horrible performance reviews on this list in the past.
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: RAID Controller (HP P400) beat by SW-RAID?
2011-09-11 22:44 RAID Controller (HP P400) beat by SW-RAID? Anthony Presley <anthony@resolution.com>
@ 2011-09-12 03:10 ` mark <dvlhntr@gmail.com>
2011-09-12 03:31 ` Re: RAID Controller (HP P400) beat by SW-RAID? Gavin Flower <GavinFlower@archidevsys.co.nz>
2011-09-12 16:56 ` Re: RAID Controller (HP P400) beat by SW-RAID? Anthony Presley <anthony@resolution.com>
6 siblings, 2 replies; 12+ messages in thread
From: mark @ 2011-09-12 03:10 UTC (permalink / raw)
To: 'Anthony Presley' <anthony@resolution.com>; +Cc: pgsql-performance
>From: pgsql-performance-owner@postgresql.org
[mailto:pgsql-performance-owner@postgresql.org] On Behalf Of Anthony Presley
>Sent: Sunday, September 11, 2011 4:45 PM
>To: pgsql-performance@postgresql.org
>Subject: [PERFORM] RAID Controller (HP P400) beat by SW-RAID?
>We've currently got PG 8.4.4 running on a whitebox hardware set up, with
(2) 5410 Xeon's, and 16GB of RAM. It's also got (4) 7200RPM SATA drives,
using the onboard IDE controller and ext3.
>A few weeks back, we purchased two refurb'd HP DL360's G5's, and were
hoping to set them up with PG 9.0.2, running replicated. These machines
have (2) 5410 Xeon's, 36GB of RAM, (6) 10k SAS drives, and are using the HP
SA P400i with 512MB of BBWC. PG is running on an ext4 (noatime) partition,
and they drives configured as RAID 1+0 (seems with this controller, I cannot
do JBOD). I've spent a few hours going back and forth benchmarking the new
systems, and have set up the DWC, and the accelerator cache using hpacucli.
I've tried accelerator caches of 25/75, 50/50, and 75/25.
>
I would start of by recommending a more current version of 9.0...like 9.0.4
since you are building a new box. The rumor mill says 9.0.5 and 9.1.0 might
be out soon (days?). but that is just rumor mill. Don't bank on it.
What kernel are you on ?
Long time HP user here, for better and worse... so here are a few other
little things I recommend.
Check the bios power management. Make sure it is set where you want it.
(IIRC the G5s have this, I know G6s and G7s do). This can help with nasty
latency problems if the box has been idle for a while then needs to start
doing work.
The p400i is not a great card, compared to more modern one, but you should
be able to beat the old setup with what you have. Faster clocked cpu's more
spindles, faster RPM spindles.
Assuming the battery is working, with XFS or ext4 you can use nobarrier
mount option and you should see some improvement.
Make sure the raid card's firmware is current. I can't stress this enough.
HP fixed a nasty bug with Raid 1+0 a few months ago where you could eat your
data... They also seem to be fixing a lot of other bugs along the way as
well. So do yourself a big favor and make sure that firmware is current. It
might just head off headache down the road.
Also make sure you have a 8.10.? (IIRC the version number right) or better
version of hpacucli... there have been some fixes to that utility as well.
IIRC most of the fixes in this have been around recognizing newere cards
(812s and 410s) but some interface bugs have been fixed as well. You may
need new packages for HP health. (I don't recall the official name, but new
versions if hpacucli might not play well with old versions of hp health.
Its HP so they have a new version about every month for firmware and their
cli utility... that’s HP for us.
Anyways that is my fast input.
Best of luck,
-Mark
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: RAID Controller (HP P400) beat by SW-RAID?
2011-09-11 22:44 RAID Controller (HP P400) beat by SW-RAID? Anthony Presley <anthony@resolution.com>
2011-09-12 03:10 ` Re: RAID Controller (HP P400) beat by SW-RAID? mark <dvlhntr@gmail.com>
@ 2011-09-12 03:31 ` Gavin Flower <GavinFlower@archidevsys.co.nz>
1 sibling, 0 replies; 12+ messages in thread
From: Gavin Flower @ 2011-09-12 03:31 UTC (permalink / raw)
To: mark <dvlhntr@gmail.com>; +Cc: 'Anthony Presley' <anthony@resolution.com>; pgsql-performance
On 12/09/11 15:10, mark wrote:
>
>> From: pgsql-performance-owner@postgresql.org
> [mailto:pgsql-performance-owner@postgresql.org] On Behalf Of Anthony Presley
>> Sent: Sunday, September 11, 2011 4:45 PM
>> To: pgsql-performance@postgresql.org
>> Subject: [PERFORM] RAID Controller (HP P400) beat by SW-RAID?
>> We've currently got PG 8.4.4 running on a whitebox hardware set up, with
> (2) 5410 Xeon's, and 16GB of RAM. It's also got (4) 7200RPM SATA drives,
> using the onboard IDE controller and ext3.
>
>> A few weeks back, we purchased two refurb'd HP DL360's G5's, and were
> hoping to set them up with PG 9.0.2, running replicated. These machines
> have (2) 5410 Xeon's, 36GB of RAM, (6) 10k SAS drives, and are using the HP
> SA P400i with 512MB of BBWC. PG is running on an ext4 (noatime) partition,
> and they drives configured as RAID 1+0 (seems with this controller, I cannot
> do JBOD). I've spent a few hours going back and forth benchmarking the new
> systems, and have set up the DWC, and the accelerator cache using hpacucli.
> I've tried accelerator caches of 25/75, 50/50, and 75/25.
>
> I would start of by recommending a more current version of 9.0...like 9.0.4
> since you are building a new box. The rumor mill says 9.0.5 and 9.1.0 might
> be out soon (days?). but that is just rumor mill. Don't bank on it.
>
>
> What kernel are you on ?
>
> Long time HP user here, for better and worse... so here are a few other
> little things I recommend.
>
> Check the bios power management. Make sure it is set where you want it.
> (IIRC the G5s have this, I know G6s and G7s do). This can help with nasty
> latency problems if the box has been idle for a while then needs to start
> doing work.
>
> The p400i is not a great card, compared to more modern one, but you should
> be able to beat the old setup with what you have. Faster clocked cpu's more
> spindles, faster RPM spindles.
>
> Assuming the battery is working, with XFS or ext4 you can use nobarrier
> mount option and you should see some improvement.
>
>
> Make sure the raid card's firmware is current. I can't stress this enough.
> HP fixed a nasty bug with Raid 1+0 a few months ago where you could eat your
> data... They also seem to be fixing a lot of other bugs along the way as
> well. So do yourself a big favor and make sure that firmware is current. It
> might just head off headache down the road.
>
> Also make sure you have a 8.10.? (IIRC the version number right) or better
> version of hpacucli... there have been some fixes to that utility as well.
> IIRC most of the fixes in this have been around recognizing newere cards
> (812s and 410s) but some interface bugs have been fixed as well. You may
> need new packages for HP health. (I don't recall the official name, but new
> versions if hpacucli might not play well with old versions of hp health.
>
> Its HP so they have a new version about every month for firmware and their
> cli utility... that’s HP for us.
>
> Anyways that is my fast input.
>
> Best of luck,
>
>
> -Mark
>
>
pg 9.1.0 has already been released!
I have had it installed and running for just under 24 hours...
though http://www.postgresql.org/ is still not showing it,
see:
http://www.postgresql.org/ftp/source/
and
http://jdbc.postgresql.org/download.html
Cheers,
Gavin
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: RAID Controller (HP P400) beat by SW-RAID?
2011-09-11 22:44 RAID Controller (HP P400) beat by SW-RAID? Anthony Presley <anthony@resolution.com>
2011-09-12 03:10 ` Re: RAID Controller (HP P400) beat by SW-RAID? mark <dvlhntr@gmail.com>
@ 2011-09-12 16:56 ` Anthony Presley <anthony@resolution.com>
1 sibling, 0 replies; 12+ messages in thread
From: Anthony Presley @ 2011-09-12 16:56 UTC (permalink / raw)
To: mark <dvlhntr@gmail.com>; +Cc: pgsql-performance
Mark,
On Sun, Sep 11, 2011 at 10:10 PM, mark <dvlhntr@gmail.com> wrote:
>
>
> >From: pgsql-performance-owner@postgresql.org
> [mailto:pgsql-performance-owner@postgresql.org] On Behalf Of Anthony
> Presley
> >Sent: Sunday, September 11, 2011 4:45 PM
> >To: pgsql-performance@postgresql.org
> >Subject: [PERFORM] RAID Controller (HP P400) beat by SW-RAID?
>
> >We've currently got PG 8.4.4 running on a whitebox hardware set up, with
> (2) 5410 Xeon's, and 16GB of RAM. It's also got (4) 7200RPM SATA drives,
> using the onboard IDE controller and ext3.
>
> >A few weeks back, we purchased two refurb'd HP DL360's G5's, and were
> hoping to set them up with PG 9.0.2, running replicated. These machines
> have (2) 5410 Xeon's, 36GB of RAM, (6) 10k SAS drives, and are using the HP
> SA P400i with 512MB of BBWC. PG is running on an ext4 (noatime) partition,
> and they drives configured as RAID 1+0 (seems with this controller, I
> cannot
> do JBOD). I've spent a few hours going back and forth benchmarking the new
> systems, and have set up the DWC, and the accelerator cache using hpacucli.
> I've tried accelerator caches of 25/75, 50/50, and 75/25.
> >
>
>
> I would start of by recommending a more current version of 9.0...like 9.0.4
> since you are building a new box. The rumor mill says 9.0.5 and 9.1.0 might
> be out soon (days?). but that is just rumor mill. Don't bank on it.
>
Looks like 9.1 was released today - I may upgrade to that for our testing.
I was just using whatever is in the repo.
> What kernel are you on ?
>
2.6.18-238.19.1.el5
> Long time HP user here, for better and worse... so here are a few other
> little things I recommend.
>
Thanks!
> Check the bios power management. Make sure it is set where you want it.
> (IIRC the G5s have this, I know G6s and G7s do). This can help with nasty
> latency problems if the box has been idle for a while then needs to start
> doing work.
>
I've checked those, they look ok.
> The p400i is not a great card, compared to more modern one, but you should
> be able to beat the old setup with what you have. Faster clocked cpu's more
> spindles, faster RPM spindles.
>
I've upgraded the CPU's to be X5470 today, to see if that helps with the
speed of
> Assuming the battery is working, with XFS or ext4 you can use nobarrier
> mount option and you should see some improvement.
>
I've been using:
noatime,data=writeback,defaults
I will try:
noatime,data=writeback,barrier=0,defaults
> Make sure the raid card's firmware is current. I can't stress this enough.
> HP fixed a nasty bug with Raid 1+0 a few months ago where you could eat
> your
> data... They also seem to be fixing a lot of other bugs along the way as
> well. So do yourself a big favor and make sure that firmware is current. It
> might just head off headache down the road.
>
I downloaded the latest firmware DVD on Thursday and ran that - everything
is up to date.
> Also make sure you have a 8.10.? (IIRC the version number right) or better
> version of hpacucli... there have been some fixes to that utility as well.
> IIRC most of the fixes in this have been around recognizing newere cards
> (812s and 410s) but some interface bugs have been fixed as well. You may
> need new packages for HP health. (I don't recall the official name, but new
> versions if hpacucli might not play well with old versions of hp health.
>
I got that as well - thanks!
> Its HP so they have a new version about every month for firmware and their
> cli utility... that’s HP for us.
>
> Anyways that is my fast input.
>
> Best of luck,
Thanks!
--
Anthony Presley
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: RAID Controller (HP P400) beat by SW-RAID?
2011-09-11 22:44 RAID Controller (HP P400) beat by SW-RAID? Anthony Presley <anthony@resolution.com>
@ 2011-09-13 06:22 ` Arjen van der Meijden <acmmailing@tweakers.net>
2011-09-13 11:33 ` Re: RAID Controller (HP P400) beat by SW-RAID? Anthony Presley <anthony@timeforge.com>
6 siblings, 1 reply; 12+ messages in thread
From: Arjen van der Meijden @ 2011-09-13 06:22 UTC (permalink / raw)
To: pgsql-performance
On 12-9-2011 0:44 Anthony Presley wrote:
> A few weeks back, we purchased two refurb'd HP DL360's G5's, and were
> hoping to set them up with PG 9.0.2, running replicated. These machines
> have (2) 5410 Xeon's, 36GB of RAM, (6) 10k SAS drives, and are using the
> HP SA P400i with 512MB of BBWC. PG is running on an ext4 (noatime)
> partition, and they drives configured as RAID 1+0 (seems with this
> controller, I cannot do JBOD).
If you really want a JBOD-setup, you can try a RAID0 for each available
disk, i.e. in your case 6 separate RAID0's. That's how we configured our
Dell H700 - which doesn't offer JBOD as well - for ZFS.
Best regards,
Arjen
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: RAID Controller (HP P400) beat by SW-RAID?
2011-09-11 22:44 RAID Controller (HP P400) beat by SW-RAID? Anthony Presley <anthony@resolution.com>
2011-09-13 06:22 ` Re: RAID Controller (HP P400) beat by SW-RAID? Arjen van der Meijden <acmmailing@tweakers.net>
@ 2011-09-13 11:33 ` Anthony Presley <anthony@timeforge.com>
0 siblings, 0 replies; 12+ messages in thread
From: Anthony Presley @ 2011-09-13 11:33 UTC (permalink / raw)
To: Arjen van der Meijden <acmmailing@tweakers.net>; +Cc: pgsql-performance
On Tue, Sep 13, 2011 at 1:22 AM, Arjen van der Meijden <
acmmailing@tweakers.net> wrote:
>
> On 12-9-2011 0:44 Anthony Presley wrote:
>
>> A few weeks back, we purchased two refurb'd HP DL360's G5's, and were
>> hoping to set them up with PG 9.0.2, running replicated. These machines
>> have (2) 5410 Xeon's, 36GB of RAM, (6) 10k SAS drives, and are using the
>> HP SA P400i with 512MB of BBWC. PG is running on an ext4 (noatime)
>> partition, and they drives configured as RAID 1+0 (seems with this
>> controller, I cannot do JBOD).
>>
>
> If you really want a JBOD-setup, you can try a RAID0 for each available
> disk, i.e. in your case 6 separate RAID0's. That's how we configured our
> Dell H700 - which doesn't offer JBOD as well - for ZFS.
>
That's a pretty good idea ... I'll try that on our second server today. In
the meantime, after tweaking it a bit, we were able to get (with iozone):
Old New Initial write
75.85 220.68 Rewrite
63.95 253.07 Read
45.04 171.35 Re-read
45 2405.23 Random read
27.56 1733.46 Random write
50.7 239.47
Not as fas as I'd like, but faster than the old disks, for sure.
--
Anthony
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: RAID Controller (HP P400) beat by SW-RAID?
2011-09-11 22:44 RAID Controller (HP P400) beat by SW-RAID? Anthony Presley <anthony@resolution.com>
@ 2011-09-14 07:22 ` Greg Smith <greg@2ndQuadrant.com>
6 siblings, 0 replies; 12+ messages in thread
From: Greg Smith @ 2011-09-14 07:22 UTC (permalink / raw)
To: pgsql-performance
On 09/11/2011 06:44 PM, Anthony Presley wrote:
> We've currently got PG 8.4.4 running on a whitebox hardware set up,
> with (2) 5410 Xeon's, and 16GB of RAM. It's also got (4) 7200RPM SATA
> drives, using the onboard IDE controller and ext3.
>
> A few weeks back, we purchased two refurb'd HP DL360's G5's, and were
> hoping to set them up with PG 9.0.2, running replicated. These
> machines have (2) 5410 Xeon's, 36GB of RAM, (6) 10k SAS drives, and
> are using the HP SA P400i with 512MB of BBWC. PG is running on an
> ext4 (noatime) partition, and they drives configured as RAID 1+0
> (seems with this controller, I cannot do JBOD). .
> To start with, I've set the "relevant" parameters in postgresql.conf
> the same on the new config as the old:
>
> fsync = on
> synchronous_commit = off
The main thing that a hardware RAID controller improves on is being able
to write synchronous commits much faster than you can do without one.
If you've turned that off, you've essentially neutralized its primary
value. In every other respect, software RAID is faster: the CPUs in
your server are much faster than the IO processor on the card, and Linux
has a lot more memory for caching than it does too. Turning off sync
commit may be fine for loading, but you'll be facing data loss at every
server interruption if you roll things out like that. It's not
realistic production performance for most places running like that.
A lot of your test results seem like they may be using different levels
of write reliability, which makes things less fair than they should be
too--in favor of the cheap IDE drives normally. Check out
http://wiki.postgresql.org/wiki/Reliable_Writes for more information
about that topic.
--
Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us
^ permalink raw reply [nested|flat] 12+ messages in thread
end of thread, other threads:[~2011-09-14 07:22 UTC | newest]
Thread overview: 12+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2011-09-11 22:44 RAID Controller (HP P400) beat by SW-RAID? Anthony Presley <anthony@resolution.com>
2011-09-11 22:51 ` Stephen Frost <sfrost@snowman.net>
2011-09-11 23:17 ` Tomas Vondra <tv@fuzzy.cz>
2011-09-12 16:42 ` Anthony Presley <anthony@resolution.com>
2011-09-12 02:12 ` Alan Hodgson <ahodgson@simkin.ca>
2011-09-12 02:25 ` Scott Marlowe <scott.marlowe@gmail.com>
2011-09-12 03:10 ` mark <dvlhntr@gmail.com>
2011-09-12 03:31 ` Gavin Flower <GavinFlower@archidevsys.co.nz>
2011-09-12 16:56 ` Anthony Presley <anthony@resolution.com>
2011-09-13 06:22 ` Arjen van der Meijden <acmmailing@tweakers.net>
2011-09-13 11:33 ` Anthony Presley <anthony@timeforge.com>
2011-09-14 07:22 ` Greg Smith <greg@2ndQuadrant.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox