pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedraid10 write performance
12+ messages / 10 participants
[nested] [flat]
* raid10 write performance
@ 2010-06-22 09:31 Grzegorz Jaśkiewicz <gryzman@gmail.com>
0 siblings, 2 replies; 12+ messages in thread
From: Grzegorz Jaśkiewicz @ 2010-06-22 09:31 UTC (permalink / raw)
To: pgsql-performance
Hi folks,
is there a general problem with raid10 performance postgresql on it?
We see very low performance on writes (2-3x slower than on less
performant servers). I wonder if it is solely problem of raid10
configuration, or if it is postgresql's thing.
Would moving WAL dir to separate disk help potentially ?
We're running centos 5.4, and server config is:
x3550 M2, xeon 4c e5530 2.4ghz , 6GB of ram
disks: ibm 300gb 2.5 SAS
raid: serveRAID M5014 SAS/SATA controller
strip size is the default 128k
thanks.
--
GJ
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: raid10 write performance
@ 2010-06-22 13:40 Justin Graf <justin@magwerks.com>
parent: Grzegorz Jaśkiewicz <gryzman@gmail.com>
1 sibling, 1 reply; 12+ messages in thread
From: Justin Graf @ 2010-06-22 13:40 UTC (permalink / raw)
To: =?utf-8?Q?_Grzegorz_Ja=C5=9Bkiewicz_ <gryzman@gmail.com>; +Cc: pgsql-performance
On 6/22/2010 4:31 AM, Grzegorz Jaśkiewicz wrote:
> Hi folks,
>
> is there a general problem with raid10 performance postgresql on it?
> We see very low performance on writes (2-3x slower than on less
> performant servers). I wonder if it is solely problem of raid10
> configuration, or if it is postgresql's thing.
>
RAID 10 is the commonly suggested layout for DB's as its performance to
redundancy is good.
The question that begs to be ask is what is the IO layout on the other
servers your comparing against.
> Would moving WAL dir to separate disk help potentially ?
>
Yes it can have a big impact.
http://www.westnet.com/~gsmith/content/postgresql/TuningPGWAL.htm
http://wiki.postgresql.org/wiki/Performance_Analysis_Tools
http://wiki.postgresql.org/wiki/Performance_Optimization
> We're running centos 5.4, and server config is:
>
> x3550 M2, xeon 4c e5530 2.4ghz , 6GB of ram
> disks: ibm 300gb 2.5 SAS
>
> raid: serveRAID M5014 SAS/SATA controller
>
> strip size is the default 128k
>
>
> thanks.
>
>
All legitimate Magwerks Corporation quotations are sent in a .PDF file attachment with a unique ID number generated by our proprietary quotation system. Quotations received via any other form of communication will not be honored.
CONFIDENTIALITY NOTICE: This e-mail, including attachments, may contain legally privileged, confidential or other information proprietary to Magwerks Corporation and is intended solely for the use of the individual to whom it addresses. If the reader of this e-mail is not the intended recipient or authorized agent, the reader is hereby notified that any unauthorized viewing, dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender by replying to this message and destroy all occurrences of this e-mail immediately.
Thank you.
Attachments:
justin.vcf (text/x-vcard, 271B) download
begin:vcard
fn:Justin Graf
n:Graf;Justin
org:Magwerks Corp
adr:;;501 Commerce Drive;Danville ;IN;46122;USA
email;internet:justin@magwerks.com
tel;work:317-241-8011 ext 703
tel;fax:317-241-8015
x-mozilla-html:FALSE
url:www.magwerks.com
version:2.1
end:vcard
* Re: raid10 write performance
@ 2010-06-22 14:29 Karl Denninger <karl@denninger.net>
parent: Justin Graf <justin@magwerks.com>
0 siblings, 2 replies; 12+ messages in thread
From: Karl Denninger @ 2010-06-22 14:29 UTC (permalink / raw)
To: Justin Graf <justin@magwerks.com>; +Cc: Grzegorz Jaśkiewicz <gryzman@gmail.com>; pgsql-performance
Justin Graf wrote:
> On 6/22/2010 4:31 AM, Grzegorz Jaśkiewicz wrote:
>
>> Would moving WAL dir to separate disk help potentially ?
>>
>>
>
> Yes it can have a big impact.
WAL on a separate spindle will make a HUGE difference in performance.
TPS rates frequently double OR BETTER with WAL on a dedicated spindle.
Strongly recommended.
Be aware that you must pay CLOSE ATTENTION to your backup strategy if
WAL is on a different physical disk. Snapshotting the data disk where
WAL is on a separate spindle and backing it up **WILL NOT WORK** and
**WILL** result in an non-restoreable backup.
The manual discusses this but it's easy to miss.... don't or you'll get
a NASTY surprise if something goes wrong.....
-- Karl
Attachments:
karl.vcf (text/x-vcard, 132B) download
begin:vcard
fn:Karl Denninger
n:Denninger;Karl
email;internet:karl@denninger.net
x-mozilla-html:TRUE
version:2.1
end:vcard
* Re: raid10 write performance
@ 2010-06-22 14:40 Greg Smith <greg@2ndquadrant.com>
parent: Grzegorz Jaśkiewicz <gryzman@gmail.com>
1 sibling, 1 reply; 12+ messages in thread
From: Greg Smith @ 2010-06-22 14:40 UTC (permalink / raw)
To: Grzegorz Jaśkiewicz <gryzman@gmail.com>; +Cc: pgsql-performance
Grzegorz Jaśkiewicz wrote:
> raid: serveRAID M5014 SAS/SATA controller
>
Do the "performant servers" have a different RAID card? This one has
terrible performance, and could alone be the source of your issue. The
ServeRAID cards are slow in general, and certainly slow running RAID10.
--
Greg Smith 2ndQuadrant US Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.us
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: raid10 write performance
@ 2010-06-22 16:01 Dave Crooke <dcrooke@gmail.com>
parent: Karl Denninger <karl@denninger.net>
1 sibling, 0 replies; 12+ messages in thread
From: Dave Crooke @ 2010-06-22 16:01 UTC (permalink / raw)
To: Karl Denninger <karl@denninger.net>; +Cc: Justin Graf <justin@magwerks.com>; Grzegorz Jaśkiewicz <gryzman@gmail.com>; pgsql-performance
Of course, no backup strategy is complete without testing a full restore
onto bare hardware :-)
On Tue, Jun 22, 2010 at 9:29 AM, Karl Denninger <karl@denninger.net> wrote:
> Justin Graf wrote:
>
> On 6/22/2010 4:31 AM, Grzegorz Jaśkiewicz wrote:
>
>
> Would moving WAL dir to separate disk help potentially ?
>
>
>
> Yes it can have a big impact.
>
> WAL on a separate spindle will make a HUGE difference in performance. TPS
> rates frequently double OR BETTER with WAL on a dedicated spindle.
>
> Strongly recommended.
>
> Be aware that you must pay CLOSE ATTENTION to your backup strategy if WAL
> is on a different physical disk. Snapshotting the data disk where WAL is on
> a separate spindle and backing it up **WILL NOT WORK** and **WILL** result
> in an non-restoreable backup.
>
> The manual discusses this but it's easy to miss.... don't or you'll get a
> NASTY surprise if something goes wrong.....
>
> -- Karl
>
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>
>
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: raid10 write performance
@ 2010-06-22 17:19 Scott Carey <scott@richrelevance.com>
parent: Karl Denninger <karl@denninger.net>
1 sibling, 0 replies; 12+ messages in thread
From: Scott Carey @ 2010-06-22 17:19 UTC (permalink / raw)
To: Karl Denninger <karl@denninger.net>; +Cc: Justin Graf <justin@magwerks.com>; Grzegorz Jaśkiewicz <gryzman@gmail.com>; pgsql-performance
On Jun 22, 2010, at 7:29 AM, Karl Denninger wrote:
> Justin Graf wrote:
>>
>> On 6/22/2010 4:31 AM, Grzegorz Jaśkiewicz wrote:
>>
>>> Would moving WAL dir to separate disk help potentially ?
>>>
>>>
>>
>> Yes it can have a big impact.
> WAL on a separate spindle will make a HUGE difference in performance. TPS rates frequently double OR BETTER with WAL on a dedicated spindle.
>
> Strongly recommended.
>
Most of the performance increase on Linux, if your RAID card has a data-safe write-back cache (battery or solid state cache persistence in case of power failure), is a separate file system, not separate spindle. Especially if ext3 is used with the default "ordered" journal, it is absolutely caustic to performance to have WAL and data on the same file system.
The whole 'separate spindle' thing is important if you don't have a good caching raid card, otherwise it doesn't matter that much.
> Be aware that you must pay CLOSE ATTENTION to your backup strategy if WAL is on a different physical disk. Snapshotting the data disk where WAL is on a separate spindle and backing it up **WILL NOT WORK** and **WILL** result in an non-restoreable backup.
>
> The manual discusses this but it's easy to miss.... don't or you'll get a NASTY surprise if something goes wrong.....
>
> -- Karl
> <karl.vcf><ATT00001..txt>
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: raid10 write performance
@ 2010-06-23 10:06 Ivan Voras <ivoras@freebsd.org>
parent: Greg Smith <greg@2ndquadrant.com>
0 siblings, 2 replies; 12+ messages in thread
From: Ivan Voras @ 2010-06-23 10:06 UTC (permalink / raw)
To: pgsql-performance
On 06/22/10 16:40, Greg Smith wrote:
> Grzegorz Jaśkiewicz wrote:
>> raid: serveRAID M5014 SAS/SATA controller
>>
>
> Do the "performant servers" have a different RAID card? This one has
> terrible performance, and could alone be the source of your issue. The
> ServeRAID cards are slow in general, and certainly slow running RAID10.
What are some good RAID10 cards nowadays?
On the other hand, RAID10 is simple enough that soft-RAID
implementations should be more than adequate - any ideas why a dedicated
card has it "slow"?
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: raid10 write performance
@ 2010-06-23 12:00 Florian Weimer <fweimer@bfk.de>
parent: Ivan Voras <ivoras@freebsd.org>
1 sibling, 1 reply; 12+ messages in thread
From: Florian Weimer @ 2010-06-23 12:00 UTC (permalink / raw)
To: Ivan Voras <ivoras@freebsd.org>; +Cc: pgsql-performance
* Ivan Voras:
> On the other hand, RAID10 is simple enough that soft-RAID
> implementations should be more than adequate - any ideas why a dedicated
> card has it "slow"?
Barrier support on RAID10 seems to require some smallish amount of
non-volatile storage which supports a high number of write operations
per second, so a software-only solution might not be available.
--
Florian Weimer <fweimer@bfk.de>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: raid10 write performance
@ 2010-06-23 12:25 Ivan Voras <ivoras@freebsd.org>
parent: Florian Weimer <fweimer@bfk.de>
0 siblings, 2 replies; 12+ messages in thread
From: Ivan Voras @ 2010-06-23 12:25 UTC (permalink / raw)
To: pgsql-performance
On 06/23/10 14:00, Florian Weimer wrote:
> * Ivan Voras:
>
>> On the other hand, RAID10 is simple enough that soft-RAID
>> implementations should be more than adequate - any ideas why a dedicated
>> card has it "slow"?
>
> Barrier support on RAID10 seems to require some smallish amount of
> non-volatile storage which supports a high number of write operations
> per second, so a software-only solution might not be available.
If I understand you correctly, this can be said in general for all
spinning-disk usage and is not specific to RAID10. (And in the case of
high, constant TPS, no amount of NVRAM will help you).
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: raid10 write performance
@ 2010-06-23 12:46 Matthew Wakeling <matthew@flymine.org>
parent: Ivan Voras <ivoras@freebsd.org>
1 sibling, 0 replies; 12+ messages in thread
From: Matthew Wakeling @ 2010-06-23 12:46 UTC (permalink / raw)
To: Ivan Voras <ivoras@freebsd.org>; +Cc: pgsql-performance
On Wed, 23 Jun 2010, Ivan Voras wrote:
> On 06/23/10 14:00, Florian Weimer wrote:
>> Barrier support on RAID10 seems to require some smallish amount of
>> non-volatile storage which supports a high number of write operations
>> per second, so a software-only solution might not be available.
>
> If I understand you correctly, this can be said in general for all
> spinning-disk usage and is not specific to RAID10. (And in the case of
> high, constant TPS, no amount of NVRAM will help you).
No. Write barriers work fine with a single disc, assuming it is set up
correctly. The barrier is a command telling the disc to make sure that one
piece of data is safe before starting to write another piece of data.
However, as soon as you have multiple discs, the individual discs do not
have a way of communicating with each other to make sure that the first
piece of data is written before the other. That's why you need a little
bit of non-volatile storage to mediate that to properly support barriers.
Of course, from a performance point of view, yes, you need some NVRAM on
any kind of spinning storage to maintain high commit rates.
Matthew
--
I wouldn't be so paranoid if you weren't all out to get me!!
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: raid10 write performance
@ 2010-06-23 12:54 Scott Marlowe <scott.marlowe@gmail.com>
parent: Ivan Voras <ivoras@freebsd.org>
1 sibling, 0 replies; 12+ messages in thread
From: Scott Marlowe @ 2010-06-23 12:54 UTC (permalink / raw)
To: Ivan Voras <ivoras@freebsd.org>; +Cc: pgsql-performance
On Wed, Jun 23, 2010 at 8:25 AM, Ivan Voras <ivoras@freebsd.org> wrote:
> On 06/23/10 14:00, Florian Weimer wrote:
>> * Ivan Voras:
>>
>>> On the other hand, RAID10 is simple enough that soft-RAID
>>> implementations should be more than adequate - any ideas why a dedicated
>>> card has it "slow"?
>>
>> Barrier support on RAID10 seems to require some smallish amount of
>> non-volatile storage which supports a high number of write operations
>> per second, so a software-only solution might not be available.
>
> If I understand you correctly, this can be said in general for all
> spinning-disk usage and is not specific to RAID10. (And in the case of
> high, constant TPS, no amount of NVRAM will help you).
Not entirely true. Let's say you have enough battery backed cache to
hold 10,000 transaction writes in memory at once. The RAID controller
can now re-order those writes so that they go from one side of the
disk to the other, instead of randomly all over the place. That will
most certainly help improve your throughput.
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: raid10 write performance
@ 2010-06-23 12:56 Scott Marlowe <scott.marlowe@gmail.com>
parent: Ivan Voras <ivoras@freebsd.org>
1 sibling, 0 replies; 12+ messages in thread
From: Scott Marlowe @ 2010-06-23 12:56 UTC (permalink / raw)
To: Ivan Voras <ivoras@freebsd.org>; +Cc: pgsql-performance
On Wed, Jun 23, 2010 at 6:06 AM, Ivan Voras <ivoras@freebsd.org> wrote:
> On 06/22/10 16:40, Greg Smith wrote:
>> Grzegorz Jaśkiewicz wrote:
>>> raid: serveRAID M5014 SAS/SATA controller
>>>
>>
>> Do the "performant servers" have a different RAID card? This one has
>> terrible performance, and could alone be the source of your issue. The
>> ServeRAID cards are slow in general, and certainly slow running RAID10.
>
> What are some good RAID10 cards nowadays?
LSI, Areca, 3Ware (now LSI I believe)
> On the other hand, RAID10 is simple enough that soft-RAID
> implementations should be more than adequate - any ideas why a dedicated
> card has it "slow"?
This is mostly a problem with some older cards that focused on RAID-5
performance, and RAID-10 was an afterthought. On many of these cards
(older PERCs for instance) it was faster to either use a bunch of
RAID-1 pairs in hardware with RAID-0 in software on top, or put the
thing into JBOD mode and do it all in software.
^ permalink raw reply [nested|flat] 12+ messages in thread
end of thread, other threads:[~2010-06-23 12:56 UTC | newest]
Thread overview: 12+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2010-06-22 09:31 raid10 write performance Grzegorz Jaśkiewicz <gryzman@gmail.com>
2010-06-22 13:40 ` Justin Graf <justin@magwerks.com>
2010-06-22 14:29 ` Karl Denninger <karl@denninger.net>
2010-06-22 16:01 ` Dave Crooke <dcrooke@gmail.com>
2010-06-22 17:19 ` Scott Carey <scott@richrelevance.com>
2010-06-22 14:40 ` Greg Smith <greg@2ndquadrant.com>
2010-06-23 10:06 ` Ivan Voras <ivoras@freebsd.org>
2010-06-23 12:00 ` Florian Weimer <fweimer@bfk.de>
2010-06-23 12:25 ` Ivan Voras <ivoras@freebsd.org>
2010-06-23 12:46 ` Matthew Wakeling <matthew@flymine.org>
2010-06-23 12:54 ` Scott Marlowe <scott.marlowe@gmail.com>
2010-06-23 12:56 ` Scott Marlowe <scott.marlowe@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox