Received: from maia.hub.org (unknown [200.46.208.211]) by mail.postgresql.org (Postfix) with ESMTP id 4ED39632E61 for ; Fri, 15 Jan 2010 01:46:43 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.208.211]) (amavisd-maia, port 10024) with ESMTP id 00787-04-3 for ; Fri, 15 Jan 2010 05:46:31 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.17.8]) by mail.postgresql.org (Postfix) with ESMTP id 49AAC634E77 for ; Fri, 15 Jan 2010 01:46:32 -0400 (AST) Received: from [192.168.1.3] (pool-96-244-14-10.bltmmd.fios.verizon.net [96.244.14.10]) by mrelayeu.kundenserver.de (node=mrbap2) with ESMTP (Nemesis) id 0Mfjsa-1NAqu73dqu-00NF6z; Fri, 15 Jan 2010 06:46:29 +0100 Message-ID: <4B500132.6050203@2ndquadrant.com> Date: Fri, 15 Jan 2010 00:46:26 -0500 From: Greg Smith User-Agent: Thunderbird 2.0.0.23 (X11/20090817) MIME-Version: 1.0 To: Fernando Hevia CC: pgsql-performance@postgresql.org Subject: Re: new server I/O setup References: <3CD96C0AB4114484BFE358A3EDE61C01@iptel.com.ar> In-Reply-To: <3CD96C0AB4114484BFE358A3EDE61C01@iptel.com.ar> Content-Type: multipart/alternative; boundary="------------030201010302090202000402" X-Provags-ID: V01U2FsdGVkX19VnWHhiY2DV+tgdPov15gVU4SYdQ9pIEDIuNe ePYHgS42130ZRs2SGE5zFFFyZ5RzriqqhORcpPZsn3B9QY7IDP KUtcLT2BqQcGf+5yOW7CA5gsgBi9y8F X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-2.427 tagged_above=-10 required=5 tests=AWL=0.171, BAYES_00=-2.599, HTML_MESSAGE=0.001 X-Spam-Level: X-Archive-Number: 201001/324 X-Sequence-Number: 37204 This is a multi-part message in MIME format. --------------030201010302090202000402 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Fernando Hevia wrote: > I justified my first choice in that WAL writes are sequentially and OS > pretty much are too, so a RAID 1 probably would hold ground against a > 12 disc RAID 10 with random writes. The problem with this theory is that when PostgreSQL does WAL writes and asks to sync the data, you'll probably discover all of the open OS writes that were sitting in the Linux write cache getting flushed before that happens. And that could lead to horrible performance--good luck if the database tries to do something after cron kicks off updatedb each night for example. I think there are two viable configurations you should be considering you haven't thought about: , but neither is quite what you're looking at: 2 discs in RAID 1 for OS 2 discs in RAID 1 for pg_xlog 10 discs in RAID 10 for postgres, ext3 2 spares. 14 discs in RAID 10 for everything 2 spares. Impossible to say which of the four possibilities here will work out better. I tend to lean toward the first one I listed above because it makes it very easy to monitor the pg_xlog activity (and the non-database activity) separately from everything else, and having no other writes going on makes it very unlikely that the pg_xlog will ever become a bottleneck. But if you've got 14 disks in there, it's unlikely to be a bottleneck anyway. The second config above will get you slightly better random I/O though, so for workloads that are really limited on that there's a good reason to prefer it. Also: the whole "use ext2 for the pg_xlog" idea is overrated far as I'm concerned. I start with ext3, and only if I get evidence that the drive is a bottleneck do I ever think of reverting to unjournaled writes just to get a little speed boost. In practice I suspect you'll see no benchmark difference, and will instead curse the decision the first time your server is restarted badly and it gets stuck at fsck. > Pd: any clue if hdparm works to deactive the disks write cache even if > they are behind the 3ware controller? You don't use hdparm for that sort of thing; you need to use 3ware's tw_cli utility. I believe that the individual drive caches are always disabled, but whether the controller cache is turned on or not depends on whether the card has a battery. The behavior here is kind of weird though--it changes if you're in RAID mode vs. JBOD mode, so be careful to look at what all the settings are. Some of these 3ware cards default to extremely aggressive background scanning for bad blocks too, you might have to tweak that downward too. -- Greg Smith 2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support greg@2ndQuadrant.com www.2ndQuadrant.com --------------030201010302090202000402 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Fernando Hevia wrote:
I justified my first choice in that WAL writes are sequentially and OS pretty much are too, so a RAID 1 probably would hold ground against a 12 disc RAID 10 with random writes.

The problem with this theory is that when PostgreSQL does WAL writes and asks to sync the data, you'll probably discover all of the open OS writes that were sitting in the Linux write cache getting flushed before that happens.  And that could lead to horrible performance--good luck if the database tries to do something after cron kicks off updatedb each night for example.

I think there are two viable configurations you should be considering you haven't thought about:
, but neither is quite what you're looking at:

2 discs in RAID 1 for OS
2 discs in RAID 1 for pg_xlog
10 discs in RAID 10 for postgres, ext3
2 spares.

14 discs in RAID 10 for everything
2 spares.

Impossible to say which of the four possibilities here will work out better.  I tend to lean toward the first one I listed above because it makes it very easy to monitor the pg_xlog activity (and the non-database activity) separately from everything else, and having no other writes going on makes it very unlikely that the pg_xlog will ever become a bottleneck.  But if you've got 14 disks in there, it's unlikely to be a bottleneck anyway.  The second config above will get you slightly better random I/O though, so for workloads that are really limited on that there's a good reason to prefer it.

Also:  the whole "use ext2 for the pg_xlog" idea is overrated far as I'm concerned.  I start with ext3, and only if I get evidence that the drive is a bottleneck do I ever think of reverting to unjournaled writes just to get a little speed boost.  In practice I suspect you'll see no benchmark difference, and will instead curse the decision the first time your server is restarted badly and it gets stuck at fsck.

Pd: any clue if hdparm works to deactive the disks write cache even if they are behind the 3ware controller?

You don't use hdparm for that sort of thing; you need to use 3ware's tw_cli utility.  I believe that the individual drive caches are always disabled, but whether the controller cache is turned on or not depends on whether the card has a battery.  The behavior here is kind of weird though--it changes if you're in RAID mode vs. JBOD mode, so be careful to look at what all the settings are.  Some of these 3ware cards default to extremely aggressive background scanning for bad blocks too, you might have to tweak that downward too.

-- 
Greg Smith    2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com  www.2ndQuadrant.com
--------------030201010302090202000402--