To: Greg Smith <gsmith@gregsmith.com>
Cc: Joe Uhl <joeuhl@gmail.com>,  pgsql-performance@postgresql.org
Subject: Re: High CPU Utilization
In-Reply-To: <alpine.GSO.2.01.0903161621230.3428@westnet.com> (Greg Smith's
	message of "Mon\, 16 Mar 2009 16\:35\:27 -0400 \(EDT\)")
User-Agent: Gnus/5.11 (Gnus v5.11) Emacs/22.1 (gnu/linux)
References: <8609D1C0-3ECD-4A4F-B9C6-1FC8496EDC7A@gmail.com>
	<200903161252.22202@hal.medialogik.com>
	<5F9840B1-D080-4F3A-81D4-57E8F1DB0955@gmail.com>
	<alpine.GSO.2.01.0903161621230.3428@westnet.com>
From: Gregory Stark <stark@enterprisedb.com>
Organization: EnterpriseDB
Date: Tue, 17 Mar 2009 00:30:20 +0000
Message-ID: <87d4ch0z4z.fsf@oxford.xeocode.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

Greg Smith <gsmith@gregsmith.com> writes:

> On Mon, 16 Mar 2009, Joe Uhl wrote:
>
>> Here is vmstat 1 30.  We are under peak load right now so I can gather
>> information from the real deal
>
> Quite helpful, reformatting a bit and picking an informative section:
>
> procs -----------memory----------    ---swap- ----io--- -system-- ----cpu----
> r  b   swpd     free   buff  cache   si   so   bi    bo   in   cs us sy id wa
> 0 34  95048 11025880  56988 15020168 0    0  3852   160 3616 8614 11  1  6 82
> 3 25  95048 10996356  57044 15044796 0    0  7892   456 3126 7115  4  3  8 85
> 1 26  95048 10991692  57052 15050100 0    0  5188   176 2566 5976  3  2 12 83
>
> This says that your server is spending all its time waiting for I/O, actual CPU
> utilization is minimal.  You're only achieving around 3-8MB/s of random I/O.
> That's the reality of what your disk I/O subsystem is capable of, regardless of
> what its sequential performance with dd looks like.  If you were to run a more
> complicated benchmark like bonnie++ instead, I'd bet that your "seeks/second"
> results are very low, even though sequential read/write is fine.
>
> The Perc5 controllers have a pretty bad reputation for performance on this
> list, even in RAID10.  Not much you can do about that beyond scrapping the
> controller and getting a better one.

Hm, well the tests I ran for posix_fadvise were actually on a Perc5 -- though
who knows if it was the same under the hood -- and I saw better performance
than this. I saw about 4MB/s for a single drive and up to about 35MB/s for 15
drives. However this was using linux md raid-0, not hardware raid.

But you shouldn't get your hopes up too much for random i/o. 3-8MB seems low
but consider the following:

 $ units
 2445 units, 71 prefixes, 33 nonlinear units

 You have: 8kB / .5|7200min
 You want: MB/s
         * 1.92
         / 0.52083333

-- 
  Gregory Stark
  EnterpriseDB          http://www.enterprisedb.com
  Ask me about EnterpriseDB's 24x7 Postgres support!