DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type:content-transfer-encoding;
	b=n0GWf8IN2w0VqM60DXXrSvhxcs/tfV/AgUhthiqev/rJoTzmiZNYwUuVh2iShL/nys
	MQJBhRkVHOPA7kg5v76quiORvVtQ1I23BAKJqguHQhP4o7IKwhO5lNVNf45blH42DDGC
	kNB+m8I+bEgqGgDMmk1Q6thhTKYYmPqQpKgWw=
MIME-Version: 1.0
In-Reply-To: <9719B46B-6CF8-4710-BE2A-1DE8E1C0C05B@gmail.com>
References: <8609D1C0-3ECD-4A4F-B9C6-1FC8496EDC7A@gmail.com>
	<200903161252.22202@hal.medialogik.com>
	<5F9840B1-D080-4F3A-81D4-57E8F1DB0955@gmail.com>
	<alpine.GSO.2.01.0903161621230.3428@westnet.com>
	<87d4ch0z4z.fsf@oxford.xeocode.com>
	<alpine.GSO.2.01.0903170017410.5259@westnet.com>
	<9719B46B-6CF8-4710-BE2A-1DE8E1C0C05B@gmail.com>
Date: Fri, 20 Mar 2009 14:29:33 -0600
Message-ID: <dcc563d10903201329l7ea173bcg76a06f5cce082bdb@mail.gmail.com>
Subject: Re: High CPU Utilization
From: Scott Marlowe <scott.marlowe@gmail.com>
To: Joe Uhl <joeuhl@gmail.com>
Cc: Greg Smith <gsmith@gregsmith.com>, Gregory Stark <stark@enterprisedb.com>,
	pgsql-performance@postgresql.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Fri, Mar 20, 2009 at 2:26 PM, Joe Uhl <joeuhl@gmail.com> wrote:
> On Mar 17, 2009, at 12:19 AM, Greg Smith wrote:
>
>> On Tue, 17 Mar 2009, Gregory Stark wrote:
>>
>>> Hm, well the tests I ran for posix_fadvise were actually on a Perc5 --
>>> though
>>> who knows if it was the same under the hood -- and I saw better
>>> performance
>>> than this. I saw about 4MB/s for a single drive and up to about 35MB/s
>>> for 15
>>> drives. However this was using linux md raid-0, not hardware raid.
>>
>> Right, it's the hardware RAID on the Perc5 I think people mainly complai=
n
>> about. =A0If you use it in JBOD mode and let the higher performance CPU =
in
>> your main system drive the RAID functions it's not so bad.
>>
>> --
>> * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
>
> I have not yet had a chance to try software raid on the standby server
> (still planning to) but wanted to follow up to see if there was any good =
way
> to figure out what the postgresql processes are spending their CPU time o=
n.
>
> We are under peak load right now, and I have Zabbix plotting CPU utilizat=
ion
> and CPU wait (from vmstat output) along with all sorts of other vitals on
> charts. =A0CPU utilization is a sustained 90% - 95% and CPU Wait is hangi=
ng
> below 10%. =A0Since being pointed at vmstat by this list I have been watc=
hing
> CPU Wait and it does get high at times (hence still wanting to try Perc5 =
in
> JBOD) but then there are sustained periods, right now included, where our
> CPUs are just getting crushed while wait and IO (only doing about 1.5 MB/=
sec
> right now) are very low.
>
> This high CPU utilization only occurs when under peak load and when our J=
DBC
> pools are fully loaded. =A0We are moving more things into our cache and
> constantly tuning indexes/tables but just want to see if there is some
> underlying cause that is killing us.
>
> Any recommendations for figuring out what our database is spending its CP=
U
> time on?

What does the cs entry on vmstat say at this time?  If you're cs is
skyrocketing then you're getting a context switch storm, which is
usually a sign that there are just too many things going on at once /
you've got an old kernel things like that.