DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type:content-transfer-encoding;
	b=tQy5lhKlJ1yY5to7PxuAtBTOABtjK0pGtkmlWWFX0KVGr0ftNi2sOP4qB6ZJa8gnka
	C6hMkke1LUb1dccjDLiodAE9IiWFhcnteuB0TVsMbuqpJvKqlHs0freTS0IJiHLwUh7J
	HB639bq2NhdB4iAqgLVv2hDNIS/yMitNnPOBs=
MIME-Version: 1.0
In-Reply-To: <468A01CE-F631-4B2F-8882-03C381503DA2@gmail.com>
References: <8609D1C0-3ECD-4A4F-B9C6-1FC8496EDC7A@gmail.com>
	<200903161252.22202@hal.medialogik.com>
	<5F9840B1-D080-4F3A-81D4-57E8F1DB0955@gmail.com>
	<alpine.GSO.2.01.0903161621230.3428@westnet.com>
	<87d4ch0z4z.fsf@oxford.xeocode.com>
	<alpine.GSO.2.01.0903170017410.5259@westnet.com>
	<9719B46B-6CF8-4710-BE2A-1DE8E1C0C05B@gmail.com>
	<dcc563d10903201329l7ea173bcg76a06f5cce082bdb@mail.gmail.com>
	<468A01CE-F631-4B2F-8882-03C381503DA2@gmail.com>
Date: Fri, 20 Mar 2009 14:58:18 -0600
Message-ID: <dcc563d10903201358w2e13f785n588809ed8f6d0bdc@mail.gmail.com>
Subject: Re: High CPU Utilization
From: Scott Marlowe <scott.marlowe@gmail.com>
To: Joe Uhl <joeuhl@gmail.com>
Cc: Greg Smith <gsmith@gregsmith.com>, Gregory Stark <stark@enterprisedb.com>,
	pgsql-performance@postgresql.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Fri, Mar 20, 2009 at 2:49 PM, Joe Uhl <joeuhl@gmail.com> wrote:
>
> On Mar 20, 2009, at 4:29 PM, Scott Marlowe wrote:

>> What does the cs entry on vmstat say at this time? =A0If you're cs is
>> skyrocketing then you're getting a context switch storm, which is
>> usually a sign that there are just too many things going on at once /
>> you've got an old kernel things like that.
>
> cs column (plus cpu columns) of vmtstat 1 30 reads as follows:
>
> cs =A0 =A0us =A0sy id wa
> 11172 95 =A04 =A01 =A00
> 12498 94 =A05 =A01 =A00
> 14121 91 =A07 =A01 =A01
> 11310 90 =A07 =A01 =A01
> 12918 92 =A06 =A01 =A01
> 10613 93 =A06 =A01 =A01
> 9382 =A094 =A04 =A01 =A01
> 14023 89 =A08 =A02 =A01
> 10138 92 =A06 =A01 =A01
> 11932 94 =A04 =A01 =A01
> 15948 93 =A05 =A02 =A01
> 12919 92 =A05 =A03 =A01
> 10879 93 =A04 =A02 =A01
> 14014 94 =A05 =A01 =A01
> 9083 =A092 =A06 =A02 =A00
> 11178 94 =A04 =A02 =A00
> 10717 94 =A05 =A01 =A00
> 9279 =A097 =A02 =A01 =A00
> 12673 94 =A05 =A01 =A00
> 8058 =A082 17 =A01 =A01
> 8150 =A094 =A05 =A01 =A01
> 11334 93 =A06 =A00 =A00
> 13884 91 =A08 =A01 =A00
> 10159 92 =A07 =A00 =A00
> 9382 =A096 =A04 =A00 =A00
> 11450 95 =A04 =A01 =A00
> 11947 96 =A03 =A01 =A00
> 8616 =A095 =A04 =A01 =A00
> 10717 95 =A03 =A01 =A00
>
> We are running on 2.6.28.7-2 kernel. =A0I am unfamiliar with vmstat outpu=
t but
> reading the man page (and that cs =3D "context switches per second") make=
s my
> numbers seem very high.

No, those aren't really all that high.  If you were hitting cs
contention, I'd expect it to be in the 25k to 100k range.  <10k
average under load is pretty reasonable.

> Our sum JDBC pools currently top out at 400 connections (and we are doing
> work on all 400 right now). =A0I may try dropping those pools down even
> smaller. Are there any general rules of thumb for figuring out how many
> connections you should service at maximum? =A0I know of the memory
> constraints, but thinking more along the lines of connections per CPU cor=
e.

Well, maximum efficiency is usually somewhere in the range of 1 to 2
times the number of cores you have, so trying to get the pool down to
a dozen or two connections would be the direction to generally head.
May not be reasonable or doable though.