Message-ID: <4BBD0DD5.3050807@emolecules.com>
Date: Wed, 07 Apr 2010 15:57:25 -0700
From: Craig James <craig_james@emolecules.com>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
	rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4
MIME-Version: 1.0
To: jd@commandprompt.com
CC: pgsql-performance@postgresql.org
Subject: Re: Occasional giant spikes in CPU load
References: <k2pda0294261004061437p9cd3810dqe9ce528b6ddb0873@mail.gmail.com>	
	<m2m603c8f071004071413w1f2d2497hbbe07d9c6491848f@mail.gmail.com>	
	<4BBCFB12.1010306@emolecules.com>	
	<1270676410.19131.21.camel@jd-desktop.unknown.charter.com>	
	<4BBCFCDE.5020405@emolecules.com>
	<1270679808.19131.28.camel@jd-desktop.unknown.charter.com>
In-Reply-To: <1270679808.19131.28.camel@jd-desktop.unknown.charter.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 4/7/10 3:36 PM, Joshua D. Drake wrote:
> On Wed, 2010-04-07 at 14:45 -0700, Craig James wrote:
>> On 4/7/10 2:40 PM, Joshua D. Drake wrote:
>>> On Wed, 2010-04-07 at 14:37 -0700, Craig James wrote:
>>>> Most of the time Postgres runs nicely, but two or three times a day we get a huge spike in the CPU load that lasts just a short time -- it jumps to 10-20 CPU loads.  Today it hit 100 CPU loads.  Sometimes days go by with no spike events.  During these spikes, the system is completely unresponsive (you can't even login via ssh).
>>>>
>>>> I managed to capture one such event using top(1) with the "batch" option as a background process.  See output below - it shows 19 active postgress processes, but I think it missed the bulk of the spike.
>>>
>>> What does iostat 5 say during the jump?
>>
>> It's very hard to say ... I'll have to start a background job to watch for a day or so.  While it's happening, you can't login, and any open windows become unresponsive.  I'll probably have to run it at high priority using nice(1) to get any data at all during the event.
>
> Do you have sar runing? Say a sar -A ?

No, I don't have it installed.  I'll have a look. At first glance it looks like a combination of what I can get with "top -b" and vmstat, but with a single program.

> My guess is that it is not CPU, it is IO and your CPU usage is all WAIT
> on IO.
>
> To have your CPUs so flooded that they are the cause of an inability to
> log in is pretty suspect.

I thought so too, except that I can't login during the flood.  If the CPUs were all doing iowaits, logging in should be easy.

Greg's suggestion that shared_buffers and work_mem are too big for an 8 GB system fits these symptoms -- if it's having a swap storm, login is effectively impossible.

Craig

>
> Joshua D. Drake
>
>
>>
>> Thanks,
>> Craig
>>
>
>