From: Craig James <craig_james@emolecules.com>
To: jd@commandprompt.com
Cc: pgsql-performance@postgresql.org
Subject: Re: Occasional giant spikes in CPU load
Date: Wed, 07 Apr 2010 15:57:25 -0700
Message-ID: <4BBD0DD5.3050807@emolecules.com> (raw)
In-Reply-To: <1270679808.19131.28.camel@jd-desktop.unknown.charter.com>
References: <k2pda0294261004061437p9cd3810dqe9ce528b6ddb0873@mail.gmail.com>
<m2m603c8f071004071413w1f2d2497hbbe07d9c6491848f@mail.gmail.com>
<4BBCFB12.1010306@emolecules.com>
<1270676410.19131.21.camel@jd-desktop.unknown.charter.com>
<4BBCFCDE.5020405@emolecules.com>
<1270679808.19131.28.camel@jd-desktop.unknown.charter.com>
On 4/7/10 3:36 PM, Joshua D. Drake wrote:
> On Wed, 2010-04-07 at 14:45 -0700, Craig James wrote:
>> On 4/7/10 2:40 PM, Joshua D. Drake wrote:
>>> On Wed, 2010-04-07 at 14:37 -0700, Craig James wrote:
>>>> Most of the time Postgres runs nicely, but two or three times a day we get a huge spike in the CPU load that lasts just a short time -- it jumps to 10-20 CPU loads. Today it hit 100 CPU loads. Sometimes days go by with no spike events. During these spikes, the system is completely unresponsive (you can't even login via ssh).
>>>>
>>>> I managed to capture one such event using top(1) with the "batch" option as a background process. See output below - it shows 19 active postgress processes, but I think it missed the bulk of the spike.
>>>
>>> What does iostat 5 say during the jump?
>>
>> It's very hard to say ... I'll have to start a background job to watch for a day or so. While it's happening, you can't login, and any open windows become unresponsive. I'll probably have to run it at high priority using nice(1) to get any data at all during the event.
>
> Do you have sar runing? Say a sar -A ?
No, I don't have it installed. I'll have a look. At first glance it looks like a combination of what I can get with "top -b" and vmstat, but with a single program.
> My guess is that it is not CPU, it is IO and your CPU usage is all WAIT
> on IO.
>
> To have your CPUs so flooded that they are the cause of an inability to
> log in is pretty suspect.
I thought so too, except that I can't login during the flood. If the CPUs were all doing iowaits, logging in should be easy.
Greg's suggestion that shared_buffers and work_mem are too big for an 8 GB system fits these symptoms -- if it's having a swap storm, login is effectively impossible.
Craig
>
> Joshua D. Drake
>
>
>>
>> Thanks,
>> Craig
>>
>
>
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: craig_james@emolecules.com, jd@commandprompt.com
Subject: Re: Occasional giant spikes in CPU load
In-Reply-To: <4BBD0DD5.3050807@emolecules.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox