Received: from maia.hub.org (unknown [200.46.208.211]) by mail.postgresql.org (Postfix) with ESMTP id 65CA3632F00 for ; Wed, 7 Apr 2010 19:37:08 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.208.211]) (amavisd-maia, port 10024) with ESMTP id 41899-02 for ; Wed, 7 Apr 2010 22:36:51 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from spinlock.commandprompt.com (host-215.commandprompt.net [207.173.203.215]) by mail.postgresql.org (Postfix) with ESMTP id B8A84632B88 for ; Wed, 7 Apr 2010 19:36:57 -0300 (ADT) Received: from localhost (localhost [127.0.0.1]) by spinlock.commandprompt.com (Postfix) with ESMTP id 49953348036; Wed, 7 Apr 2010 15:36:56 -0700 (PDT) Received: from spinlock.commandprompt.com ([127.0.0.1]) by localhost (spinlock.commandprompt.com.commandprompt.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id moSwFPVopvVI; Wed, 7 Apr 2010 15:36:56 -0700 (PDT) Received: from [192.168.1.100] (unknown [75.142.159.62]) by spinlock.commandprompt.com (Postfix) with ESMTPSA id E66C4348034; Wed, 7 Apr 2010 15:36:55 -0700 (PDT) Subject: Re: Occasional giant spikes in CPU load From: "Joshua D. Drake" Reply-To: jd@commandprompt.com To: Craig James Cc: pgsql-performance@postgresql.org In-Reply-To: <4BBCFCDE.5020405@emolecules.com> References: <4BBCFB12.1010306@emolecules.com> <1270676410.19131.21.camel@jd-desktop.unknown.charter.com> <4BBCFCDE.5020405@emolecules.com> Content-Type: text/plain; charset="UTF-8" Organization: Command Prompt, Inc. Date: Wed, 07 Apr 2010 15:36:48 -0700 Message-ID: <1270679808.19131.28.camel@jd-desktop.unknown.charter.com> Mime-Version: 1.0 X-Mailer: Evolution 2.28.1 Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.002 tagged_above=-10 required=5 tests=AWL=-0.817, BAYES_40=-0.185 X-Spam-Level: X-Archive-Number: 201004/77 X-Sequence-Number: 38327 On Wed, 2010-04-07 at 14:45 -0700, Craig James wrote: > On 4/7/10 2:40 PM, Joshua D. Drake wrote: > > On Wed, 2010-04-07 at 14:37 -0700, Craig James wrote: > >> Most of the time Postgres runs nicely, but two or three times a day we get a huge spike in the CPU load that lasts just a short time -- it jumps to 10-20 CPU loads. Today it hit 100 CPU loads. Sometimes days go by with no spike events. During these spikes, the system is completely unresponsive (you can't even login via ssh). > >> > >> I managed to capture one such event using top(1) with the "batch" option as a background process. See output below - it shows 19 active postgress processes, but I think it missed the bulk of the spike. > > > > What does iostat 5 say during the jump? > > It's very hard to say ... I'll have to start a background job to watch for a day or so. While it's happening, you can't login, and any open windows become unresponsive. I'll probably have to run it at high priority using nice(1) to get any data at all during the event. Do you have sar runing? Say a sar -A ? > > Would vmstat be informative? Yes. My guess is that it is not CPU, it is IO and your CPU usage is all WAIT on IO. To have your CPUs so flooded that they are the cause of an inability to log in is pretty suspect. Joshua D. Drake > > Thanks, > Craig > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.