Received: from maia.hub.org (unknown [200.46.208.211]) by mail.postgresql.org (Postfix) with ESMTP id BD40F6334F2 for ; Wed, 7 Apr 2010 18:50:49 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.208.211]) (amavisd-maia, port 10024) with ESMTP id 33979-10 for ; Wed, 7 Apr 2010 21:50:33 +0000 (UTC) X-Greylist: delayed 00:05:31.757236 by SQLgrey-1.7.6 Received: from mail-pw0-f46.google.com (mail-pw0-f46.google.com [209.85.160.46]) by mail.postgresql.org (Postfix) with ESMTP id 039666334DD for ; Wed, 7 Apr 2010 18:50:38 -0300 (ADT) Received: by pwj9 with SMTP id 9so1446161pwj.19 for ; Wed, 07 Apr 2010 14:50:37 -0700 (PDT) Received: by 10.141.15.4 with SMTP id s4mr17288rvi.112.1270676705531; Wed, 07 Apr 2010 14:45:05 -0700 (PDT) Received: from Craig-Jamess-MacBook-Pro.local (wsip-70-167-124-126.sd.sd.cox.net [70.167.124.126]) by mx.google.com with ESMTPS id b2sm5623327rvn.6.2010.04.07.14.45.03 (version=SSLv3 cipher=RC4-MD5); Wed, 07 Apr 2010 14:45:04 -0700 (PDT) Message-ID: <4BBCFCDE.5020405@emolecules.com> Date: Wed, 07 Apr 2010 14:45:02 -0700 From: Craig James User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 MIME-Version: 1.0 To: jd@commandprompt.com CC: pgsql-performance@postgresql.org Subject: Re: Occasional giant spikes in CPU load References: <4BBCFB12.1010306@emolecules.com> <1270676410.19131.21.camel@jd-desktop.unknown.charter.com> In-Reply-To: <1270676410.19131.21.camel@jd-desktop.unknown.charter.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.223 tagged_above=-10 required=5 tests=AWL=-0.113, BAYES_05=-1.11 X-Spam-Level: X-Archive-Number: 201004/74 X-Sequence-Number: 38324 On 4/7/10 2:40 PM, Joshua D. Drake wrote: > On Wed, 2010-04-07 at 14:37 -0700, Craig James wrote: >> Most of the time Postgres runs nicely, but two or three times a day we get a huge spike in the CPU load that lasts just a short time -- it jumps to 10-20 CPU loads. Today it hit 100 CPU loads. Sometimes days go by with no spike events. During these spikes, the system is completely unresponsive (you can't even login via ssh). >> >> I managed to capture one such event using top(1) with the "batch" option as a background process. See output below - it shows 19 active postgress processes, but I think it missed the bulk of the spike. > > What does iostat 5 say during the jump? It's very hard to say ... I'll have to start a background job to watch for a day or so. While it's happening, you can't login, and any open windows become unresponsive. I'll probably have to run it at high priority using nice(1) to get any data at all during the event. Would vmstat be informative? Thanks, Craig