Received: from maia.hub.org (unknown [200.46.208.211]) by mail.postgresql.org (Postfix) with ESMTP id 311DA633554 for ; Wed, 7 Apr 2010 20:07:46 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.208.211]) (amavisd-maia, port 10024) with ESMTP id 47069-02 for ; Wed, 7 Apr 2010 23:07:29 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from mail-iw0-f187.google.com (mail-iw0-f187.google.com [209.85.223.187]) by mail.postgresql.org (Postfix) with ESMTP id 62ABD632F00 for ; Wed, 7 Apr 2010 20:07:35 -0300 (ADT) Received: by iwn17 with SMTP id 17so939882iwn.19 for ; Wed, 07 Apr 2010 16:07:34 -0700 (PDT) Received: by 10.231.145.145 with SMTP id d17mr750728ibv.30.1270681654034; Wed, 07 Apr 2010 16:07:34 -0700 (PDT) Received: from Craig-Jamess-MacBook-Pro.local (wsip-70-167-124-126.sd.sd.cox.net [70.167.124.126]) by mx.google.com with ESMTPS id a1sm11519805ibs.12.2010.04.07.16.07.32 (version=SSLv3 cipher=RC4-MD5); Wed, 07 Apr 2010 16:07:33 -0700 (PDT) Message-ID: <4BBD1033.9080905@emolecules.com> Date: Wed, 07 Apr 2010 16:07:31 -0700 From: Craig James User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 MIME-Version: 1.0 To: Tom Lane CC: pgsql-performance@postgresql.org Subject: Re: Occasional giant spikes in CPU load References: <4BBCFB12.1010306@emolecules.com> <8127.1270677548@sss.pgh.pa.us> In-Reply-To: <8127.1270677548@sss.pgh.pa.us> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.948 tagged_above=-10 required=5 tests=AWL=0.651, BAYES_00=-2.599 X-Spam-Level: X-Archive-Number: 201004/80 X-Sequence-Number: 38330 On 4/7/10 2:59 PM, Tom Lane wrote: > Craig James writes: >> Most of the time Postgres runs nicely, but two or three times a day we get a huge spike in the CPU load that lasts just a short time -- it jumps to 10-20 CPU loads. Today it hit 100 CPU loads. Sometimes days go by with no spike events. During these spikes, the system is completely unresponsive (you can't even login via ssh). >> I managed to capture one such event using top(1) with the "batch" option as a background process. See output below - it shows 19 active postgress processes, but I think it missed the bulk of the spike. > >> Any ideas where I can look to find what's triggering this? > >> Postgres 8.3.0 > ^^^^^ > > If it's really 8.3.0, try updating to 8.3.something-recent. We've fixed > a whole lot of bugs since then. Good advice, I've been meaning to do this, maybe this will be a kick in the pants to motivate me. > I have a suspicion that this might be an sinval overrun scenario, in > which case you'd need to update to 8.4 to get a real fix. But updating > in the 8.3 branch would be cheap and easy. > > If it is sinval overrun, it would presumably be triggered by a whole lot > of catalog changes being made at approximately the same time. Can you > correlate the spikes with anything like that? Not that I know of. Just regular web traffic. On the backup server these events happen occasionally even when there is little or no web traffic, and nobody logged in doing maintenance. > > regards, tom lane >