Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id 914DC632A8F for ; Thu, 24 Jun 2010 22:01:58 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 10028-03 for ; Fri, 25 Jun 2010 01:01:50 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from spinlock.commandprompt.com (host-215.commandprompt.net [207.173.203.215]) by mail.postgresql.org (Postfix) with ESMTP id 752FA632A04 for ; Thu, 24 Jun 2010 22:01:51 -0300 (ADT) Received: from localhost (localhost [127.0.0.1]) by spinlock.commandprompt.com (Postfix) with ESMTP id 8B2F2348034; Thu, 24 Jun 2010 18:01:44 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at commandprompt.com Received: from spinlock.commandprompt.com ([127.0.0.1]) by localhost (spinlock.commandprompt.com.commandprompt.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hDpx9NWyzLEO; Thu, 24 Jun 2010 18:01:42 -0700 (PDT) Received: from [192.168.1.100] (unknown [71.94.249.122]) by spinlock.commandprompt.com (Postfix) with ESMTPSA id 314F0348002; Thu, 24 Jun 2010 18:01:42 -0700 (PDT) Subject: Re: Occasional giant spikes in CPU load From: "Joshua D. Drake" Reply-To: jd@commandprompt.com To: Craig James Cc: pgsql-performance@postgresql.org In-Reply-To: <4C23FD52.40508@emolecules.com> References: <4BBCFB12.1010306@emolecules.com> <4BBD3A17.90209@emolecules.com> <4C23FD52.40508@emolecules.com> Content-Type: text/plain; charset="UTF-8" Organization: Command Prompt, Inc. Date: Thu, 24 Jun 2010 18:01:40 -0700 Message-ID: <1277427700.2528.3.camel@jd-desktop.unknown.charter.com> Mime-Version: 1.0 X-Mailer: Evolution 2.28.3 Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0.8 tagged_above=-10 required=5 tests=BAYES_50=0.8 X-Spam-Level: X-Archive-Number: 201006/454 X-Sequence-Number: 39310 On Thu, 2010-06-24 at 17:50 -0700, Craig James wrote: > I'm reviving this question because I never figured it out. To summarize: At random intervals anywhere from a few times per hour to once or twice a day, we see a huge spike in CPU load that essentially brings the system to a halt for up to a minute or two. Previous answers focused on "what is it doing", i.e. is it really Postgres or something else? > > Now the question has narrowed down to this: what could trigger EVERY postgres backend to do something at the same time? See the attached output from "top -b", which shows what is happening during one of the CPU spikes. checkpoint causing IO Wait. What does sar say about these times? Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Custom Development, Engineering