Received: from localhost (unknown [200.46.208.211]) by mail.postgresql.org (Postfix) with ESMTP id 37D13633C61 for ; Mon, 16 Mar 2009 19:09:18 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by localhost (mx1.hub.org [200.46.208.211]) (amavisd-maia, port 10024) with ESMTP id 56823-03 for ; Mon, 16 Mar 2009 19:09:09 -0300 (ADT) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-qy0-f132.google.com (mail-qy0-f132.google.com [209.85.221.132]) by mail.postgresql.org (Postfix) with ESMTP id 3DBAF632246 for ; Mon, 16 Mar 2009 19:09:15 -0300 (ADT) Received: by qyk38 with SMTP id 38so591949qyk.15 for ; Mon, 16 Mar 2009 15:09:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=IEvCzHHMy1louy71m4WEDIIVbrzqmor12yk1vv/0Gzg=; b=Q9LsQoRRhYWuJqjO/KvVKeqtAXAoi0KJb/E/0h2o1CpaXkgIl/HfPB9v9xTkrPPZ8R RoKdlzNaTVVx5naAJL3KzA9D60dAZzGtSLCrQkiOjTdaPDFiOyes2x81H1Ajg+GS+GoJ bJbAGOalQA+4HSAe3S96oB545TO9IGXScZ4CQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=LUCHI2FHTPRHohb2E3AAoL2NJq7GMp5X204gJjq4RN7Ew/bp6qxlrgajnNXgGKcxgP DtllktTaGVHoNHLTLHsYCU+fmaoNPLsrfQFZrkBei6rGcOUNEoLj1sWGKyOpxuuBQyGQ a4FM+y+hrPDNZ5fL0KawqsxYg557RYUVnqSwo= MIME-Version: 1.0 Received: by 10.224.45.66 with SMTP id d2mr6579177qaf.152.1237241353905; Mon, 16 Mar 2009 15:09:13 -0700 (PDT) In-Reply-To: References: <8609D1C0-3ECD-4A4F-B9C6-1FC8496EDC7A@gmail.com> <200903161252.22202@hal.medialogik.com> <5F9840B1-D080-4F3A-81D4-57E8F1DB0955@gmail.com> Date: Mon, 16 Mar 2009 16:09:13 -0600 Message-ID: Subject: Re: High CPU Utilization From: Scott Marlowe To: Joe Uhl Cc: Greg Smith , pgsql-performance@postgresql.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0 tagged_above=0 required=5 tests=none X-Spam-Level: X-Archive-Number: 200903/222 X-Sequence-Number: 33144 On Mon, Mar 16, 2009 at 2:50 PM, Joe Uhl wrote: > I dropped the pool sizes and brought things back up. =A0Things are stable= , > site is fast, CPU utilization is still high. =A0Probably just a matter of= time > before issue comes back (we get slammed as kids get out of school in the > US). Yeah, I'm guessing your server (or more specifically its RAID card) just aren't up to the task. We had the same problem last year with a machine with 16 Gig ram and dual dual core 3.0GHz xeons with a Perc 5 something or other. No matter how we tuned it or played with it, we just couldn't get good random performance out of it. It's since been replaced by a white box unit with a tyan mobo and dual 4 core opterons and an Areca 1680 and a 12 drive RAID-10. We can sustain 30 to 60 Megs a second random access with 0 to 10% iowait. Here's a typical vmstat 10 output when our load factor is hovering around 8= ... r b swpd free buff cache si so bi bo in cs us sy id = wa st 4 1 460 170812 92856 29928156 0 0 604 3986 4863 10146 74 3 20 3 0 7 1 460 124160 92912 29939660 0 0 812 5701 4829 9733 70 3 23 3 0 13 0 460 211036 92984 29947636 0 0 589 3178 4429 9964 69 3 25 3 0 7 2 460 90968 93068 29963368 0 0 1067 4463 4915 11081 78 3 14 5 0 7 3 460 115216 93100 29963336 0 0 3008 3197 4032 11812 69 4 15 12 0 6 1 460 142120 93088 29923736 0 0 1112 6390 4991 11023 75 4 15 6 0 6 0 460 157896 93208 29932576 0 0 698 2196 4151 8877 71 2 23 3 0 11 0 460 124868 93296 29948824 0 0 963 3645 4891 10382 74 3 19 4 0 5 3 460 95960 93272 29918064 0 0 592 30055 5550 7430 56 3 18 23 0 9 0 460 95408 93196 29914556 0 0 1090 3522 4463 10421 71 3 21 5 0 9 0 460 128632 93176 29916412 0 0 883 4774 4757 10378 76 4 17 3 0 Note the bursty parts where we're shoving out 30Megs a second and the wait jumps to 23%. That's about as bad as it gets during the day for us. NBote that in your graph your bi column appears to be dominating your bo column, so it looks like you're reaching a point where the write cache on the controller gets full and you're real throughput is shown to be ~ 1 megabyte a second outbound, and the inbound traffic either has priority or is just filling in the gaps. It looks to me like your RAID card is prioritizing reads over writes, and the whole system is just slowing to a crawl. I'm willing to bet that if you were running pure SW RAID with no RAID controller you'd get better numbers.