Received: from localhost (unknown [200.46.208.211]) by mail.postgresql.org (Postfix) with ESMTP id 72191634B15 for ; Mon, 16 Mar 2009 17:35:53 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by localhost (mx1.hub.org [200.46.208.211]) (amavisd-maia, port 10024) with ESMTP id 34775-04-4 for ; Mon, 16 Mar 2009 17:35:44 -0300 (ADT) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from westnet.com (westnet.com [216.187.52.2]) by mail.postgresql.org (Postfix) with ESMTP id B1936634489 for ; Mon, 16 Mar 2009 17:35:29 -0300 (ADT) Received: from westnet.com (localhost [127.0.0.1]) by westnet.com (8.14.0/8.14.0) with ESMTP id n2GKZR6Y010161; Mon, 16 Mar 2009 16:35:27 -0400 (EDT) Received: from localhost (gsmith@localhost) by westnet.com (8.14.0/8.13.2/Submit) with ESMTP id n2GKZR96010158; Mon, 16 Mar 2009 16:35:27 -0400 (EDT) X-Authentication-Warning: westnet.com: gsmith owned process doing -bs Date: Mon, 16 Mar 2009 16:35:27 -0400 (EDT) From: Greg Smith X-X-Sender: gsmith@westnet.com To: Joe Uhl cc: pgsql-performance@postgresql.org Subject: Re: High CPU Utilization In-Reply-To: <5F9840B1-D080-4F3A-81D4-57E8F1DB0955@gmail.com> Message-ID: References: <8609D1C0-3ECD-4A4F-B9C6-1FC8496EDC7A@gmail.com> <200903161252.22202@hal.medialogik.com> <5F9840B1-D080-4F3A-81D4-57E8F1DB0955@gmail.com> User-Agent: Alpine 2.01 (GSO 1184 2008-12-16) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0 tagged_above=0 required=5 tests=none X-Spam-Level: X-Archive-Number: 200903/220 X-Sequence-Number: 33142 On Mon, 16 Mar 2009, Joe Uhl wrote: > Here is vmstat 1 30. We are under peak load right now so I can gather > information from the real deal Quite helpful, reformatting a bit and picking an informative section: procs -----------memory---------- ---swap- ----io--- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 34 95048 11025880 56988 15020168 0 0 3852 160 3616 8614 11 1 6 82 3 25 95048 10996356 57044 15044796 0 0 7892 456 3126 7115 4 3 8 85 1 26 95048 10991692 57052 15050100 0 0 5188 176 2566 5976 3 2 12 83 This says that your server is spending all its time waiting for I/O, actual CPU utilization is minimal. You're only achieving around 3-8MB/s of random I/O. That's the reality of what your disk I/O subsystem is capable of, regardless of what its sequential performance with dd looks like. If you were to run a more complicated benchmark like bonnie++ instead, I'd bet that your "seeks/second" results are very low, even though sequential read/write is fine. The Perc5 controllers have a pretty bad reputation for performance on this list, even in RAID10. Not much you can do about that beyond scrapping the controller and getting a better one. What you might do in order to reduce the total number of writes needed is some standard postgresql.conf tuning; see http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server What you could do here is increase shared_buffers, checkpoint_segments, and checkpoint_completion_target as described there. Having more buffers dedicated to the database and having less checkpoints can result in less random I/O showing up, as popular data pages will stay in RAM for longer without getting written out so much. -- * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD