Received: from localhost (unknown [200.46.208.211]) by mail.postgresql.org (Postfix) with ESMTP id 10F3B633519 for ; Mon, 16 Mar 2009 17:10:38 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by localhost (mx1.hub.org [200.46.208.211]) (amavisd-maia, port 10024) with ESMTP id 28255-01 for ; Mon, 16 Mar 2009 17:10:22 -0300 (ADT) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-qy0-f106.google.com (mail-qy0-f106.google.com [209.85.221.106]) by mail.postgresql.org (Postfix) with ESMTP id 5E987632B04 for ; Mon, 16 Mar 2009 17:10:06 -0300 (ADT) Received: by qyk4 with SMTP id 4so1653559qyk.15 for ; Mon, 16 Mar 2009 13:10:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:from:to :in-reply-to:content-type:content-transfer-encoding:mime-version :subject:date:references:x-mailer; bh=JldeN1c/cT4hHnnJSYX5d9kQK2ZcZ6fzWKoTVcBtKto=; b=vZnjZY0DRDUmMcQEepXuimFZilhliblLT3ANFKm+LWgp1oLFrEKRbJXDYzAUSJ5xzl Bh4t5QYaEDdAvu7A3SMVXS/vyoUTAoWXbotzP+GpQyu38oxBzXYBSdHCtMr1qdAaGP7E YRJAAtNn5bMdt8F4dpxtKc25k9XiCt+7A1urg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:from:to:in-reply-to:content-type :content-transfer-encoding:mime-version:subject:date:references :x-mailer; b=QcSfJoB80UDZa4R2EPH9wPVcfTACASqpb96QvzoQVNMde9QJnSky2slTZqHLRJ1Fto hAQouJxgBlozpWfC/+cGQ2qDwFUfxhZv0K1cW0FRGxztOf7uao3ASjEO4bR/JPn9KlXR Z1+f9NUKVxPp+1Cu4BqKtRWCdgrtuUClwx9t4= Received: by 10.220.74.3 with SMTP id s3mr1850011vcj.54.1237234203880; Mon, 16 Mar 2009 13:10:03 -0700 (PDT) Received: from iad16341atl.atl0.cbeyond.net ([74.7.6.178]) by mx.google.com with ESMTPS id 4sm9289648yxq.23.2009.03.16.13.10.03 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 16 Mar 2009 13:10:03 -0700 (PDT) Message-Id: <5F9840B1-D080-4F3A-81D4-57E8F1DB0955@gmail.com> From: Joe Uhl To: pgsql-performance@postgresql.org In-Reply-To: <200903161252.22202@hal.medialogik.com> Content-Type: text/plain; charset=WINDOWS-1252; format=flowed; delsp=yes Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Apple Message framework v930.3) Subject: Re: High CPU Utilization Date: Mon, 16 Mar 2009 16:10:01 -0400 References: <8609D1C0-3ECD-4A4F-B9C6-1FC8496EDC7A@gmail.com> <200903161252.22202@hal.medialogik.com> X-Mailer: Apple Mail (2.930.3) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0 tagged_above=0 required=5 tests=none X-Spam-Level: X-Archive-Number: 200903/219 X-Sequence-Number: 33141 Here is vmstat 1 30. We are under peak load right now so I can gather =20= information from the real deal :) Had an almost complete lockup a moment ago, number of non-idle =20 postgres connections was 637. Going to drop our JDBC pool sizes a bit =20= and bounce everything. procs -----------memory---------- ---swap-- -----io---- -system-- ----=20= cpu---- r b swpd free buff cache si so bi bo in cs us sy =20= id wa 12 35 95056 11102380 56856 14954948 3 4 669 541 1 2 =20= 23 3 54 19 12 39 95056 11092484 56876 14963204 0 0 6740 1204 10066 =20 13277 91 5 0 4 8 42 95056 11081712 56888 14972244 0 0 8620 1168 10659 17020 =20= 78 6 0 15 10 30 95052 11069768 56904 14982628 0 0 8944 976 9809 15109 =20= 81 6 1 12 4 27 95048 11059576 56916 14991296 0 0 8852 440 7652 13294 =20= 63 4 2 32 5 42 95048 11052524 56932 14996496 0 0 4700 384 6383 11249 =20= 64 4 4 28 5 33 95048 11047492 56956 15001428 0 0 3852 572 6029 14010 =20= 36 4 5 56 7 35 95048 11041184 56960 15005480 0 0 3964 136 5042 10802 =20= 40 3 1 56 1 33 95048 11037988 56968 15009240 0 0 3892 168 3384 6479 =20 26 1 3 69 3 28 95048 11029332 56980 15015744 0 0 6724 152 4964 12844 =20= 11 2 8 79 0 34 95048 11025880 56988 15020168 0 0 3852 160 3616 8614 =20 11 1 6 82 3 25 95048 10996356 57044 15044796 0 0 7892 456 3126 7115 =20= 4 3 8 85 1 26 95048 10991692 57052 15050100 0 0 5188 176 2566 5976 =20= 3 2 12 83 0 29 95048 10985408 57060 15054968 0 0 4200 80 2586 6582 =20= 4 1 12 83 1 29 95048 10980828 57064 15058992 0 0 4560 64 2966 7557 =20= 7 2 6 85 2 28 95048 10977192 57072 15063176 0 0 3860 72 2695 6742 =20 11 1 7 81 2 29 95048 10969120 57088 15067808 0 0 5084 84 3296 8067 =20 14 1 0 84 0 25 95048 10962096 57104 15072984 0 0 4440 500 2721 6263 =20 12 1 6 80 0 23 95044 10955320 57108 15079260 0 0 5712 232 2678 5990 =20= 6 1 6 87 2 25 95044 10948644 57120 15084524 0 0 5120 184 3499 8143 =20 20 3 9 69 3 21 95044 10939744 57128 15090644 0 0 5756 264 4724 10272 =20= 32 3 5 60 1 19 95040 10933196 57144 15095024 12 0 4440 180 2585 5244 =20 13 2 15 70 0 21 95040 10927596 57148 15098684 0 0 3248 136 2973 7292 =20= 8 1 9 81 1 20 95040 10920708 57164 15104244 0 0 5192 360 1865 4547 =20= 3 1 9 87 1 24 95040 10914552 57172 15105856 0 0 2308 16 1948 4450 =20= 6 1 1 93 0 24 95036 10909148 57176 15110240 0 0 3824 152 1330 2632 =20= 3 1 6 90 1 21 95036 10900628 57192 15116332 0 0 5680 180 1898 3986 =20= 4 1 11 84 0 19 95036 10888356 57200 15121736 0 0 5952 120 2252 3991 =20 12 1 8 79 2 22 95036 10874336 57204 15128252 0 0 6320 112 2831 6755 =20= 5 2 8 85 3 26 95036 10857592 57220 15134020 0 0 5124 216 3067 5296 =20 32 6 3 59 Alan, my apologies if you get this twice. Didn't reply back to the =20 list on first try. On Mar 16, 2009, at 3:52 PM, Alan Hodgson wrote: > On Monday 16 March 2009, Joe Uhl wrote: >> Right now (not under peak load) this server is running at 68% CPU >> utilization and its SATA raid 10 is doing about 2MB/s writes and =20 >> 11MB/ >> s reads. When I run dd I can hit 200+MB/s writes and 230+ MB/s =20 >> reads, >> so we are barely using the available IO. Further when I run dd the >> CPU utilization of that process only approaches 20%-30% of one core. > > What does vmstat say when it's slow? The output of "vmstat 1 30" =20 > would be > informative. > > note: dd is sequential I/O. Normal database usage is random I/O. > > --=20 > Even a sixth-grader can figure out that you can=92t borrow money to =20= > pay off > your debt > > --=20 > Sent via pgsql-performance mailing list = (pgsql-performance@postgresql.org=20 > ) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-performance