Received: from malur.postgresql.org ([217.196.149.56])
 by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from
 <pgsql-performance-owner+M51412=pgsql-performance-ar@postgresql.org>)
 id 1WV6aL-00060p-TN
 for pgsql-performance@arkaria.postgresql.org; Tue, 01 Apr 2014 21:51:10 +0000
Received: from localhost ([127.0.0.1] helo=postgresql.org)
 by malur.postgresql.org with smtp (Exim 4.80) (envelope-from
 <pgsql-performance-owner+M51412=pgsql-performance-ar@postgresql.org>)
 id 1WV6aL-0001Bk-8E
 for pgsql-performance@arkaria.postgresql.org; Tue, 01 Apr 2014 21:51:09 +0000
Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29])
 by malur.postgresql.org with esmtp (Exim 4.80)
 (envelope-from <nielskristian@autouncle.com>) id 1WV6aK-0001Bc-8t
 for pgsql-performance@postgresql.org; Tue, 01 Apr 2014 21:51:08 +0000
Received: from mail-la0-f43.google.com ([209.85.215.43])
 by magus.postgresql.org with esmtp (Exim 4.80)
 (envelope-from <nielskristian@autouncle.com>) id 1WV6aG-0004Ws-Ac
 for pgsql-performance@postgresql.org; Tue, 01 Apr 2014 21:51:07 +0000
Received: by mail-la0-f43.google.com with SMTP id e16so7635482lan.30
 for <pgsql-performance@postgresql.org>; Tue, 01 Apr 2014 14:51:03 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20130820;
 h=x-gm-message-state:content-type:mime-version:subject:from
 :in-reply-to:date:cc:message-id:references:to;
 bh=MaPO5X91b0jnqeQXldczbjPgQbGY3JECOanB9eggAgw=;
 b=ck6fYp410Fy65OBmPi+y1ZbaFmwTlHbWXE6u4gqCq24IfZCWM1VsXgvysl9lj94M5Y
 eXEnNK9oSlhXtn88HdKvLsg6ZOHbCQUy/5mFE2DEsY3XIACz56jrlJUGcyr8F7Cy8Gy5
 0ZHS3fu6sG5UxMOF+eMHrii4kwvMjfUOs/okfZ+7AFodccB4ILoApCW0DtjXSIaDJf5A
 KMLicKQ5kDV0/rUMTu6gLog5YyuW1MQr3xjcRELuVpTT+5c6SnPPmmirXJsD2lANDyEp
 mQ7Z5VQTdxD6BmVw1V/s3EX/GWg35MJNTBb40cFw0XzUSZo2b9gOd1Uz6tjvTRIfok7G
 +vYg==
X-Gm-Message-State: 
 ALoCoQnFT5lZhXihf3p8G48hZfueVQ1g/NYZBa8Y0s6eIwYnIGsbz/ebOTHv0KfXNN2zQoJlAExl
X-Received: by 10.112.128.131 with SMTP id no3mr154780lbb.57.1396389062916;
 Tue, 01 Apr 2014 14:51:02 -0700 (PDT)
Received: from [192.168.0.16] (x1-6-84-1b-5e-20-07-92.cpe.webspeed.dk.
 [80.167.117.15])
 by mx.google.com with ESMTPSA id tq9sm12940108lbb.10.2014.04.01.14.51.00
 for <multiple recipients>
 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128);
 Tue, 01 Apr 2014 14:51:01 -0700 (PDT)
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_D5B93B2A-9F2F-4CFD-A36B-4BD266215DE1"
Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\))
Subject: Re: Sudden crazy high CPU usage
From: =?iso-8859-1?Q?Niels_Kristian_Schj=F8dt?= <nielskristian@autouncle.com>
In-Reply-To: 
 <CAJDzJM5+PjqS_cXvWuf258wV=08hCyBgQbNqeTxJ0Rs3QhtMHg@mail.gmail.com>
Date: Tue, 1 Apr 2014 23:50:59 +0200
Cc: "pgsql-performance@postgresql.org list" <pgsql-performance@postgresql.org>
Message-Id: <AC7A3740-68AF-42A1-A9CF-3F70ECF54A3C@autouncle.com>
References: <24BBD031-7384-4FFF-848B-90F5664C0063@autouncle.com>
 <CAL_0b1u9ZnhUrNQjC=L0j0NfbEkSNroBX4S1N4g5Rv+b77g5Gw@mail.gmail.com>
 <CAJDzJM5+PjqS_cXvWuf258wV=08hCyBgQbNqeTxJ0Rs3QhtMHg@mail.gmail.com>
To: Will Platnick <wplatnick@gmail.com>
X-Mailer: Apple Mail (2.1874)
X-Pg-Spam-Score: -2.6 (--)
List-Archive: <http://archives.postgresql.org/pgsql-performance>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-ID: <pgsql-performance.postgresql.org>
List-Owner: <mailto:pgsql-performance-owner@postgresql.org>
List-Post: <mailto:pgsql-performance@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-performance>
List-Unsubscribe: 
 <mailto:majordomo@postgresql.org?body=unsub%20pgsql-performance>
X-Mailing-List: pgsql-performance
Precedence: bulk
Sender: pgsql-performance-owner@postgresql.org


--Apple-Mail=_D5B93B2A-9F2F-4CFD-A36B-4BD266215DE1
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=iso-8859-1

Sorry, but nothing unusual here either, I have compared the time just =
before with the same time the days before and the throughput pattern is =
exactly the same. No differences.


Den 31/03/2014 kl. 22.01 skrev Will Platnick <wplatnick@gmail.com>:

>=20
> In New Relic, go back a half hour before the problem started so you =
can't see that this spike happened and send the same screenshot in. My =
guess is you have increased activity hitting the DB. Do you have =
pgbouncer or some kind of connection pooling sitting in front? 198 open =
server connections could account for an increase in load like you're =
seeing. Do you have postgresql addon in New Relic to show you how many =
queries are hitting the system to correlate data to?
>=20
> On Mon, Mar 31, 2014 at 1:36 PM, Sergey Konoplev <gray.ru@gmail.com> =
wrote:
> On Mon, Mar 31, 2014 at 3:25 AM, Niels Kristian Schj=F8dt
> <nielskristian@autouncle.com> wrote:
> > I'm running postgresql 9.3 on a production server. An hour ago, out =
of the "blue", I ran into an issue I have never encountered before: my =
server started to use CPU as crazy. The server is a standard ubuntu =
12.04 LTE installation running only Postgres and Redis.
> >
> > The incident can be seen on the in numbers below:
> >
> > https://s3-eu-west-1.amazonaws.com/autouncle-public/other/cpu.png
>=20
> The increase doesn't look so sudden. My guess is that the server got
> some new activity. The advice is to setup the statistics collecting
> script by the link [1] and review the results for a period of hour or
> so. It shows charts of statements by CPU/IO/calls with aggregated
> stats, so you could probably find out more than with pure
> pg_stat_statements.
>=20
> [1] =
https://github.com/grayhemp/pgcookbook/blob/master/statement_statistics_co=
llecting_and_reporting.md
>=20
> --
> Kind regards,
> Sergey Konoplev
> PostgreSQL Consultant and DBA
>=20
> http://www.linkedin.com/in/grayhemp
> +1 (415) 867-9984, +7 (901) 903-0499, +7 (988) 888-1979
> gray.ru@gmail.com
>=20
>=20
> --
> Sent via pgsql-performance mailing list =
(pgsql-performance@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>=20


--Apple-Mail=_D5B93B2A-9F2F-4CFD-A36B-4BD266215DE1
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=iso-8859-1

<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Diso-8859-1"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div =
apple-content-edited=3D"true">Sorry, but nothing unusual here either, I =
have compared the time just before with the same time the days before =
and the throughput pattern is exactly the same. No =
differences.</div><div><br></div>
<br><div style=3D""><div>Den 31/03/2014 kl. 22.01 skrev Will Platnick =
&lt;<a =
href=3D"mailto:wplatnick@gmail.com">wplatnick@gmail.com</a>&gt;:</div><br =
class=3D"Apple-interchange-newline"><blockquote type=3D"cite"><div =
dir=3D"ltr"><br><div class=3D"gmail_extra"><span =
style=3D"font-family:arial,sans-serif;font-size:13px">In New Relic, go =
back a half hour before the problem started so you can't see that this =
spike happened and send the same screenshot in. My guess is you have =
increased activity hitting the DB. Do you have pgbouncer or some kind of =
connection pooling sitting in front? 198 open server connections could =
account for an increase in load like you're seeing. Do you have =
postgresql addon in New Relic to show you how many queries are hitting =
the system to correlate data to?</span><br>

<br><div class=3D"gmail_quote">On Mon, Mar 31, 2014 at 1:36 PM, Sergey =
Konoplev <span dir=3D"ltr">&lt;<a href=3D"mailto:gray.ru@gmail.com" =
target=3D"_blank">gray.ru@gmail.com</a>&gt;</span> wrote:<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px =
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left=
-style:solid;padding-left:1ex"><div>On Mon, Mar 31, 2014 at 3:25 AM, =
Niels Kristian Schj=F8dt<br>


&lt;<a href=3D"mailto:nielskristian@autouncle.com" =
target=3D"_blank">nielskristian@autouncle.com</a>&gt; wrote:<br>
&gt; I'm running postgresql 9.3 on a production server. An hour ago, out =
of the "blue", I ran into an issue I have never encountered before: my =
server started to use CPU as crazy. The server is a standard ubuntu =
12.04 LTE installation running only Postgres and Redis.<br>




&gt;<br>
&gt; The incident can be seen on the in numbers below:<br>
&gt;<br>
&gt; <a =
href=3D"https://s3-eu-west-1.amazonaws.com/autouncle-public/other/cpu.png"=
 =
target=3D"_blank">https://s3-eu-west-1.amazonaws.com/autouncle-public/othe=
r/cpu.png</a><br>
<br>
</div>The increase doesn't look so sudden. My guess is that the server =
got<br>
some new activity. The advice is to setup the statistics collecting<br>
script by the link [1] and review the results for a period of hour =
or<br>
so. It shows charts of statements by CPU/IO/calls with aggregated<br>
stats, so you could probably find out more than with pure<br>
pg_stat_statements.<br>
<br>
[1] <a =
href=3D"https://github.com/grayhemp/pgcookbook/blob/master/statement_stati=
stics_collecting_and_reporting.md" =
target=3D"_blank">https://github.com/grayhemp/pgcookbook/blob/master/state=
ment_statistics_collecting_and_reporting.md</a><br>




<span><font color=3D"#888888"><br>
--<br>
Kind regards,<br>
Sergey Konoplev<br>
PostgreSQL Consultant and DBA<br>
<br>
<a href=3D"http://www.linkedin.com/in/grayhemp" =
target=3D"_blank">http://www.linkedin.com/in/grayhemp</a><br>
<a href=3D"tel:%2B1%20%28415%29%20867-9984" value=3D"+14158679984" =
target=3D"_blank">+1 (415) 867-9984</a>, <a =
href=3D"tel:%2B7%20%28901%29%20903-0499" value=3D"+79019030499" =
target=3D"_blank">+7 (901) 903-0499</a>, <a =
href=3D"tel:%2B7%20%28988%29%20888-1979" value=3D"+79888881979" =
target=3D"_blank">+7 (988) 888-1979</a><br>




<a href=3D"mailto:gray.ru@gmail.com" =
target=3D"_blank">gray.ru@gmail.com</a><br>
</font></span><div><br>
<br>
--<br>
Sent via pgsql-performance mailing list (<a =
href=3D"mailto:pgsql-performance@postgresql.org" =
target=3D"_blank">pgsql-performance@postgresql.org</a>)<br>
To make changes to your subscription:<br>
<a href=3D"http://www.postgresql.org/mailpref/pgsql-performance" =
target=3D"_blank">http://www.postgresql.org/mailpref/pgsql-performance</a>=
<br>
</div></blockquote></div><br></div></div>
</blockquote></div><br></body></html>=

--Apple-Mail=_D5B93B2A-9F2F-4CFD-A36B-4BD266215DE1--