From: Niels Kristian Schjødt <nielskristian@autouncle.com>
To: Merlin Moncure <mmoncure@gmail.com>
Cc: pgsql-performance@postgresql.org list <pgsql-performance@postgresql.org>
Subject: Re: Sudden crazy high CPU usage
Date: Mon, 31 Mar 2014 16:24:30 +0200
Message-ID: <2ABC90E9-8312-45A4-96E0-CFE202CCD9CE@autouncle.com> (raw)
In-Reply-To: <CAHyXU0y9hviyKWvQZxX5UWfH9M2LYvwvAOPQ_DUPva2b71t12g@mail.gmail.com>
References: <24BBD031-7384-4FFF-848B-90F5664C0063@autouncle.com>
<CAHyXU0y9hviyKWvQZxX5UWfH9M2LYvwvAOPQ_DUPva2b71t12g@mail.gmail.com>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-performance>
Thanks, this seems to persist after a reboot of the server though, and I have never in my server’s 3 months life time experienced anything like it.
Niels Kristian Schjødt
Co-founder & Developer
E-Mail: nielskristian@autouncle.com
Mobile: 0045 28 73 04 93
www.autouncle.com
Follow us: Facebook | Google+ | LinkedIn | Twitter
Get app for: iPhone & iPad | Android
Den 31/03/2014 kl. 15.47 skrev Merlin Moncure <mmoncure@gmail.com>:
> On Mon, Mar 31, 2014 at 5:25 AM, Niels Kristian Schjødt
> <nielskristian@autouncle.com> wrote:
>> I'm running postgresql 9.3 on a production server. An hour ago, out of the "blue", I ran into an issue I have never encountered before: my server started to use CPU as crazy. The server is a standard ubuntu 12.04 LTE installation running only Postgres and Redis.
>>
>> The incident can be seen on the in numbers below:
>>
>> https://s3-eu-west-1.amazonaws.com/autouncle-public/other/cpu.png
>>
>> I imidiatly took a look at pg_stat_activity but nothing in there seemed suspicious. I also had a look at the postgres log, but nothing was in there too. I have pg_stat_statements running, so I reseted that one, and nothing really suspicious occurred in there, expect for the fact, that all queries were taking 100x times longer than usual.
>>
>> I have tried the following with no luck:
>>
>> * Restart clients connecting to the db
>> * Restart postgres
>> * Restart the whole server
>>
>> I have run memory tests on the server as well, and nothing seems to be wrong.
>>
>> No changes in any software running on the servers has been made within the last 24 hours.
>>
>> The question is: I have a streaming replication server running, which I have now done a failover to, and it runs fine. However I still have no clue why my master suddenly has become so CPU consuming, and how I can debug / trace it further down?
>
> Using linux 6? One possible culprit is "Transparent Huge Page
> Compaction". It tends to hit severs with a lot of memory, especially
> if they've configured a lot of shared buffers. Google it a for a lot
> of info.
>
> There may be other issues masquerading as this one but it's the first
> thing to rule out. Symptoms are very high cpu utilization and poor
> performance that strikes without warning and then resolves also
> without warning (typically seconds or minutes after the event).
>
> For starters, take a look at the value of:
>
> /sys/kernel/mm/redhat_transparent_hugepage/enabled
>
> And do some due diligence research.
>
> merlin
Attachments:au_logo.png (image/png, 11.3K) downloadview image
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: nielskristian@autouncle.com, mmoncure@gmail.com
Subject: Re: Sudden crazy high CPU usage
In-Reply-To: <2ABC90E9-8312-45A4-96E0-CFE202CCD9CE@autouncle.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox