Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.72) (envelope-from ) id 1UA6Yk-0006DI-Ac for pgsql-performance@arkaria.postgresql.org; Mon, 25 Feb 2013 22:30:10 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.72) (envelope-from ) id 1UA6Yj-0006v4-44 for pgsql-performance@arkaria.postgresql.org; Mon, 25 Feb 2013 22:30:09 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtp (Exim 4.72) (envelope-from ) id 1UA6Yi-0006ut-2T for pgsql-performance@postgresql.org; Mon, 25 Feb 2013 22:30:08 +0000 Received: from smtp.01.com ([199.36.142.181]) by magus.postgresql.org with esmtp (Exim 4.72) (envelope-from ) id 1UA6Yd-0003qJ-NK for pgsql-performance@postgresql.org; Mon, 25 Feb 2013 22:30:07 +0000 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp-out-1.01.com (Postfix) with ESMTP id 4F0822940BC for ; Mon, 25 Feb 2013 16:30:01 -0600 (CST) X-Virus-Scanned: amavisd-new at smtp-out-1.01.com Received: from smtp.01.com ([127.0.0.1]) by localhost (smtp-out-1.01.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8kwxVEY+puH2 for ; Mon, 25 Feb 2013 16:30:01 -0600 (CST) Received: from smtp.01.com (localhost.localdomain [127.0.0.1]) by smtp-out-1.01.com (Postfix) with ESMTP id 276F62940FC for ; Mon, 25 Feb 2013 16:30:01 -0600 (CST) Received: from [10.10.10.5] (8.202-62-69.ftth.swbr.surewest.net [69.62.202.8]) by smtp-out-1.01.com (Postfix) with ESMTPSA id EFD1F2940BC for ; Mon, 25 Feb 2013 16:30:00 -0600 (CST) Message-ID: <512BE5E8.1040702@pgexperts.com> Date: Mon, 25 Feb 2013 14:30:00 -0800 From: Jeff Frost Organization: PostgresSQL Experts, Inc. User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130110 Thunderbird/17.0.2 MIME-Version: 1.0 To: pgsql-performance@postgresql.org Subject: Re: High CPU usage / load average after upgrading to Ubuntu 12.04 References: <60B572D9298D944580F7D51195DD30804357FA4ABF@VMBX125.ihostexchange.net> <511BE879.8080706@agliodbs.com> <511C2F10.3080409@agliodbs.com> <60B572D9298D944580F7D51195DD30804357FA4DE2@VMBX125.ihostexchange.net> <511D2F11.6040706@agliodbs.com> <60B572D9298D944580F7D51195DD308043580CAA0B@VMBX125.ihostexchange.net> <511D79FE.7060705@agliodbs.com> <60B572D9298D944580F7D51195DD308043580CAA87@VMBX125.ihostexchange.net> <511E7DC0.4090506@agliodbs.com> <5122C507.7060303@agliodbs.com> <5122FF50.8030903@catalyst.net.nz> <5123BB94.7030603@agliodbs.com> <512409AF.8010308@agliodbs.com> <512551D2.9090407@agliodbs.com> <51259102.9040707@agliodbs .com> In-Reply-To: <51259102.9040707@agliodbs.com> X-Enigmail-Version: 1.5.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Pg-Spam-Score: -1.9 (-) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-performance Precedence: bulk Sender: pgsql-performance-owner@postgresql.org On 02/20/13 19:14, Josh Berkus wrote: >> Sounds to me like your IO system is stalling on fsyncs or something >> like that. On machines with plenty of IO cranking up completion >> target usuall smooths things out. > It certainly seems like it does. However, I can't demonstrate the issue > using any simpler tool than pgbench ... even running four test_fsyncs in > parallel didn't show any issues, nor do standard FS testing tools. > We were really starting to think that the system had an IO problem that we couldn't tickle with any synthetic tools. Then one of our other customers who upgraded to Ubuntu 12.04 LTS and is also experiencing issues came across the following LKML thread regarding pdflush on 3.0+ kernels: https://lkml.org/lkml/2012/10/9/210 So, I went and built a couple custom kernels with this patch removed: https://patchwork.kernel.org/patch/825212/ and the bad behavior stopped. Best performance was with a 3.5 kernel with the patch removed. -- Jeff Frost CTO, PostgreSQL Experts, Inc. Phone: 1-888-PG-EXPRT x506 FAX: 415-762-5122 http://www.pgexperts.com/ -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance