Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id 80DC51337BD7 for ; Wed, 2 Mar 2011 14:10:55 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 89323-01-2 for ; Wed, 2 Mar 2011 18:10:47 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from tilapia.imsweb.com (tilapia.imsweb.com [204.132.58.83]) by mail.postgresql.org (Postfix) with ESMTP id B050A1337BDD for ; Wed, 2 Mar 2011 14:10:46 -0400 (AST) Received: from FALCON.omni.imsweb.com (unknown [172.22.64.17]) by tilapia.imsweb.com (Postfix) with ESMTP id 7001F27674; Wed, 2 Mar 2011 13:10:44 -0500 (EST) Received: from falcon.omni.imsweb.com ([172.22.64.17]) by FALCON.omni.imsweb.com ([172.22.64.17]) with mapi; Wed, 2 Mar 2011 13:10:37 -0500 From: "Vaughn, Adam (IMS)" To: 'Kevin Grittner' , "'pgsql-admin@postgresql.org'" Cc: "Depuy, Scott (IMS)" , "Meagher, Kevin (IMS)" Date: Wed, 2 Mar 2011 13:10:37 -0500 Subject: Re: long running commits Thread-Topic: [ADMIN] long running commits Thread-Index: AcvVPaWCWznqzlTMRTiIuxUUv/Or5wDxvVJQ Message-ID: <6B760A41CEE02F46BADF29B4A874D6010F11C13ABF@FALCON.omni.imsweb.com> References: <6B760A41CEE02F46BADF29B4A874D6010F11C13A63@FALCON.omni.imsweb.com> <4D67DC90020000250003B04B@gw.wicourts.gov> In-Reply-To: <4D67DC90020000250003B04B@gw.wicourts.gov> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.91 tagged_above=-10 required=5 tests=BAYES_00=-1.9, T_RP_MATCHES_RCVD=-0.01 X-Spam-Level: X-Archive-Number: 201103/26 X-Sequence-Number: 36898 Thanks for the suggestions. I made all of the changes you mentioned except = for the shared_buffers (which will require a downtime I have set for tonigh= t). I do have another question though, why did you pick 512 MB for the new = setting of shared_buffers? Everything I've ever read says that 25% of avail= able RAM is a conservative value for shared_buffers. Also, we had another one of these instances earlier today. During the 23 mi= nute commit a single CPU was at 98% and it looked like all writes were back= ed up waiting for the commit to finalize. During the time our writing never= got above 25 MB/s (far less than we can handle). Is it possible that we're= missing an index somewhere or there's something else going on? Thanks again in advance -----Original Message----- From: Kevin Grittner [mailto:Kevin.Grittner@wicourts.gov] Sent: Friday, February 25, 2011 5:45 PM To: Vaughn, Adam (IMS); 'pgsql-admin@postgresql.org' Cc: Depuy, Scott (IMS); Meagher, Kevin (IMS) Subject: Re: [ADMIN] long running commits "Vaughn, Adam (IMS)" wrote: > In the past 2 days we have been experiencing problems on our > production server with very long running commit statements which > seem to cause the entire instance to become blocked and > non-responsive. These are the log messages for the 2 commit > statements (these occurred inside of a 1 hour block) > > duration: 945599.193 ms execute S_2: COMMIT > duration: 967507.896 ms execute S_2: COMMIT > > Can anyone provide some assistance in how we should diagnose this > problem (tools) and some possible solutions for solving it? Look for prior posts by Greg Smith on this topic. (Or better yet, buy his book.) I don't believe anyone has a better handle on this issue than he does. That said, if you want emergency relief before you hear from him, and a safer place from which to start tuning toward your optimal settings based on the process he describes, I would make these changes: > bgwriter_delay 100 > bgwriter_lru_maxpages 100 > bgwriter_lru_multiplier 2.2 > shared_buffers 6GB bgwriter_delay 200 bgwriter_lru_maxpages 1000 bgwriter_lru_multiplier 4 shared_buffers 512MB These settings may not give you quite the same performance between lockups, but the lockups should be less frequent and severe, and may disappear entirely. You may not want to stay here, but it's probably a relatively safe place from which to start the process of incremental adjustments to find your best settings. The overall idea of these settings is to push more of the writes toward the disk soon enough to prevent a debilitating glut at fsync time. We're getting better at dealing with this in each release, so you may want to consider moving forward from 8.3 to help with it, too. -Kevin Information in this e-mail may be confidential. It is intended only for the= addressee(s) identified above. If you are not the addressee(s), or an empl= oyee or agent of the addressee(s), please note that any dissemination, dist= ribution, or copying of this communication is strictly prohibited. If you h= ave received this e-mail in error, please notify the sender of the error.