Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1Xnuyw-0002fj-9T for pgsql-performance@arkaria.postgresql.org; Mon, 10 Nov 2014 19:50:34 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.80) (envelope-from ) id 1Xnuyv-0001Ql-Bc for pgsql-performance@arkaria.postgresql.org; Mon, 10 Nov 2014 19:50:33 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1Xnuyu-0001Qe-4V for pgsql-performance@postgresql.org; Mon, 10 Nov 2014 19:50:32 +0000 Received: from 01.zmailcloud.com ([192.198.85.104] helo=mx-out-1.zmailcloud.com) by makus.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1Xnuyq-00046Q-OY for pgsql-performance@postgresql.org; Mon, 10 Nov 2014 19:50:30 +0000 Received: from smtp.01.com (smtp.01.com [10.10.0.43]) by mx-out-1.zmailcloud.com (Postfix) with ESMTP id 1C7B2564DB0 for ; Mon, 10 Nov 2014 13:50:28 -0600 (CST) Received: from localhost (localhost [127.0.0.1]) by smtp-out-1.01.com (Postfix) with ESMTP id 1798EA03E9 for ; Mon, 10 Nov 2014 13:50:28 -0600 (CST) X-Virus-Scanned: amavisd-new at smtp-out-1.01.com Received: from smtp.01.com ([127.0.0.1]) by localhost (smtp-out-1.01.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id G6aG3-GHs3a7 for ; Mon, 10 Nov 2014 13:50:28 -0600 (CST) Received: from smtp.01.com (localhost [127.0.0.1]) by smtp-out-1.01.com (Postfix) with ESMTP id EE2A9A03BC for ; Mon, 10 Nov 2014 13:50:27 -0600 (CST) Received: from localhost (localhost [127.0.0.1]) by smtp-out-1.01.com (Postfix) with ESMTP id DFD67A03E9 for ; Mon, 10 Nov 2014 13:50:27 -0600 (CST) X-Virus-Scanned: amavisd-new at smtp-out-1.01.com Received: from smtp.01.com ([127.0.0.1]) by localhost (smtp-out-1.01.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id ftyvsUbCSuFm for ; Mon, 10 Nov 2014 13:50:27 -0600 (CST) Received: from [192.168.47.100] (50-0-164-241.dsl.dynamic.fusionbroadband.com [50.0.164.241]) by smtp-out-1.01.com (Postfix) with ESMTPSA id 94DDFA03BC for ; Mon, 10 Nov 2014 13:50:27 -0600 (CST) Message-ID: <54611702.50308@agliodbs.com> Date: Mon, 10 Nov 2014 11:50:26 -0800 From: Josh Berkus Organization: PostgreSQL Experts Inc. User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: "pgsql-performance@postgresql.org" Subject: Lock pileup causes server to stall Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Pg-Spam-Score: -1.9 (-) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-performance Precedence: bulk Sender: pgsql-performance-owner@postgresql.org All, pg version: 9.3.5 RHEL 6.5 128GB/32 cores Configured with shared_buffers=16GB Java/Tomcat/JDBC application Server has an issue that whenever we get lock waits (transaction lock waits, usually on an FK dependancy) lasting over a minute or more than 10 at once, *all* queries on the server slow to a crawl, taking 100X to 400X normal execution times. Other info: * This applies even to queries which are against other databases, so it's not purely a lock blocking issue. * this database routinely has a LOT of lock conlicts, churning through 1 million multixacts per day * pgBouncer is also involved in this stack, and may be contributing to the problem in some way * at no time is the DB server out of CPU (max usage = 38%), RAM, or doing major IO (max %util = 22%). * BIND statements can be slow as well as EXECUTEs. I don't have full query logs from a stall period yet, so I'll have more information when I do: for example, is it ALL queries which are slow or just some of them? However, I thought this list would have some other ideas where to look. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance