Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1XCL92-0004UZ-9l for pgsql-performance@arkaria.postgresql.org; Wed, 30 Jul 2014 04:05:40 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.80) (envelope-from ) id 1XCL91-0006aU-3f for pgsql-performance@arkaria.postgresql.org; Wed, 30 Jul 2014 04:05:39 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.80) (envelope-from ) id 1XCL8y-0006aI-OJ for pgsql-performance@postgresql.org; Wed, 30 Jul 2014 04:05:36 +0000 Received: from mail-pa0-x22e.google.com ([2607:f8b0:400e:c03::22e]) by magus.postgresql.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:256) (Exim 4.80) (envelope-from ) id 1XCL8v-0001Yd-SS for pgsql-performance@postgresql.org; Wed, 30 Jul 2014 04:05:35 +0000 Received: by mail-pa0-f46.google.com with SMTP id lj1so774812pab.5 for ; Tue, 29 Jul 2014 21:05:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:message-id:date:user-agent:mime-version:cc:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=AKZoUJUutt+ZLKGSNxBAd5XVDJ4TROZAO15GsG/UAW0=; b=H5HjIwB75uNajm1JvLIqGy1az+l1+gXqmvxNQvOzKC9Cliltg9SIgd0sIiwhmNSUCn QbqEclg/twwbRmny+5ucSgZSaUqhGLJkYQ7jkOR6Vi6S+IFJdzuCSHMIyF39atKcPjsW +NwGjPFa7xmYB1NVhmgp57FY6JJlr1kQS7KStRkPRb/bnXY+EP6Z961XUYs9E/LOEFSS KAnKOBSvyuEz31JhhaSVN9cr38xaNR3BFpiFmLORvKMaWjLgKDuOFoqsKEf1931noFaA FIC/apMf16JQYmYyfND44x+JZw0ZujSug24aEmIMDlxmH9IwQBop26xFztqfH6h1YGCZ wgFg== X-Received: by 10.68.213.34 with SMTP id np2mr243015pbc.167.1406693130660; Tue, 29 Jul 2014 21:05:30 -0700 (PDT) Received: from [192.168.2.104] ([116.226.74.154]) by mx.google.com with ESMTPSA id dw8sm3309637pab.35.2014.07.29.21.05.28 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 29 Jul 2014 21:05:29 -0700 (PDT) From: Rural Hunter X-Google-Original-From: Rural Hunter Message-ID: <53D86F05.5060007@gmail.com> Date: Wed, 30 Jul 2014 12:05:25 +0800 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.0 MIME-Version: 1.0 CC: "pgsql-performance@postgresql.org" Subject: Re: Very slow planning performance on partition table References: <53CFB6BC.4050606@gmail.com> <25010.1406253203@sss.pgh.pa.us> <53D26875.2080604@gmail.com> <53D51E49.102@gmail.com> <24564.1406478487@sss.pgh.pa.us> <53D7597E.2080209@gmail.com> In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Pg-Spam-Score: -1.0 (-) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-performance Precedence: bulk Sender: pgsql-performance-owner@postgresql.org This happened again. This time I got the connection status(between pgbouncer host to pgsql host) at postgresql side. When the problem happens, the connection status is this: ESTABLISHED: 188 CLOSE_WAIT: 116 The count of connections in CLOSE_WAIT is abnormal. Comparing with normal situation, there is usually no close_wait connection. The connection status sample is like this: ESTABLISHED: 117 CLOSE_WAIT: 0 I have 4 users configured in pgbouncer and the pool_size is 50. So the max number of connections from pgbouncer should be less than 200. The connection spike happens very quickly. I created a script to check the connections from pgbouncer. The script checks the connections from pgbouncer every 5 mins. This is the log: 10:55:01 CST pgbouncer is healthy. connection count: 73 11:00:02 CST pgbouncer is healthy. connection count: 77 11:05:01 CST pgbouncer is healthy. connection count: 118 11:10:01 CST pgbouncer is healthy. connection count: 115 11:15:01 CST pgbouncer is healthy. connection count: 75 11:20:01 CST pgbouncer is healthy. connection count: 73 11:25:02 CST pgbouncer is healthy. connection count: 75 11:30:01 CST pgbouncer is healthy. connection count: 77 11:35:01 CST pgbouncer is healthy. connection count: 84 11:40:10 CST Problematic connection count: 292, will restart pgbouncer... Now I suspect there is some network problem between the hosts of pgbouncer and pgsql. Will check more. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance