Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1ZbbvV-0008Gt-Oz for pgsql-performance@arkaria.postgresql.org; Mon, 14 Sep 2015 22:08:41 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.84) (envelope-from ) id 1ZbbvV-0000KJ-9g for pgsql-performance@arkaria.postgresql.org; Mon, 14 Sep 2015 22:08:41 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84) (envelope-from ) id 1ZbbvT-0000K0-Jh for pgsql-performance@postgresql.org; Mon, 14 Sep 2015 22:08:39 +0000 Received: from mail-bn1on0055.outbound.protection.outlook.com ([157.56.110.55] helo=na01-bn1-obe.outbound.protection.outlook.com) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84) (envelope-from ) id 1ZbbvO-0001DN-ST for pgsql-performance@postgresql.org; Mon, 14 Sep 2015 22:08:38 +0000 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Jim.Nasby@bluetreble.com; Received: from decina.local (70.113.16.71) by BLUPR11MB0673.namprd11.prod.outlook.com (10.163.216.143) with Microsoft SMTP Server (TLS) id 15.1.268.17; Mon, 14 Sep 2015 22:08:18 +0000 Subject: Re: Server slowing down over time To: Tomas Vondra , References: <55E9879F.2060105@2ndquadrant.com> From: Jim Nasby Message-ID: <55F74547.40505@BlueTreble.com> Date: Mon, 14 Sep 2015 17:08:07 -0500 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <55E9879F.2060105@2ndquadrant.com> Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [70.113.16.71] X-ClientProxiedBy: BN4PR15CA0013.namprd15.prod.outlook.com (25.164.54.151) To BLUPR11MB0673.namprd11.prod.outlook.com (25.163.216.143) X-Microsoft-Exchange-Diagnostics: 1; BLUPR11MB0673; 2:YWOmC0lBIJjGzjwRTCxoI9uB7CObRAEBdYmx9pEWUEGDmpiSvXNeRNex1a+lRkwYBeRufd5XqOFLfpD5Tok0Jdj4rlOn1pAi/6bHgsHf15Fq8iC0xqW5Yg3Evr6hIlIdtDn1kgV6ErXGCia+qhFOqZpo0oSgTfAMLt6gHrFteRk=; 3:bXm9XkhVgEdVhAX5HS0WyF7EHo1TaK1PKpmZccBxegF71EJ4aJ4SIsmLTL7atVPYOIbUihH91hKkD8KwEdpMPsAykhd1wzO6PlqSAQEKtEC4kqwwg8wYHauIvPT5AZSlD/Ee+Pb0Q9KpDGRAUVXcTw==; 25:hDS7knxihiKgY8grEoCT4XwaSKWZi4Q5PHr29aSnvFdzK036I/CIsWEcVod3wNkK3JMPTPnOufMKrDTPc52xRIj3/aPk1BgcYIjnwsLMGS7ikvdl8pHCgtXcmn8zaxFIZesF+eZBPWMzyspBT9cSD7pjjFTA12+N40SWp/2SgvyqTYMUHC3c8quZPQm+HWqkPHPSOhb4vYwFByFVsb3flAxCRtR0jomY8ZOrhDIcjY34X/PHQIMUIYPpdYuAyO9N; 4:atI7xexZmIwsnVzvBu3xawH7PUu86RFJFJNVTg0t7Ic96lUbcNGLrzYcMXodcthxxz4AZb9mz/8EzMvSU+aTDjH+WhlIHAfRzmnx66dGSBVe8OoCMcvfv3OrhxuhBDR3sManYGo29dYkcwcB34kzuLFuQwgT+fG0FcAI1RlNqRkp/klZQMI8thU53SNqLSN6q/jnMVGBQPqxcWnpgbV9VUgxehv+DXXtI5AAsGYSgji1/iD5U283ERBIORyFunmWJYopch19p1pjPb1GFPNxp0C3leTpmf/tO5DKAtlmPlfm1KVJ/6RDpEY05hwwCNeoI5NNW50iwP5w9cqnPujFikg8xdNTsABV0nggZs47nhk= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BLUPR11MB0673; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(5005006)(520078)(8121501046)(520075)(3002001); SRVR:BLUPR11MB0673; BCL:0; PCL:0; RULEID:; SRVR:BLUPR11MB0673; X-Forefront-PRVS: 0699FCD394 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6009001)(24454002)(199003)(189002)(479174004)(377454003)(87976001)(47776003)(77156002)(62966003)(4001350100001)(189998001)(15975445007)(92566002)(36756003)(68736005)(40100003)(122386002)(81156007)(50466002)(97736004)(4001540100001)(64126003)(76176999)(65816999)(105586002)(107886002)(54356999)(5001960100002)(5001860100001)(50986999)(2950100001)(101416001)(42186005)(33656002)(83506001)(5007970100001)(5001770100001)(77096005)(5004730100002)(5001830100001)(106356001)(65956001)(66066001)(65806001)(64706001)(46102003)(19580395003)(23676002)(86362001)(85282002)(336755003)(18886065003); DIR:OUT; SFP:1101; SCL:1; SRVR:BLUPR11MB0673; H:decina.local; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: bluetreble.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtCTFVQUjExTUIwNjczOzIzOlZKVWhFZUVLcjEzUEYvVTU4NDdMSTVtMUVS?= =?utf-8?B?bmxzR1V1UmxtcVl0MFJyS1FrWE94OFhZRG5MdzBWNEdSUjlRcDNTUCtWSlZL?= =?utf-8?B?RkhOVzRFdGpSOUllZ0YxYlFJK2hoV2FWYmZIZHQxemVNTTN3YkE1RVNMMDlw?= =?utf-8?B?VGRJNVpaQjBENzl1Y0g3b1BmVGs3elgvSmdGenBQSWlRb25pc1QxSXBVR3ow?= =?utf-8?B?MUF1cE81dDd5UzZ4eEg3M3lFNkdETGlYcFA0ME9wd2lUTzBJWHhjZnpxM09F?= =?utf-8?B?M01WVGhjYjNJMXU0djRJNGpVNTQvemY0aGFucnIyQkdKS1pLWGYyYk05eGpO?= =?utf-8?B?SWtCaVV1S1AwSWZtdDNOeDVaNUwwelBJeDlFME1VVW1WcW1YcS81eFBQRXhN?= =?utf-8?B?UlZJU2tDd0UrRy9BeVNrZXc5TVRpTzI3NzluUmxpVDNmVHhjcUhjQlI5M1d4?= =?utf-8?B?ZTliY0N6MGZOanRUMlJsWlFuM0dTUUlyd2duMjFqeWRnTUw0R3h5Wjd6TnF5?= =?utf-8?B?K0lnVXdwSVMra20zWCtqeHUyVDF1WUFpV3RPUVdIclVmRjlqcS9PYVV5R3pP?= =?utf-8?B?WnVjWEd1VGVIYlBkUkdzN0owdHB4QVM4Q2NMWllodVZMakc0ZkZUOEZ0OE92?= =?utf-8?B?YnZNZzRtWkp1YllpSG1RazFmd2k2RnlnWWwwZmhBQjRBamVZUVpZeXltWS9w?= =?utf-8?B?Z3BJVFZxc0xPT0ROK3ZJMXROMnVXTkYrRm8xTEtWRkFNemFDdFFYZ3BXVHFi?= =?utf-8?B?RHpwRXZ2dThubUFFRHpnN1JGdjRkT28wbUpnMndXOVdmWWR0Zm5ES0piNTQx?= =?utf-8?B?UVhXVWdDSmdDc21mM2VtVnZUWWlsdzNxT0tSRDB6ck0rQno0MWdMbGxzSGtO?= =?utf-8?B?WnNYTkZlWmVDdEg3RjBPUjN4R2ZnTk5GaVVaK0ZuMEs1ZmU2Um9sSkxPV09V?= =?utf-8?B?YUdHeE03MnNYN2M1b1BmUUpCczM3Z0h1eXdlVXIwNzdJM0E1ZlBkaEhuYzVT?= =?utf-8?B?SlVyYXhkd2gwUzc0KzZBVHUrZ1JuYnJtM2sxUGp1UTVKdUM3dnoxdWs5c2JH?= =?utf-8?B?Qnh2cnFTaDVHQjg3a2lsUitRRmYvSzM3bEdjU1dPMkJmNTkrVERUcHdKbmhD?= =?utf-8?B?SWlzZW10Q05TeWpzUG5NWlhiR3VCclZrdGxXT2tZSWM5ZERWS2pMZ1lBZ3h1?= =?utf-8?B?LzVRYnJMMkJLMzJsNm5tSnRBZ1FEb0dDMitzL0VUM293NTZUQ25wL0NvbGJ6?= =?utf-8?B?aVhNWkhyMGdmak1xU0NyV0xZLzBSN3F2amlNNmxpMEp3OGRIeXZnRVkwWGZi?= =?utf-8?B?WFBTb3hQMndQY2RVTGtNMThRUVRqOGpSUFRPS0xvUUV4c0p5Q2RsZmMxRklS?= =?utf-8?B?QlVTMW15V1dUTWl4MjZYazVORkpvdzA3dGIvYXcreG5CUUJ2SWw0QUY5V3hp?= =?utf-8?B?elhBTElqaGpxcXB0bDhRcCttdFk3eDUrZldKcEtDWlJBdFZuTFlvVzZTeVlN?= =?utf-8?B?TTJVVjMwbjg0Q0loYUVSbDZETG1jSVlvMkpsaWYwcTUvYmNBQVJZUDk3clpF?= =?utf-8?B?L1FLTnp3UndwSXY2bXZFR1hRblY3cG5LbE1MNE0vTjg4VHRlMHpidEVhNTZs?= =?utf-8?B?SjR0d2pFd0ZjQlhXNE9SOEwrMkl5Zkp2eFBOV3Jqa0QrL0ZleHdrL21KWkJQ?= =?utf-8?B?NEpmRjNHc294VXRJT1VFVlNjMGkvUVJzYS9ITjRqd2t3cGpmSTBIZmVSeTFN?= =?utf-8?B?MlRCZ2JZV1RCbEZ2cUVGcEZlOVN0bW03MEVOSU83Y2NjSmNZRmUvbzloelhN?= =?utf-8?B?MTJjWCtGVXpWVE5ROUNUS08xa3lHWHQ5L3FLUnA3ZHNBV1hpd20yNitXaDNT?= =?utf-8?B?LzhkMzNxVnQ2ODRKVnBsVFc2MjBDR2pKY1NwazFuYkJYMlpzVFJkSlk4c3pO?= =?utf-8?B?M2xWbDhYcWhnPT0=?= X-Microsoft-Exchange-Diagnostics: 1; BLUPR11MB0673; 5:2xo+jc7uKf/B2aJqz97dEhot/kU/twZmk9VQumZsUMnQOpR2eRi0ucGv+YEm7nLNImv+YObLuRFrUeKqaJk01+6r9eWU38ex0aVjnO0vXTvWIkcJ33lTFOE6mjjKT8RWXeUlyQKZXaLv7KUxW6/V/g==; 24:EGJd9V47Tog+lmJTykqGpPTkYatWLiKTADIIW3Mu3Wfmds0G+Ys8VilkkCS/DSdesTCmqHrQvOK5Mjma2ETs1neCymIwii4VQhzODwQOLnM=; 20:Ml/w35f5kIKg1F/wvoM1ct4v8rP0P54YpaX1jR8QrAvRzN3ov1f9ue9Lv2TnMz/6SOQvyc0Y7btXAgFeeB2Cog== SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: bluetreble.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Sep 2015 22:08:18.2031 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLUPR11MB0673 X-Pg-Spam-Score: -1.9 (-) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-performance Precedence: bulk Sender: pgsql-performance-owner@postgresql.org On 9/4/15 6:59 AM, Tomas Vondra wrote: >> Finally, whatever causes the database to wait also causes the >> Windows instance to slow down. But restarting Postgresql fixes >> this as well. > > That's a bit strange, I guess. If you're not observing light CPU and I/O > usage, then the instance should not be slow, unless there's something > else going on - possibly at the virtualization level (e.g. another busy > instance on the same hardware, some sort of accounting that limits the > resources after a time, etc.) I've experienced something similar on linux before. Database is slow, but neither CPU or IO is maxed. IIRC there was nothing disturbing in pg_locks either. I don't recall the server actually slowing down, but this would have been on something with at least 24 cores, so... My suspicion has always been that there's some form of locking that isn't being caught by the tools. -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance