Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lRcIf-00046I-89 for pgsql-admin@arkaria.postgresql.org; Wed, 31 Mar 2021 14:58:29 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1lRcIe-0008Nn-6l for pgsql-admin@arkaria.postgresql.org; Wed, 31 Mar 2021 14:58:28 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lRcId-0008Ng-Vs for pgsql-admin@lists.postgresql.org; Wed, 31 Mar 2021 14:58:27 +0000 Received: from mout.perfora.net ([74.208.4.196]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lRcIb-00089V-Ku for pgsql-admin@lists.postgresql.org; Wed, 31 Mar 2021 14:58:26 +0000 Received: from [192.168.1.165] ([98.169.214.254]) by mrelay.perfora.net (mreueus002 [74.208.5.2]) with ESMTPSA (Nemesis) id 0MOgc4-1lVTeX2ACM-006ArI; Wed, 31 Mar 2021 16:58:19 +0200 Subject: Re: Performance bottleneck. High active sessions but postmaster kernel threads are in a sleep state, low CPU utilization To: Laurenz Albe Cc: Hotmail , pgsql-admin@lists.postgresql.org References: <11921e4e3efa7782168b8e02b89d004ba5b6b00a.camel@cybertec.at> From: MichaelDBA Message-ID: <41c4a64a-6c74-ba06-a914-c175790a7637@sqlexec.com> Date: Wed, 31 Mar 2021 10:58:15 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 PostboxApp/7.0.47 MIME-Version: 1.0 In-Reply-To: <11921e4e3efa7782168b8e02b89d004ba5b6b00a.camel@cybertec.at> Content-Type: multipart/alternative; boundary="------------0947600BAE45CD12932357FE" Content-Language: en-US X-Provags-ID: V03:K1:2Na/S00M+lhgcmeca30aTL01jYJILgEWNf5QhSO9AEfL8c2HhwP v30WlDlgdOjuyMupnLVIpYQ4d7JjjrMYqTr5dM3DChb6o1NUkc8xASsxzmEK5YC37aA3blr KdqNW3JXrWfPlM33CKFhv55LDlVq4pPOHCL6qSbM7e/t+1H1F2GDZ9wVCWXngO7qUregUoo N/hrIcHsmy9hyTnsoTPxA== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:rSEskTO+Z1U=:7gXI2+JfTbe/blR5rbZjaC E8io9ML98M/1OgB43Z+QrGAwFWM/SBt3/HW0VFbsB767w1t2S2jLUiXlNmQI4JmUd2BkSTw1f FsTavn2+8Ec3PHYUlc1Rux1jDMWE75DSArO7BX10WI3WoDGD+6z220ywGMwTAJyNYzGE5elIO xueDHrR8YeRYXIUesXwzI/w0kw3qXG8qFbqdTbDvmROQGRtSWDfvqktwZCrevw9J04JzdVz9s 6jc48u0S5E0qf9NdqG2HEkGMsieWYe0d5TWCGjnquF7weh+kWR/t5/UPSiqO5oEtqoEeVCYQD BXED1f8Pzf6BT8jOiB8ENkMpnH5U/fVzeWOpo/Hf5CW/Mlzj4Ol+vB8wapSeV5RJ+8TUqOxry r7xTSfaVsRBX8lgKRix4Ma+WMMni17kxwng2gFLAUBjZWIny92OLWKmhUvrI/BhnDDb+qCMe2 t43J/KiFnkO0d6n6e7xdzn7nGvzMQQB64O4BlTHEsDnVzUdg/oXd List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk This is a multi-part message in MIME format. --------------0947600BAE45CD12932357FE Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Hi, You showed a lot of detail from the operating system perspective, but nothing from pg_stat_activity or pg_locks during those contention times where the cpu is low but lot of concurrent activity in the database. I would guess that there is a lot of lightweight and/or spinlocks going on. This activity usually occurs when there appears to be no heavy cpu activity, no cpu io wait activity, but the db is being hit big time with a bunch of concurrent active connections contending over the same resources as Laurenz mentioned. You can identify lightweight locks/spinlocks with the general wait type: wait_event_type = LWLock. The wait_event column will drill down even further.  You can also help yourself by installing and using pg_wait_sampling extension. Because locks come and go so fast in pg_stat_activity, this historical perspective from the sampling extension may provide a better picture. https://github.com/postgrespro/pg_wait_sampling Regards, Michael Vitale Laurenz Albe wrote on 3/31/2021 5:47 AM: > On Tue, 2021-03-30 at 12:22 -0600, Hotmail wrote: >> We are trying to gain some insight into a performance bottleneck that we are hitting while load testing Postgres on 11.11. >> [hundreds of active sessions, but CPU is not maxed out] > With that many active sessions you are probably hitting some contention inside > the database. Look at "wait_event" and "wait_event_type" in pg_stat_activity. > > Yours, > Laurenz Albe --------------0947600BAE45CD12932357FE Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit Hi,

You showed a lot of detail from the operating system perspective, but nothing from pg_stat_activity or pg_locks during those contention times where the cpu is low but lot of concurrent activity in the database.

I would guess that there is a lot of lightweight and/or spinlocks going on. This activity usually occurs when there appears to be no heavy cpu activity, no cpu io wait activity, but the db is being hit big time with a bunch of concurrent active connections contending over the same resources as Laurenz mentioned.

You can identify lightweight locks/spinlocks with the general wait type: wait_event_type = LWLock. The wait_event column will drill down even further.  You can also help yourself by installing and using pg_wait_sampling extension.  Because locks come and go so fast in pg_stat_activity, this historical perspective from the sampling extension may provide a better picture.
https://github.com/postgrespro/pg_wait_sampling

Regards,
Michael Vitale

Laurenz Albe wrote on 3/31/2021 5:47 AM:
On Tue, 2021-03-30 at 12:22 -0600, Hotmail wrote:

We are trying to gain some insight into a performance bottleneck that we are hitting while load testing Postgres on 11.11.
[hundreds of active sessions, but CPU is not maxed out]

With that many active sessions you are probably hitting some contention inside
the database.  Look at "wait_event" and "wait_event_type" in pg_stat_activity.

Yours,
Laurenz Albe

--------------0947600BAE45CD12932357FE--