From: MichaelDBA <MichaelDBA@sqlexec.com>
To: Laurenz Albe <laurenz.albe@cybertec.at>
Cc: Hotmail <crajac66@hotmail.com>
Cc: pgsql-admin@lists.postgresql.org
Subject: Re: Performance bottleneck. High active sessions but postmaster kernel threads are in a sleep state, low CPU utilization
Date: Wed, 31 Mar 2021 10:58:15 -0400
Message-ID: <41c4a64a-6c74-ba06-a914-c175790a7637@sqlexec.com> (raw)
In-Reply-To: <11921e4e3efa7782168b8e02b89d004ba5b6b00a.camel@cybertec.at>
References: <ef99be8f-149a-477b-92bb-1aac5d6e7cfa@Spark>
<BYAPR06MB4023F39ED590D9E1179414C3D77D9@BYAPR06MB4023.namprd06.prod.outlook.com>
<11921e4e3efa7782168b8e02b89d004ba5b6b00a.camel@cybertec.at>
Hi,
You showed a lot of detail from the operating system perspective, but
nothing from pg_stat_activity or pg_locks during those contention times
where the cpu is low but lot of concurrent activity in the database.
I would guess that there is a lot of lightweight and/or spinlocks going
on. This activity usually occurs when there appears to be no heavy cpu
activity, no cpu io wait activity, but the db is being hit big time with
a bunch of concurrent active connections contending over the same
resources as Laurenz mentioned.
You can identify lightweight locks/spinlocks with the general wait type:
wait_event_type = LWLock. The wait_event column will drill down even
further. You can also help yourself by installing and using
pg_wait_sampling extension. Because locks come and go so fast in
pg_stat_activity, this historical perspective from the sampling
extension may provide a better picture.
https://github.com/postgrespro/pg_wait_sampling
Regards,
Michael Vitale
Laurenz Albe wrote on 3/31/2021 5:47 AM:
> On Tue, 2021-03-30 at 12:22 -0600, Hotmail wrote:
>> We are trying to gain some insight into a performance bottleneck that we are hitting while load testing Postgres on 11.11.
>> [hundreds of active sessions, but CPU is not maxed out]
> With that many active sessions you are probably hitting some contention inside
> the database. Look at "wait_event" and "wait_event_type" in pg_stat_activity.
>
> Yours,
> Laurenz Albe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: MichaelDBA@sqlexec.com, laurenz.albe@cybertec.at, crajac66@hotmail.com, pgsql-admin@lists.postgresql.org
Subject: Re: Performance bottleneck. High active sessions but postmaster kernel threads are in a sleep state, low CPU utilization
In-Reply-To: <41c4a64a-6c74-ba06-a914-c175790a7637@sqlexec.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox