From: Hotmail <crajac66@hotmail.com>
To: pgsql-admin@lists.postgresql.org, Laurenz Albe <laurenz.albe@cybertec.at>
Subject: Re: Performance bottleneck. High active sessions but postmaster kernel threads are in a sleep state, low CPU utilization
Date: Wed, 31 Mar 2021 10:22:09 -0600
Message-ID: <BYAPR06MB4023DF70385EC0C8E3E50C63D77C9@BYAPR06MB4023.namprd06.prod.outlook.com> (raw)
In-Reply-To: <BYAPR06MB4023DC744EFE7E1E55BA41C0D77C9@BYAPR06MB4023.namprd06.prod.outlook.com>
References: <ef99be8f-149a-477b-92bb-1aac5d6e7cfa@Spark>
<BYAPR06MB4023F39ED590D9E1179414C3D77D9@BYAPR06MB4023.namprd06.prod.outlook.com>
<11921e4e3efa7782168b8e02b89d004ba5b6b00a.camel@cybertec.at>
<BYAPR06MB4023DC744EFE7E1E55BA41C0D77C9@BYAPR06MB4023.namprd06.prod.outlook.com>
We actually keep track of the wait events in another grafana graph. We use the following query to generate the graph. (We assume that an active session that has a NULL wait_event is "ON CPU”. Not sure if our assuming an active session with a null wait being on cpu is valid. We sample pg_stat_activity every 30 seconds.
SELECT
coalesce(wait_event, 'ON CPU') AS type,
count(*)
FROM
pg_stat_activity
WHERE
state = 'active'
AND usename != 'repmgr'
GROUP BY
TYPE
ORDER BY
count(*) DESC ";
Here’s a sample from our wait_event graph during times when we see high active sessions but a sleeping postgres OS process. Is it possible we could be hitting an un-instremented wait event?
Craig
On Mar 31, 2021, 10:00 AM -0600, Hotmail <crajac66@hotmail.com>, wrote:
> Unfortunately, the wait_event and wait_event_type columns are all NULL for these active sessions.
>
> Craig
> On Mar 31, 2021, 3:47 AM -0600, Laurenz Albe <laurenz.albe@cybertec.at>, wrote:
> > On Tue, 2021-03-30 at 12:22 -0600, Hotmail wrote:
> > > We are trying to gain some insight into a performance bottleneck that we are hitting while load testing Postgres on 11.11.
> > > [hundreds of active sessions, but CPU is not maxed out]
> >
> > With that many active sessions you are probably hitting some contention inside
> > the database. Look at "wait_event" and "wait_event_type" in pg_stat_activity.
> >
> > Yours,
> > Laurenz Albe
> > --
> > Cybertec | https://www.cybertec-postgresql.com
> >
Attachments:Screen Shot 2021-03-31 at 10.18.01 AM.png (image/png, 108.5K) downloadview image
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: crajac66@hotmail.com, laurenz.albe@cybertec.at
Subject: Re: Performance bottleneck. High active sessions but postmaster kernel threads are in a sleep state, low CPU utilization
In-Reply-To: <BYAPR06MB4023DF70385EC0C8E3E50C63D77C9@BYAPR06MB4023.namprd06.prod.outlook.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox