From: Justin Pryzby <pryzby@telsasoft.com>
To: James Pang (chaolpan) <chaolpan@cisco.com>
Cc: pgsql-performance@lists.postgresql.org
Subject: Re: postgres backend process hang on " D " state
Date: Sun, 29 May 2022 21:19:44 -0500
Message-ID: <20220530021943.GP19626@telsasoft.com> (raw)
In-Reply-To: <PH0PR11MB5191C0B3BE53A16AC7EED9FED6DD9@PH0PR11MB5191.namprd11.prod.outlook.com>
References: <PH0PR11MB5191521AF97E422D288DB9DBD6DA9@PH0PR11MB5191.namprd11.prod.outlook.com>
<20220529150157.GM19626@telsasoft.com>
<PH0PR11MB5191C0B3BE53A16AC7EED9FED6DD9@PH0PR11MB5191.namprd11.prod.outlook.com>
On Mon, May 30, 2022 at 01:19:56AM +0000, James Pang (chaolpan) wrote:
> 1. extensions
> shared_preload_libraries = 'orafce,pgaudit,pg_cron,pg_stat_statements,set_user'
> 2. psql can not login now ,it hang there too, so can not check anything from pg_stats_* views
> 3. one main app user and 2 schemas ,no long running transactions .
> 4. we use /pgdata , it's on xfs , lvm/vg RHEL8.4 ,it's a shared storage, no use root filesystem.
> /dev/mapper/pgdatavg-pgdatalv 500G 230G 271G 46% /pgdata
> /dev/mapper/pgdatavg-pgarchivelv 190G 1.5G 189G 1% /pgarchive
> /dev/mapper/pgdatavg-pgwallv 100G 34G 67G 34% /pgwal
What are the LVM PVs ? Is it a scsi/virt device ? Or iscsi/drbd/???
I didn't hear back if there's any kernel errors.
Is the storage broken/stuck/disconnected ?
Can you run "time find /pgdata /pgarchive /pgwal -ls |wc" ?
Could you run "ps -u postgres -O wchan============================="
Can you strace one of the stuck backends ?
It sounds like you'll have to restart the service or VM (forcibly if necessary)
to resolve the immediate issue and then collect the other info, and leave a
"psql" open to try to (if the problem recurs) check pg_stat_activity and other
DB info.
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: pryzby@telsasoft.com, chaolpan@cisco.com, pgsql-performance@lists.postgresql.org
Subject: Re: postgres backend process hang on " D " state
In-Reply-To: <20220530021943.GP19626@telsasoft.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox