Message-ID: <7b0ac868e97508aac3661a6a665fea00be2b923e.camel@cybertec.at>
Subject: Re: Postgresql 16.9 fast shutdown hangs with walsenders eating 100%
 CPU
From: Laurenz Albe <laurenz.albe@cybertec.at>
To: Klaus Darilion <klaus.darilion@nic.at>, 
 "pgsql-general@lists.postgresql.org"
	 <pgsql-general@lists.postgresql.org>
Date: Mon, 21 Jul 2025 15:35:11 +0200
In-Reply-To: <DBAPR03MB6358854AD71C8ABA5CA10A8DF15DA@DBAPR03MB6358.eurprd03.prod.outlook.com>
References: 
	<DBAPR03MB6358854AD71C8ABA5CA10A8DF15DA@DBAPR03MB6358.eurprd03.prod.outlook.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.56.2 (3.56.2-1.fc42) 
MIME-Version: 1.0
Archived-At: <https://www.postgresql.org/message-id/7b0ac868e97508aac3661a6a665fea00be2b923e.camel%40cybertec.at>
Precedence: bulk

On Mon, 2025-07-21 at 10:47 +0000, Klaus Darilion wrote:
> (Note: I have also attached the whole email for better readability of the=
 logs)

Your mail looks good enough the way it is:
https://postgr.es/m/DBAPR03MB6358854AD71C8ABA5CA10A8DF15DA%40DBAPR03MB6358.=
eurprd03.prod.outlook.com

> Our setup: 5 Node Patroni Cluster with PostgreSQL 16.9.
> db1: current leader
> db2: sync-replica
> db3/4/5: replica
> =C2=A0
> The replicas connect to the leader using the host IP of the leader. So th=
ere are
> 4 walsender for patroni, 1 sync and 3 async.
> =C2=A0
> The patroni cluster utilizes a service IP-address (VIP). The VIP is used =
by all
> clients connecting to the current leader. These clients are:
> - some web-apps doing normal DB queries (read/write)
> - 2 barman backup clients using streaming replication
> - 58 logical replication clients
> =C2=A0
> Additionally we use https://github.com/EnterpriseDB/pg_failover_slots to =
sync and
> advance the logical replication slots on the replicas. The failover_slots=
 plugin
> periodically connects to leader using the VIP.
> =C2=A0
> We had a planned maintenance and wanted to switch the leader from db1 to =
db2:
> 12:32:18: patronictl switchover --leader db1 --candidate db2
> =C2=A0
> So postmaster received the fast shutdown request from Patroni and started
> shutting down the client connection processes:
> =C2=A0
> Usually the switchover only takes a few seconds. After waiting a few minu=
tes
> we became anxious and started debugging.
> =C2=A0
> Using "ps -Alf|grep postgres" we saw that there were no more normal clien=
t
> connections, but still 58 logical replicaton walsender processes and
> 6 streaming replication walsenders.
> "top" revealed that the walsenders were eating CPU.

We have had a somewhat similar report:
https://www.postgresql.org/message-id/flat/18985-64431d78bcabae95%40postgre=
sql.org

What is the logical decoding plugin you are using?

If it is "pgoutput", what are the walsenders doing? You can try "strace" an=
d
use "gdb" to break into the walsenders and take a stack trace.

Yours,
Laurenz Albe