MIME-Version: 1.0
References: 
 <CA+fnDAa8Kbbx3KkqpfxutTx41RchJsT3Lykqwcedm0CVr5Tw=A@mail.gmail.com>
 <CAFC+b6pQXYMks5f4XJ-QX4QpxnYnrNz9Zrkg8EiJ5-O+PxKxBQ@mail.gmail.com>
 <CAO6_Xqpzj_V1gF0ikPCVkA3Q2K5YqRnAdHiV5TqpLNAZsdwWbQ@mail.gmail.com>
 <CAKZiRmxxD5y2hArj3p4Ueq2jvQoTUsUr7tjeK44OnSqxAD1ZmA@mail.gmail.com>
 <20260708194951.34705f93@ardentperf.com>
 <CAO6_XqrUbM+FNMHGksmezKNPftwD41EwW6Li=iqfzr-GmcF+fg@mail.gmail.com>
 <20260709221955.50de31cf@ardentperf.com>
In-Reply-To: <20260709221955.50de31cf@ardentperf.com>
From: Anthonin Bonnefoy <anthonin.bonnefoy@datadoghq.com>
Date: Fri, 10 Jul 2026 08:49:17 +0200
Message-ID: 
 <CAO6_Xqqw1jJfu5FdSgaUpWvaJipCoZp=RS-2qDaOJg8-iQiohA@mail.gmail.com>
Subject: Re: hang during shutdown
To: Jeremy Schneider <schneider@ardentperf.com>
Cc: Jakub Wartak <jakub.wartak@enterprisedb.com>,
	Srinath Reddy Sadipiralla <srinath2133@gmail.com>,
	PostgreSQL mailing lists <pgsql-hackers@postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
 <https://www.postgresql.org/message-id/CAO6_Xqqw1jJfu5FdSgaUpWvaJipCoZp%3DRS-2qDaOJg8-iQiohA%40mail.gmail.com>
Precedence: bulk

On Fri, Jul 10, 2026 at 7:20=E2=80=AFAM Jeremy Schneider
<schneider@ardentperf.com> wrote:
>
> On Thu, 9 Jul 2026 09:09:25 +0200
> Anthonin Bonnefoy <anthonin.bonnefoy@datadoghq.com> wrote:
>
> > As you didn't have a shutdown checkpoint, the symptoms look very
> > similar (though you didn't confirm whether you were using logical
> > replication, so I'm gonna assume you do).
>
> No logical replication. Are we sure this only impacts logical and not
> physical? Thanks for the pointer - interesting one.

Yeah, this is only for logical replication. If you didn't have one,
then it's unrelated.

> We do use physical replication. Replicas were shut down and they closed
> their connections to this writer instance around 04:25 just before
> pg_ctl stop was executed on the writer itself.
>
> Something was on the writer DB's CPU before shutdown; metrics show total
> CPU for that linux namespace (k8s pod) around 0.86 cores until 04:24 and
> then the CPU starts dropping when the database shuts down. Bu 04:28 the
> CPU is under 0.01 cores and the CPU remains effectively zero until the
> processes are SIGKILL'd 30 minutes later.

Looking at the last WAL record that was on the whole during shutdown
could be interesting information, if you have it available. If the
shutdown was stuck due to a specific state of the WAL, it could give
you some leads.
Other information could be useful, like the types of writes that
happened after you started the shutdown, or to confirm whether the
RUNNING_XACT you mention is indeed from the autovac and whether it
closed the transaction or not.