Date: Wed, 8 Jul 2026 19:49:51 -0700
From: Jeremy Schneider <schneider@ardentperf.com>
To: Jakub Wartak <jakub.wartak@enterprisedb.com>
Cc: Anthonin Bonnefoy <anthonin.bonnefoy@datadoghq.com>, Srinath Reddy
 Sadipiralla <srinath2133@gmail.com>, PostgreSQL mailing lists
 <pgsql-hackers@postgresql.org>
Subject: Re: hang during shutdown
Message-ID: <20260708194951.34705f93@ardentperf.com>
In-Reply-To: 
 <CAKZiRmxxD5y2hArj3p4Ueq2jvQoTUsUr7tjeK44OnSqxAD1ZmA@mail.gmail.com>
References: 
 <CA+fnDAa8Kbbx3KkqpfxutTx41RchJsT3Lykqwcedm0CVr5Tw=A@mail.gmail.com>
	<CAFC+b6pQXYMks5f4XJ-QX4QpxnYnrNz9Zrkg8EiJ5-O+PxKxBQ@mail.gmail.com>
	<CAO6_Xqpzj_V1gF0ikPCVkA3Q2K5YqRnAdHiV5TqpLNAZsdwWbQ@mail.gmail.com>
	<CAKZiRmxxD5y2hArj3p4Ueq2jvQoTUsUr7tjeK44OnSqxAD1ZmA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Archived-At: 
 <https://www.postgresql.org/message-id/20260708194951.34705f93%40ardentperf.com>
Precedence: bulk

On Mon, 6 Jul 2026 14:42:10 +0200
Jakub Wartak <jakub.wartak@enterprisedb.com> wrote:

> On Mon, Jul 6, 2026 at 8:30=E2=80=AFAM Anthonin Bonnefoy
> <anthonin.bonnefoy@datadoghq.com> wrote:
> >
> > Hi,
> >
> > On Sun, Jul 5, 2026 at 11:55=E2=80=AFPM Jeremy Schneider
> > <schneider@ardentperf.com> wrote:
> >
> > Did you have an active logical replication stream at the time of the
> > shutdown? If so, it's very likely due to a bug in the logical
> > walsender[..]
>=20
> Aren't walsenders in general problematic for shutdowns? (not just
> logical, of course it might be this). The PG 19 is going to have
> wal_sender_shutdown_timeout, and from description it seems to match
> [0] to something like that. 'netstat -ntpo' would help to see from
> such hanged system (if walsenders running there at this point at all).

Between log messages and wal contents, it appears
PostmasterStateMachine transitioned into PM_WAIT_BACKENDS but never
progressed beyond this. I don't think walsenders are involved until it
transitions into PM_WAIT_XLOG_SHUTDOWN which doesn't start until after
the shutdown checkpoint (which never arrived in the wal).

I remain pretty stumped :)

I have put some code together that can automatically capture cmdline,
comm, status, wchan, io, syscall, stack and sched from /proc for every
single process in the postgres namespace/container. If privs are
missing (stack is probably usually blocked) then it'll still capture
other files. My initial prototype does three captures, with 3 seconds
in between each. This can help determine whether processes or stuck or
slow.

I'm still figuring out how to trigger it at the right time, because the
SIGKILL was triggered by terminationGracePeriodSeconds on kubernetes. I
didn't find an obvious way to just run diag at the end (I think preStop
hook fires at the beginning). I'll figure something out.

-Jeremy


--=20
To know the thoughts and deeds that have marked man's progress is to
feel the great heart throbs of humanity through the centuries; and if
one does not feel in these pulsations a heavenward striving, one must
indeed be deaf to the harmonies of life.

Helen Keller, The Story Of My Life, 1902, 1903, 1905, introduction by
Ralph Barton Perry (Garden City, NY: Doubleday & Company, 1954), p90.