MIME-Version: 1.0
References: 
 <CA+fnDAa8Kbbx3KkqpfxutTx41RchJsT3Lykqwcedm0CVr5Tw=A@mail.gmail.com>
 <CAFC+b6pQXYMks5f4XJ-QX4QpxnYnrNz9Zrkg8EiJ5-O+PxKxBQ@mail.gmail.com>
 <CAO6_Xqpzj_V1gF0ikPCVkA3Q2K5YqRnAdHiV5TqpLNAZsdwWbQ@mail.gmail.com>
 <CAKZiRmxxD5y2hArj3p4Ueq2jvQoTUsUr7tjeK44OnSqxAD1ZmA@mail.gmail.com>
 <20260708194951.34705f93@ardentperf.com>
In-Reply-To: <20260708194951.34705f93@ardentperf.com>
From: Anthonin Bonnefoy <anthonin.bonnefoy@datadoghq.com>
Date: Thu, 9 Jul 2026 09:09:25 +0200
Message-ID: 
 <CAO6_XqrUbM+FNMHGksmezKNPftwD41EwW6Li=iqfzr-GmcF+fg@mail.gmail.com>
Subject: Re: hang during shutdown
To: Jeremy Schneider <schneider@ardentperf.com>
Cc: Jakub Wartak <jakub.wartak@enterprisedb.com>,
	Srinath Reddy Sadipiralla <srinath2133@gmail.com>,
	PostgreSQL mailing lists <pgsql-hackers@postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
 <https://www.postgresql.org/message-id/CAO6_XqrUbM%2BFNMHGksmezKNPftwD41EwW6Li%3Diqfzr-GmcF%2Bfg%40mail.gmail.com>
Precedence: bulk

On Thu, Jul 9, 2026 at 4:49=E2=80=AFAM Jeremy Schneider
<schneider@ardentperf.com> wrote:
> Between log messages and wal contents, it appears
> PostmasterStateMachine transitioned into PM_WAIT_BACKENDS but never
> progressed beyond this. I don't think walsenders are involved until it
> transitions into PM_WAIT_XLOG_SHUTDOWN which doesn't start until after
> the shutdown checkpoint (which never arrived in the wal).
>
> I remain pretty stumped :)

The walsender logical bug happens when pmState is in PM_SHUTDOWN,
after sending a SIGUSR2 to the checkpointer.
The checkpointer then asks the walsender to stop through
WalSndInitStopping() in ShutdownXLOG and wait for them to finish,
before creating the shutdown checkpoint. If the walsender is stuck in
an infinite loop, the shutdown sequence is stuck in this state.

As you didn't have a shutdown checkpoint, the symptoms look very
similar (though you didn't confirm whether you were using logical
replication, so I'm gonna assume you do).

Just looking at the logs won't tell you much, the most visible symptom
would be the walsender process stuck at 100%. Another signal would be
a OVERWRITE_CONTRECORD record written just after restart, as the last
record would have been partially. Looking at the last WAL record when
the instance was stuck could be interesting, though if it's a partial
record, it won't be visible with pg_waldump.

If you want to compare, you can reproduce the walsender bug with this
query (you also need an active logical walsender):

-- Start from a fresh segment
SELECT pg_switch_wal();
BEGIN;
-- A logical message record is:
--  24 bytes of record header
--  5 bytes fragment header
--  24 bytes xl_logical_message
--  2 bytes NULL for prefix + message
--  x bytes message length
-- Total size of a WAL logical message is 55 bytes + message length
-- With WAL long header,  the first  page has 8152 bytes available
-- With WAL short header, the second page has 8168 bytes available
--  We need to write 16320 bytes (+/- 8 bytes of alignment)
SELECT pg_logical_emit_message(false, '', repeat('a', 16294));
ROLLBACK;

And immediately try to restart (this needs to be the last WAL record),
the shutdown should hang.

Regards,
Anthonin Bonnefoy