agora inbox for pgsql-bugs@postgresql.org
help / color / mirror / Atom feedBUG #19623: Postmaster livelocks respawning io workers when children die after crash restart; pg_ctl stop fails
5+ messages / 3 participants
[nested] [flat]
* BUG #19623: Postmaster livelocks respawning io workers when children die after crash restart; pg_ctl stop fails
@ 2026-08-17 04:13 PG Bug reporting form <noreply@postgresql.org>
2026-08-20 01:28 ` Re: BUG #19623: Postmaster livelocks respawning io workers when children die after crash restart; pg_ctl stop fails Zexin Li <lizi.openmind@gmail.com>
0 siblings, 1 reply; 5+ messages in thread
From: PG Bug reporting form @ 2026-08-17 04:13 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org; +Cc: m0935388420@gmail.com
The following bug has been logged on the website:
Bug reference: 19623
Logged by: KUAN-TING KUO
Email address: m0935388420@gmail.com
PostgreSQL version: 18.4
Operating system: Windows 11 Pro for Workstations 10.0.26200 (x64)
Description:
PostgreSQL version: 18.4 ("PostgreSQL 18.4 on x86_64-windows, compiled by
msvc-19.44.35228, 64-bit"; conda-forge build)
Operating system: Windows 11 Pro for Workstations 10.0.26200 (x64)
Configuration: initdb defaults except port/listen_addresses; io_method
= worker (default), io_workers = 3 (default)
Note on version: tested on 18.4 (the build I have); the code path
described below is
unchanged in the REL_18_6 tag / REL_18_STABLE
(postmaster.c, the io worker
branch of process_pm_child_exit(), lines 2504-2512 in
REL_18_6).
Summary
-------
If, after a crash restart ("all server processes terminated;
reinitializing"),
the postmaster's freshly spawned children die immediately at process start,
the postmaster on 18.4 goes into a hot loop that:
* respawns io workers ~3 times per second, forever;
* writes nothing further to the server log;
* keeps the listen socket open (postmaster.pid still says "ready",
pg_ctl status says "server is running"), while every connection is
reset (pg_isready exit 2);
* dispatches signals only every couple of minutes, so `pg_ctl reload`
took 134 s to be honoured and `pg_ctl stop -m fast` / `-m immediate`
with timeouts of 10-240 s reported "server does not shut down";
only TerminateProcess ended it.
The condition that made every child die is Windows-specific (see
"How I hit it"), but the postmaster behaviour it exposes looks like a
generic
18.x issue in process_pm_child_exit(): the reap loop calls
maybe_adjust_io_workers() synchronously for every dead io worker, so as long
as a replacement worker dies faster than the next CreateProcess()/fork()
completes, the `while ((pid = waitpid(-1, ...)) > 0)` loop never drains and
control never returns to ServerLoop() (no WaitEventSetWait -> no signal
dispatch, no LaunchMissingBackgroundProcesses, no shutdown handling).
HandleChildCrash() returns early because FatalError is still set from the
first crash, so none of these deaths is logged.
How I hit it (Windows)
----------------------
The postmaster had been started with `pg_ctl -w start` from a cmd.exe that
owned a (hidden) console. That console's conhost.exe was later killed
together
with the cmd.exe (a `taskkill /T /F` on the cmd process tree; conhost.exe is
a
child of the console-owning process, the postmaster is not, so the
postmaster
survived attached to a console whose server is gone;
AttachConsole(postmaster)
from another process fails with error 233 ERROR_PIPE_NOT_CONNECTED). From
then
on every process the postmaster creates with CreateProcess() inherits that
dead
console and dies during process initialisation with exit status 0xC0000142
(STATUS_DLL_INIT_FAILED). The postmaster itself keeps running and, until a
child needs to be spawned, still works (a SIGHUP sent in this state was
logged
immediately).
Reproduction (18.4, Windows; deterministic, done 4/4 times on a fresh
initdb)
------------------------------------------------------------------------------
1. initdb -D data -U postgres -A trust --no-locale -E UTF8; set port = 5499,
listen_addresses = '127.0.0.1'.
2. From Python, start a hidden console whose cmd.exe runs
`pg_ctl -D data -l server.log -w start` and then lingers:
subprocess.Popen(["cmd.exe", "/c", "owner.bat"],
creationflags=CREATE_NEW_CONSOLE,
startupinfo=<SW_HIDE>)
Wait for pg_isready = 0.
3. `taskkill /T /F /PID <that cmd.exe pid>` -> kills cmd.exe and its child
conhost.exe; postmaster survives (verify: AttachConsole(pid) -> 233).
4. `psql -h 127.0.0.1 -p 5499 -U postgres -c "select 1"` ->
"server closed the connection unexpectedly". server.log:
LOG: client backend (PID 52868) was terminated by exception
0xC0000142
HINT: See C include file "ntstatus.h" for a description of the
hexadecimal value.
LOG: terminating any other active server processes
LOG: all server processes terminated; reinitializing
and nothing after that.
5. Observe (numbers from one run, all runs alike):
- postmaster main thread ~80-100 % of one core; sampled 20x with a
GetThreadContext-based sampler: every sample inside
KERNELBASE!CreateProcessInternalW (NtCreateUserProcess /
BasepQueryAppCompat / CsrClientCallServer), never in a wait.
- 72 distinct child postgres.exe processes appeared in 20 s; opening each
one and waiting: all 72 exited with 0xC0000142. While still suspended,
each child already had the postmaster's 150 MB shared-memory range
reserved (VirtualQueryEx: RESERVE/PRIVATE at the same base), i.e.
pgwin32_ReserveSharedMemoryRegion() succeeded and the child was
resumed;
this is not the ASLR/487 retry loop.
- `pg_ctl reload` (SIGHUP) sent right after step 4 was logged
("received SIGHUP, reloading configuration files") only 134 s later.
- `pg_ctl stop -m immediate -t 240` -> "server does not shut down" after
261 s; server.log grew by nothing but that one SIGHUP line.
- port still LISTENING, postmaster.pid still present with status "ready",
pg_isready -> 2 throughout.
Where I think the loop is (src/backend/postmaster/postmaster.c,
REL_18_STABLE)
------------------------------------------------------------------------------
process_pm_child_exit():
while ((pid = waitpid(-1, &exitstatus, WNOHANG)) > 0)
{
...
/* Was it an IO worker? */
if (maybe_reap_io_worker(pid))
{
if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
HandleChildCrash(pid, exitstatus, _("io worker"));
maybe_adjust_io_workers(); <-- spawns a replacement
here
continue;
}
maybe_adjust_io_workers() spawns synchronously (`while (io_worker_count <
io_workers) StartChildProcess(B_IO_WORKER)`). On this machine one
CreateProcess() of postgres.exe takes ~300-450 ms while a child that fails
initialisation is dead a few ms after ResumeThread(). So by the time the
freshly spawned worker's launch returns, the previously spawned worker has
already died and its exit is sitting in the win32 waitpid() completion
queue:
the while loop finds another dead io worker, spawns another replacement, and
so
on. The loop only exits on the rare occasion that a death has not been
queued
yet when waitpid() polls, which is why signals were serviced roughly every
two
minutes rather than never.
HandleChildCrash() (same file) begins with
if (FatalError || Shutdown == ImmediateShutdown)
return;
and FatalError is only cleared when the startup process completes, so every
death after "reinitializing" is silent. (Side note, not 18-specific: the
startup process itself dies the same way in this state; StartupStatus
becomes
STARTUP_CRASHED but the "shutting down due to startup process failure" exit
is
only reached from PM_NO_CHILDREN, and nothing moves pmState away from
PM_STARTUP once HandleChildCrash() returns early, so even without io workers
the postmaster would sit in PM_STARTUP indefinitely with no log entry.
REL_16/REL_17 have the same shape there.)
Expected behaviour
------------------
Either the postmaster should give up (as the comment above the
STARTUP_CRASHED
check says: "we don't try to reinitialize when the startup process fails,
because more than likely it will just fail again and we will keep trying
forever"), or at least: repeated child deaths after a crash restart should
be
logged, replacement io workers should not be spawned from inside the reap
loop
(deferring to LaunchMissingBackgroundProcesses() would let signal handling
and
shutdown requests run between attempts), and some backoff/limit should apply
so a persistently failing child kind cannot monopolise the postmaster.
I can rerun the reproduction with additional instrumentation or provide a
minidump of the spinning postmaster on request.
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: BUG #19623: Postmaster livelocks respawning io workers when children die after crash restart; pg_ctl stop fails
2026-08-17 04:13 BUG #19623: Postmaster livelocks respawning io workers when children die after crash restart; pg_ctl stop fails PG Bug reporting form <noreply@postgresql.org>
@ 2026-08-20 01:28 ` Zexin Li <lizi.openmind@gmail.com>
2026-08-20 01:44 ` Re: BUG #19623: Postmaster livelocks respawning io workers when children die after crash restart; pg_ctl stop fails Michael Paquier <michael@paquier.xyz>
2026-08-20 07:06 ` Re: BUG #19623: Postmaster livelocks respawning io workers when children die after crash restart; pg_ctl stop fails Michael Paquier <michael@paquier.xyz>
0 siblings, 2 replies; 5+ messages in thread
From: Zexin Li @ 2026-08-20 01:28 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org; +Cc: m0935388420@gmail.com; michael@paquier.xyz; ayushtiwari.slg01@gmail.com
On Mon, Aug 17, 2026, KUAN-TING KUO wrote:
> (Side note, not 18-specific: the startup process itself dies the same
> way in this state; StartupStatus becomes STARTUP_CRASHED but the
> "shutting down due to startup process failure" exit is only reached
> from PM_NO_CHILDREN, and nothing moves pmState away from PM_STARTUP
> once HandleChildCrash() returns early, so even without io workers the
> postmaster would sit in PM_STARTUP indefinitely with no log entry.
> REL_16/REL_17 have the same shape there.)
Thanks for the report. I think this side note, rather than the io
worker respawn, is the actual bug. I reproduced the silent respawn
loop on Linux master with a hack that makes every child die in
InitPostmasterChild(), and it is unchanged with io_method=sync: the
respawned children are then the checkpointer and the background
writer, but the loop and the silence stay the same. Moving the io
worker respawn out of the reap loop didn't change anything observable
either. (Same with io_method=sync on a stock 18.6 on Windows
following your recipe -- though in my runs the postmaster stayed
responsive to pg_ctl throughout, so I can't speak to the
shutdown-hang part of the report.)
The reason nothing is logged and nothing moves the state machine is
that this state is only reachable while FatalError is set: FatalError
is cleared when WAL redo starts (PMSIGNAL_RECOVERY_STARTED), so a
startup process that crashes during reinitialization before that
point leaves HandleChildCrash() a no-op, as you describe.
This case used to be caught earlier. Until commit 9b43e6793b0f
(affdb2dd5c67 on REL_18_STABLE, first released in 18.4, backpatched
through v15), process_pm_child_exit() had a PM_STARTUP shortcut that
logged "aborting startup due to startup process failure" and exited,
without consulting FatalError. That commit removed the shortcut to
fix a real problem -- the direct exit orphaned the checkpointer and
background writer, which have been running during PM_STARTUP since
v15 -- but the path it falls back to does nothing in the
FatalError-still-set case, so the give-up behavior was lost. On a
build of that commit's parent, the same scenario ends after one
reinitialization cycle with
LOG: startup process (PID 46910) exited with exit code 2
LOG: aborting startup due to startup process failure
and a postmaster exit, as before.
The attached patch restores the give-up while keeping the
orphaned-children fix: if the startup process crashes while
FatalError is still set, log the exit and go through
HandleFatalError(), so the remaining children are signalled and the
existing STARTUP_CRASHED check at PM_NO_CHILDREN terminates the
postmaster. HandleFatalError() loses its Assert(!FatalError) for
that; the repeated call re-signals children launched since the
previous call, which this path wants anyway. Nothing is lost by
exiting: in the wedged state the startup process is never relaunched,
so the server could never have recovered on its own.
With the patch, the reproduction ends about 100 ms after the startup
crash with
LOG: startup process (PID 19613) exited with exit code 2
LOG: aborting startup due to startup process failure
LOG: shutting down due to startup process failure
(12 forks in total, versus 40,000+ in six seconds without the patch;
without the patch these lines never appear). make check passes, and a
SIGKILLed backend on a healthy server still goes through a normal,
logged crash restart and recovers.
Adding Michael and Ayush in CC, as committer and author of
9b43e6793b0f.
Regards,
Zexin Li
Attachments:
[application/octet-stream] 0001-Fix-postmaster-wedge-when-startup-process-crashes-du.patch (4.6K, ../../CAAP6ZkS9HXN_3VVaF7yB2oeamHKs-E9_9sYDkQv+fu_hL4mr9Q@mail.gmail.com/2-0001-Fix-postmaster-wedge-when-startup-process-crashes-du.patch)
download | inline diff:
From a52a89204e7664d4a60e6f07fca92ae2ecf1c840 Mon Sep 17 00:00:00 2001
From: Zexin Li <lizi.openmind@gmail.com>
Date: Thu, 20 Aug 2026 00:43:06 +0000
Subject: [PATCH] Fix postmaster wedge when startup process crashes during
crash restart
Commit 9b43e6793b0f removed the PM_STARTUP shortcut that made the
postmaster exit when the startup process died unexpectedly, relying on
the regular crash path (HandleChildCrash()) to also cover this case, so
that the other children that may now be running during PM_STARTUP are
cleaned up rather than orphaned.
However, HandleChildCrash() returns without doing anything when
FatalError is already set, which is exactly the state when a relaunched
startup process crashes during a crash restart before WAL redo has
started (receipt of PMSIGNAL_RECOVERY_STARTED is what clears
FatalError). In that case nothing moves the state machine off
PM_STARTUP: StartupStatus is set to STARTUP_CRASHED, but the only place
consulting that flag (PM_NO_CHILDREN handling) is never reached, and
the postmaster keeps relaunching the checkpointer, the background
writer, and the IO workers forever, without writing anything to the
log. In the report, a Windows postmaster whose console had gone away
-- so that every child it created died with 0xC0000142 during process
initialization -- burned a full CPU core silently respawning children,
while pg_ctl still reported the server as running and shutdown requests
were not honored.
Fix by restoring the historical give-up behavior for this case: when
the startup process crashes while FatalError is still set, log the
child exit, signal the remaining children, and proceed towards
PM_NO_CHILDREN, where the existing STARTUP_CRASHED check makes the
postmaster exit. Unlike before 9b43e6793b0f, this goes through the
regular fatal-error path instead of exiting directly, so no children
are orphaned.
Bug: #19623
Reported-by: KUAN-TING KUO <m0935388420@gmail.com>
Author: Zexin Li <lizi.openmind@gmail.com>
Discussion: https://postgr.es/m/19623-f9bd331940be1273@postgresql.org
---
src/backend/postmaster/postmaster.c | 33 ++++++++++++++++++++++++-----
1 file changed, 28 insertions(+), 5 deletions(-)
diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 1711743f1a..2f0580699d 100644
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -432,6 +432,7 @@ static void process_pm_shutdown_request(void);
static void dummy_handler(SIGNAL_ARGS);
static void CleanupBackend(PMChild *bp, int exitstatus);
static void HandleChildCrash(int pid, int exitstatus, const char *procname);
+static void HandleFatalError(QuitSignalReason reason, bool consider_sigabrt);
static void LogChildExit(int lev, const char *procname,
int pid, int exitstatus);
static void PostmasterStateMachine(void);
@@ -2333,8 +2334,30 @@ process_pm_child_exit(void)
}
else
StartupStatus = STARTUP_CRASHED;
- HandleChildCrash(pid, exitstatus,
- _("startup process"));
+
+ /*
+ * If the startup process crashed while we were still
+ * reinitializing after a previous crash -- FatalError is
+ * still set, because this startup process died before WAL
+ * redo started -- then HandleChildCrash() would return
+ * without doing anything, and nothing else would ever move
+ * the state machine off PM_STARTUP: we would keep
+ * relaunching the remaining background processes forever,
+ * without logging anything. Instead, give up on startup:
+ * signal the remaining children and head for
+ * PM_NO_CHILDREN, where STARTUP_CRASHED makes us exit.
+ */
+ if (StartupStatus == STARTUP_CRASHED &&
+ FatalError && Shutdown != ImmediateShutdown)
+ {
+ LogChildExit(LOG, _("startup process"), pid, exitstatus);
+ ereport(LOG,
+ (errmsg("aborting startup due to startup process failure")));
+ HandleFatalError(PMQUIT_FOR_CRASH, true);
+ }
+ else
+ HandleChildCrash(pid, exitstatus,
+ _("startup process"));
continue;
}
@@ -2725,15 +2748,15 @@ CleanupBackend(PMChild *bp,
* happened. Commonly the caller will have logged the reason for entering
* FatalError state.
*
- * This should only be called when not already in FatalError or
- * ImmediateShutdown state.
+ * This should only be called when not already in ImmediateShutdown state.
+ * Calling it again while FatalError is already set is fine (and re-signals
+ * any children launched since the previous call).
*/
static void
HandleFatalError(QuitSignalReason reason, bool consider_sigabrt)
{
int sigtosend;
- Assert(!FatalError);
Assert(Shutdown != ImmediateShutdown);
SetQuitSignalReason(reason);
--
2.34.1
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: BUG #19623: Postmaster livelocks respawning io workers when children die after crash restart; pg_ctl stop fails
2026-08-17 04:13 BUG #19623: Postmaster livelocks respawning io workers when children die after crash restart; pg_ctl stop fails PG Bug reporting form <noreply@postgresql.org>
2026-08-20 01:28 ` Re: BUG #19623: Postmaster livelocks respawning io workers when children die after crash restart; pg_ctl stop fails Zexin Li <lizi.openmind@gmail.com>
@ 2026-08-20 01:44 ` Michael Paquier <michael@paquier.xyz>
2026-08-20 02:21 ` Re: BUG #19623: Postmaster livelocks respawning io workers when children die after crash restart; pg_ctl stop fails Michael Paquier <michael@paquier.xyz>
1 sibling, 1 reply; 5+ messages in thread
From: Michael Paquier @ 2026-08-20 01:44 UTC (permalink / raw)
To: Zexin Li <lizi.openmind@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org; m0935388420@gmail.com; ayushtiwari.slg01@gmail.com
On Thu, Aug 20, 2026 at 10:28:53AM +0900, Zexin Li wrote:
> Thanks for the report. I think this side note, rather than the io
> worker respawn, is the actual bug. I reproduced the silent respawn
> loop on Linux master with a hack that makes every child die in
> InitPostmasterChild(), and it is unchanged with io_method=sync:
Would you mind sharing that, as a matter of reproducibility?
> Adding Michael and Ayush in CC, as committer and author of
> 9b43e6793b0f.
Thanks for the poke, I'll look into that.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../aoZb624sa9KRsRG4@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: BUG #19623: Postmaster livelocks respawning io workers when children die after crash restart; pg_ctl stop fails
2026-08-17 04:13 BUG #19623: Postmaster livelocks respawning io workers when children die after crash restart; pg_ctl stop fails PG Bug reporting form <noreply@postgresql.org>
2026-08-20 01:28 ` Re: BUG #19623: Postmaster livelocks respawning io workers when children die after crash restart; pg_ctl stop fails Zexin Li <lizi.openmind@gmail.com>
2026-08-20 01:44 ` Re: BUG #19623: Postmaster livelocks respawning io workers when children die after crash restart; pg_ctl stop fails Michael Paquier <michael@paquier.xyz>
@ 2026-08-20 02:21 ` Michael Paquier <michael@paquier.xyz>
0 siblings, 0 replies; 5+ messages in thread
From: Michael Paquier @ 2026-08-20 02:21 UTC (permalink / raw)
To: Zexin Li <lizi.openmind@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org; m0935388420@gmail.com; ayushtiwari.slg01@gmail.com
On Thu, Aug 20, 2026 at 10:44:11AM +0900, Michael Paquier wrote:
> On Thu, Aug 20, 2026 at 10:28:53AM +0900, Zexin Li wrote:
>> Thanks for the report. I think this side note, rather than the io
>> worker respawn, is the actual bug. I reproduced the silent respawn
>> loop on Linux master with a hack that makes every child die in
>> InitPostmasterChild(), and it is unchanged with io_method=sync:
>
> Would you mind sharing that, as a matter of reproducibility?
Never mind, I have reproduced it reusing a marker file in
InitPostmasterChild() to force an exit(), and then got a postmaster
spinning. :)
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../aoZku8LRQ59z1PKp@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: BUG #19623: Postmaster livelocks respawning io workers when children die after crash restart; pg_ctl stop fails
2026-08-17 04:13 BUG #19623: Postmaster livelocks respawning io workers when children die after crash restart; pg_ctl stop fails PG Bug reporting form <noreply@postgresql.org>
2026-08-20 01:28 ` Re: BUG #19623: Postmaster livelocks respawning io workers when children die after crash restart; pg_ctl stop fails Zexin Li <lizi.openmind@gmail.com>
@ 2026-08-20 07:06 ` Michael Paquier <michael@paquier.xyz>
1 sibling, 0 replies; 5+ messages in thread
From: Michael Paquier @ 2026-08-20 07:06 UTC (permalink / raw)
To: Zexin Li <lizi.openmind@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org; m0935388420@gmail.com; ayushtiwari.slg01@gmail.com
On Thu, Aug 20, 2026 at 10:28:53AM +0900, Zexin Li wrote:
> HandleFatalError() loses its Assert(!FatalError) for
> that; the repeated call re-signals children launched since the
> previous call, which this path wants anyway. Nothing is lost by
> exiting: in the wedged state the startup process is never relaunched,
> so the server could never have recovered on its own.
That's also exactly the reason why this impacts only v18 and newer
versions. HandleFatalError() could be called multiple times before
f0b7ab725139, not after it.
Re-adding the shortcut of the startup process that 9b43e6793b0f has
deleted to act as a replacement of HandleChildCrash() when FatalError
is set, leaving the early exit HandleChildCrash() intact works here at
the end. We could edit HandleChildCrash() so as the state machine
advances if we are under pmState == PM_STARTUP, or just give up on
HandleChildCrash() for the startup process entirely, but I cannot get
much excited about that based on what was looking for. We cannot do
an ExitPostmaster() either, or we would be exposed again to the
orphaned process problems that 9b43e6793b0f has addressed (we are
still OK after this patch, retesting a startup failure with a zeroed
WAL segment, test posted on the thread of 9b43e6793b0f to emulate the
orphan case).
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../aoanbLkHeIwcmj1I@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2026-08-20 07:06 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-08-17 04:13 BUG #19623: Postmaster livelocks respawning io workers when children die after crash restart; pg_ctl stop fails PG Bug reporting form <noreply@postgresql.org>
2026-08-20 01:28 ` Zexin Li <lizi.openmind@gmail.com>
2026-08-20 01:44 ` Michael Paquier <michael@paquier.xyz>
2026-08-20 02:21 ` Michael Paquier <michael@paquier.xyz>
2026-08-20 07:06 ` Michael Paquier <michael@paquier.xyz>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox