Received: from malur.postgresql.org ([217.196.149.56])
	by arkaria.postgresql.org with esmtps  (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
	(Exim 4.94.2)
	(envelope-from <pgsql-hackers-owner+archive@lists.postgresql.org>)
	id 1r8vqI-00CfqA-R2
	for pgsql-hackers@arkaria.postgresql.org; Fri, 01 Dec 2023 05:13:34 +0000
Received: from localhost ([127.0.0.1] helo=malur.postgresql.org)
	by malur.postgresql.org with esmtp (Exim 4.94.2)
	(envelope-from <pgsql-hackers-owner+archive@lists.postgresql.org>)
	id 1r8vqG-005PSa-Es
	for pgsql-hackers@arkaria.postgresql.org; Fri, 01 Dec 2023 05:13:32 +0000
Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29])
	by malur.postgresql.org with esmtps  (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
	(Exim 4.94.2)
	(envelope-from <pryzby@telsasoft.com>)
	id 1r8vqG-005PSC-4U
	for pgsql-hackers@lists.postgresql.org; Fri, 01 Dec 2023 05:13:32 +0000
Received: from charmander.telsasoft.com ([50.244.222.1] helo=pryzbyj2023.telsasoft)
	by magus.postgresql.org with esmtps  (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
	(Exim 4.94.2)
	(envelope-from <pryzby@telsasoft.com>)
	id 1r8vqC-009Ts9-Nj; Fri, 01 Dec 2023 05:13:31 +0000
Received: by pryzbyj2023.telsasoft (Postfix, from userid 1000)
	id ED5C6FC1D; Thu, 30 Nov 2023 23:13:25 -0600 (CST)
Date: Thu, 30 Nov 2023 23:13:25 -0600
From: Justin Pryzby <pryzby@telsasoft.com>
To: pgsql-hackers@postgresql.org
Cc: Thomas Munro <tmunro@postgresql.org>
Subject: processes stuck in shutdown following OOM/recovery
Message-ID: <ZWlrdQarrZvLsgIk@pryzbyj2023>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
List-Id: <pgsql-hackers.lists.postgresql.org>
List-Help: <https://lists.postgresql.org/manage/>
List-Subscribe: <https://lists.postgresql.org/manage/>
List-Post: <mailto:pgsql-hackers@lists.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@lists.postgresql.org>
List-Archive: <https://www.postgresql.org/list/pgsql-hackers>
Archived-At: <https://www.postgresql.org/message-id/ZWlrdQarrZvLsgIk%40pryzbyj2023>
Precedence: bulk

If postgres starts, and one of its children is immediately killed, and
the cluster is also told to stop, then, instead, the whole system gets
wedged.

$ initdb -D ./pgdev.dat1
$ pg_ctl -D ./pgdev.dat1 start -o '-c port=5678'
$ kill -9 2524495; sleep 0.05; pg_ctl -D ./pgdev.dat1 stop -m fast # 2524495 is a child's pid
.......................................................... failed
pg_ctl: server does not shut down

$ ps -wwwf --ppid 2524494
UID          PID    PPID  C STIME TTY          TIME CMD
pryzbyj  2524552 2524494  0 20:47 ?        00:00:00 postgres: checkpointer 

(gdb) bt
#0  0x00007f0ce2d08c03 in epoll_wait (epfd=10, events=0x55cb4cbaac28, maxevents=1, timeout=timeout@entry=156481) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1  0x000055cb4c219208 in WaitEventSetWaitBlock (set=set@entry=0x55cb4cbaabc0, cur_timeout=cur_timeout@entry=156481, occurred_events=occurred_events@entry=0x7ffd80130410, 
    nevents=nevents@entry=1) at ../src/backend/storage/ipc/latch.c:1583
#2  0x000055cb4c219e02 in WaitEventSetWait (set=0x55cb4cbaabc0, timeout=timeout@entry=300000, occurred_events=occurred_events@entry=0x7ffd80130410, nevents=nevents@entry=1, 
    wait_event_info=wait_event_info@entry=83886084) at ../src/backend/storage/ipc/latch.c:1529
#3  0x000055cb4c219f87 in WaitLatch (latch=<optimized out>, wakeEvents=wakeEvents@entry=41, timeout=timeout@entry=300000, wait_event_info=wait_event_info@entry=83886084)
    at ../src/backend/storage/ipc/latch.c:539
#4  0x000055cb4c1aabc2 in CheckpointerMain () at ../src/backend/postmaster/checkpointer.c:523
#5  0x000055cb4c1a8207 in AuxiliaryProcessMain (auxtype=auxtype@entry=CheckpointerProcess) at ../src/backend/postmaster/auxprocess.c:153
#6  0x000055cb4c1ae63d in StartChildProcess (type=type@entry=CheckpointerProcess) at ../src/backend/postmaster/postmaster.c:5331
#7  0x000055cb4c1b07f3 in ServerLoop () at ../src/backend/postmaster/postmaster.c:1792
#8  0x000055cb4c1b1c56 in PostmasterMain (argc=argc@entry=5, argv=argv@entry=0x55cb4cbaa380) at ../src/backend/postmaster/postmaster.c:1466
#9  0x000055cb4c0f4c1b in main (argc=5, argv=0x55cb4cbaa380) at ../src/backend/main/main.c:198

I noticed this because of the counter-effective behavior of systemd+PGDG
unit files to run "pg_ctl stop" whenever a backend is killed for OOM:
https://www.postgresql.org/message-id/ZVI112aVNCHOQgfF@pryzbyj2023

This affects v15, and fails at 7ff23c6d27 but not its parent.

commit 7ff23c6d277d1d90478a51f0dd81414d343f3850 (HEAD)
Author: Thomas Munro <tmunro@postgresql.org>
Date:   Mon Aug 2 17:32:20 2021 +1200

    Run checkpointer and bgwriter in crash recovery.

-- 
Justin