From: CyberDem0n (@CyberDem0n) <noreply+CyberDem0n@github.com>
To: patroni/patroni <noreply+patroni-patroni@github.com>
Subject: [patroni/patroni] PR #3805: Don't hang on SIGTERM: stop the HA loop instead of raising SystemExit
Date: Wed, 18 Nov 2026 11:55:46 +0000
Message-ID: <gh-patroni-patroni-3805@github.com> (raw)
## Problem
Behave tests sometimes fail because Patroni doesn't stop on SIGTERM. The `I shut down postgres-N` step takes about 300s until the test SIGKILLs Patroni, and Postgres is left running.
Example: [job 134738968444](https://github.com/patroni/patroni/actions/runs/44285915575/job/134738968444), `features/standby_cluster.feature:43`.
- `When I shut down postgres-1` took 301.317s, and the Patroni log ends right before SIGTERM was sent.
- The orphaned primary later diverged in timeline, and `pg_rewind` wiped `postgres-1_cb.log` in `PGDATA`.
- So the `on_role_change standby_leader batman1` assertion failed, and the next scenarios failed as a cascade.
The last 100 failed `tests.yaml` runs contain 4 more behave jobs with a ~301s `I shut down` step: kubernetes/ubuntu/3.8, etcd/macos/3.9 (×2) and raft/macos/3.8. All five hung jobs ran on Python 3.8/3.9, although about half of the behave jobs run on 3.10+.
## Root cause
`sigterm_handler()` raises `SystemExit`. Python runs signal handlers in the main thread between bytecode instructions. On Python 3.9 and older that also happens right after `POP_BLOCK`, i.e. after a `with` block was left but before `__exit__()` was called. An exception raised there skips `__exit__()`, and the lock stays locked forever, even a C `threading.Lock`. Python 3.10 removed this window ([bpo-29988](https://bugs.python.org/issue29988)), but Python 3.14 has a new one: `__enter__()` is now called by an ordinary `CALL` instruction, which runs pending signal handlers after `__enter__()` returned and before the `with` block's exception handler is set up. An `__enter__()` implemented in Python, e.g. `threading.Condition`, has a similar window on every version.
What happened in the failed job (Python 3.8.18):
1. SIGTERM arrived ~85ms into an HA cycle of the leader `postgres-1`. The previous cycle, which also didn't reload slots, logged `no action` at +86ms, so SIGTERM hit the tail of the cycle, where `Ha.touch_member()` runs. This cycle's `no action` message was never logged.
2. `touch_member()` reads `Postgresql.state`, i.e. `with self._state_lock: return self._state`. `SystemExit` was raised in the window above and leaked `_state_lock`.
3. `abstract_main()` called `shutdown()`. `api.shutdown()` stopped serving the REST API. Then `Ha.shutdown()` → `Postgresql.stop()` → `_do_stop()` → `set_state(STOPPING)` deadlocked on `_state_lock`, before `postmaster.signal_stop()`.
This matches every symptom:
- Patroni logged nothing after SIGTERM.
- Postgres never got a shutdown request; its heartbeat connection stayed idle until SIGKILL.
- At 10:51:29 `postgres-0` got `The handshake operation timed out` from `GET https://127.0.0.1:8009/patroni`. The kernel still accepted connections, but nobody served them anymore. It was the only request to that REST API after SIGTERM, so the worker pool wasn't exhausted, and connections time out since #3706. `serve_forever()` had stopped, i.e. `api.shutdown()` had already run.
The handler also called `api_sigterm()`, which takes the non-reentrant `_sigterm_lock`. It could deadlock if SIGTERM arrived while the main thread held that lock, either in `received_sigterm` (checked on every loop iteration) or in `shutdown()`.
## Fix
- **Startup:** until the daemon object is constructed, the handler still raises `SystemExit`.
- Startup can block, e.g. `ensure_dcs_access()` waiting for DCS.
- The logger thread isn't running yet, and `shutdown()` isn't called at this stage.
- **Main loop:** `abstract_main()` calls the new `enable_graceful_sigterm()` right before `run()`. From then on the handler doesn't raise and doesn't take any locks.
- It sets `_received_sigterm` and calls `_wakeup()`. `Patroni` implements `_wakeup()` as `Ha.wakeup()`, so the HA loop doesn't sleep until the next cycle.
- `_run_cycle()` skips `schedule_next_run()` if SIGTERM arrived during the cycle, and `run()` doesn't start the next cycle if SIGTERM arrived during a SIGHUP reload.
- `api_sigterm()`, i.e. `POST /sigterm` (Windows behave), calls `_wakeup()` too.
- **`dcs.event`** is now a `SignalSafeEvent` instead of `threading.Event`. Setting a `threading.Event` from the handler could deadlock. `Event.set()` takes the non-reentrant lock of a `Condition`, and the main thread holds that lock in `Event.clear()` and in `Event.wait()` before it blocks.
- `SignalSafeEvent` is set while its `threading.Lock` is unlocked. `set()` is just `Lock.release()`, which never blocks.
- `wait(timeout)` blocks in `Lock.acquire(timeout)`. On Linux a signal interrupts it: CPython runs the handler and retries, and if the handler released the lock, the acquire succeeds immediately.
- A release before the acquire isn't lost either, so the wakeup can't be missed.
- As before, only the main thread waits for and clears the event, in `watch()`. Other threads only set it: REST API request handlers via `Ha.wakeup()`, and the etcd3, Kubernetes, ZooKeeper and Raft watchers.
- **Raft:** `RaftController` cycles are 50ms, so it doesn't need `_wakeup()`.
- **Fallback:** a `sigterm-guard` thread terminates Patroni if the main loop is still running `loop_wait` seconds after SIGTERM. Nothing is raised in the main thread, so no lock can leak.
- `run()` registers the write end of a `socketpair()` with `signal.set_wakeup_fd()` and starts the thread, which reads the other end. CPython's C-level signal handler writes the signal number there immediately, even if the main thread is blocked in C code, e.g. in libpq, and can't run `sigterm_handler()`.
- When the deadline passes, `_emergency_exit()` logs a critical message and calls `_emergency_shutdown()`. Then it gives the logger thread up to 1s to write the queued records (the new `PatroniLogger.flush()`) and calls `os._exit(1)`.
- `Patroni._emergency_shutdown()` asks Postgres for a fast shutdown with `signal_stop()` and doesn't wait for it. If the request can't be sent, it kills Postgres with `signal_kill()`, because a primary left running would keep accepting writes after the leader key expires. If the cluster is paused, Postgres keeps running, as with the regular shutdown. It takes none of Patroni's locks, e.g. `_state_lock`, which the hung HA cycle could hold.
- `_emergency_exit()` arms `signal.alarm(5)` first. If these steps block too, e.g. reading `postmaster.pid` on a stalled filesystem, or in `signal_kill()`, which waits for the killed processes without a timeout, the kernel terminates the process 5s later. For the same reason the stack of the main thread isn't logged: formatting it reads source files.
- That requires the default action of SIGALRM, but an ignored SIGALRM and the signal mask are inherited from the parent process, even across `exec()`. So `run()` sets the default action while the guard runs and restores the previous handler afterwards, and the guard thread unblocks SIGALRM before it arms the alarm.
- `os._exit(1)` is called in `finally`, so it runs even if a step fails, and a failure to log the critical message doesn't skip the emergency shutdown.
- `run()` clears `_main_loop_running` and closes the socket in `finally`. The thread exits then, so it doesn't fire during `shutdown()`, unless the loop exits just as the deadline passes.
- The main thread never takes `_sigterm_lock`: `received_sigterm` and `shutdown()` access the flag without it, and only REST API threads take it in `api_sigterm()`. Otherwise `KeyboardInterrupt` raised by SIGINT while the lock is held would leak it and deadlock `shutdown()`.
- The guard isn't started on Windows, which has no `signal.alarm()`.
### Behavior changes
- An HA cycle that is running when SIGTERM arrives now finishes before shutdown. Previously it was interrupted at an arbitrary point.
- **A cycle that doesn't finish within `loop_wait` after SIGTERM** makes the guard terminate Patroni: Postgres is asked for a fast shutdown, and Patroni exits with status 1 right away. That includes legitimately slow cycles, e.g. DCS retries (up to `retry_timeout`) or a synchronous graceful demote. On master `SystemExit` interrupts such a cycle right away, and the regular shutdown runs, unless it deadlocks.
- Unlike the regular shutdown, the guard doesn't run callbacks, doesn't wait for synchronous standbys and doesn't remove the leader key, which expires after `ttl` seconds. An armed watchdog isn't disarmed, so it can reset the node, as after SIGKILL.
- The guard also works if the main thread is blocked in C code that retries on `EINTR`, e.g. a libpq query. On master the supervisor has to SIGKILL Patroni then, and Postgres keeps running.
- The guard only helps if the supervisor's stop timeout exceeds `loop_wait` plus about 1s. The sample systemd unit has `TimeoutSec=30`; Docker's default is 10s, the same as the default `loop_wait`.
- If Patroni runs as PID 1, e.g. in a container, the guard ends the container right away: `main()` returns when the daemon's child process exits, and the kernel then kills the remaining processes in the container, so Postgres may be killed before its fast shutdown finishes. On master, SIGTERM interrupts such a cycle, and the regular shutdown stops Postgres, unless it deadlocks or the cycle is blocked in C code.
- **etcd (v2) and Consul replicas** long-poll the leader key over HTTP. `Ha.wakeup()` interrupts the long-poll (#3808), so they wake up immediately too, see `test_sigterm_interrupts_long_poll`. DNS resolution, connection setup and TLS negotiation can't be interrupted.
- **Unchanged:**
- Where signals don't interrupt `Lock.acquire()`, e.g. on macOS (`sys.thread_info.lock == 'mutex+cond'`), the handler runs only when the wait times out, up to `loop_wait` seconds later. The wait ends at most `loop_wait` seconds after the previous HA cycle started, while the guard's deadline is `loop_wait` seconds after SIGTERM, which arrived after that cycle ended. So the main loop has at least the duration of that cycle to exit before the guard fires.
- SIGINT (`KeyboardInterrupt`) has the same hazard but is used only interactively.
## Alternatives considered
- **Raising the exception from another thread** doesn't help. `sys.exit()` ends only the thread that calls it. `_thread.interrupt_main()` and `PyThreadState_SetAsyncExc()` make the main thread raise at arbitrary points too, and leak locks the same way (see Testing).
- **`os._exit()`** can't hang, but it skips the graceful shutdown, like SIGKILL: Postgres keeps running and the leader key isn't released. This PR uses it only in the guard, after asking Postgres to stop, similar to the thread watchdog in #3786.
- **A SIGALRM handler that raises `SystemExit`**, the fallback before 7fc7ce11, has master's problem, only `loop_wait` seconds later: the exception can leak a lock, and `shutdown()` deadlocks on it. A probe on Python 3.14.10 leaked `Postgresql._state_lock` in 61 of 100 runs. It also can't interrupt C code that retries on `EINTR`, e.g. libpq.
- **Shutting down from another thread** would race with the HA cycle, which keeps managing Postgres and the leader key. The HA loop has to stop at a safe point first. The guard only asks Postgres to stop, flushes the log and exits.
- **Setting a `threading.Event` from another thread** works if the handler wakes that thread up without taking locks, e.g. by releasing a `Lock` it waits on, but that takes a dedicated thread. An existing thread could poll the flag instead, e.g. in `RestApiServer.service_actions()`, which `serve_forever()` calls every 0.5s, but then shutdown starts up to 0.5s later and depends on the REST API. `SignalSafeEvent` needs no thread. #3806 (closed) implemented the polling variant.
## Testing
- **Minimal models**, with a real SIGTERM sent at a random time:
- Lock leak: a loop reads a property that does `with lock: return value`, and `finally:` calls a `stop()` that takes the same lock. The SIGTERM handler calls `sys.exit()`, or sets a flag that another thread polls every 1ms.
- Wakeup: the loop runs `try: event.wait(nap) finally: event.clear()` until the handler sets a flag, and the handler sets the event.
| Hung runs | 3.8.20 | 3.12.3 | 3.14.10 |
|---|---|---|---|
| The handler calls `sys.exit()` (master) | 1/300 | 0/300 | 117/300 |
| Same, with one busy background thread | 22/300 | 0/300 | 78/300 |
| Another thread calls `_thread.interrupt_main()` | 31/300 | 0/300 | 64/300 |
| Another thread calls `PyThreadState_SetAsyncExc(main thread, SystemExit)` | 0/300 | 0/300 | 45/300 |
| The handler calls `threading.Event.set()`, `nap=0` | 60/100 | 57/100 | 48/100 |
| Same, `nap=0.001` | 0/300 | 0/300 | 0/300 |
| The handler calls `SignalSafeEvent.set()` (this PR), `nap=0` | 0/100 | 0/100 | 0/100 |
Every hung run is blocked in `stop()`, or in `Event.set()` called by the handler. If another thread calls `sys.exit()`, only that thread exits, and the loop keeps running (10 of 10 runs on each version).
- **Fault injection** (Python 3.8.20, etcd3, PostgreSQL 16, leader with `synchronous_mode` and permanent slots)
- A tracer calls the real SIGTERM handler at every point of one leader HA cycle where CPython 3.8 checks for pending signals and an exception could leak a lock. These are right after `POP_BLOCK` of `with`/`try`-`finally`, or right after a lock was acquired in `__enter__()`/`acquire()`. On master that gives 75 points, including stdlib, urllib3 and psycopg code.
- **master:** 19 of 75 points hang, and Postgres is never asked to stop.
- Leaked locks: `Postgresql._state_lock` and `_role_lock`, `NamedConnection._lock`, the psycopg connection lock, `AbstractDCS._cluster_thread_lock`, etcd3 `KVCache._object_cache_lock`, urllib3 pool queues, the `Event` in `dcs.watch()`, and `_sigterm_lock`.
- Only `Postgresql.state` in `Ha.touch_member()` reproduces all symptoms of the failed job:
```
MainThread
set_state (patroni/postgresql/__init__.py:733)
_do_stop (patroni/postgresql/__init__.py:908)
stop (patroni/postgresql/__init__.py:881)
<lambda> (patroni/ha.py:2593)
while_not_sync_standby (patroni/ha.py:1153)
shutdown (patroni/ha.py:2593)
_shutdown (patroni/__main__.py:276)
shutdown (patroni/daemon.py:200)
abstract_main (patroni/daemon.py:246)
```
- Leaked logging locks don't hang before Postgres is signaled.
- **This PR:** 66 of 66 points shut down gracefully. There are fewer points because `SignalSafeEvent` has no `with` blocks. The handler never raises, and Postgres is asked to stop within 0.51s (median 0.38s).
- **Missed wakeups:** the tracer also called the handler before every instruction on the path to the HA loop sleep. That covers `_run_cycle()`, `schedule_next_run()`, `Ha.watch()`, `Etcd3.watch()`, `AbstractDCS.watch()` and `SignalSafeEvent` (`loop_wait: 2`).
- All 213 points shut down within 0.50s (median 0.41s).
- Negative control: with `_wakeup()` disabled, points before `Lock.acquire()` take 1.91s, i.e. the rest of `loop_wait`.
- **Real SIGTERM** sent at random times to the etcd3 leader, mostly while it sleeps in `Lock.acquire()`.
- 50 of 50 shut down, median 0.28s, maximum 0.50s.
- With `_wakeup()` disabled: median 1.36s, maximum 1.91s.
- **Hung HA cycle:** Python 3.12.3, etcd3, PostgreSQL 16, a single node (leader), `loop_wait: 10`. A hang was injected into `Ha.run_cycle()`, then SIGTERM, then SIGKILL after 30s like systemd.
| Hang | master | fa1f6f1b (no fallback) | 7def6cac (SIGALRM fallback) | this PR (guard) |
|---|---|---|---|---|
| Python-level wait (`threading.Event().wait()`) | exited after 1.6s | SIGKILL, Postgres orphaned | Postgres asked to stop after 10.0s, Patroni exited after 11.2s with status 0 (2 of 2 runs) | Postgres asked to stop after 10.0s, Patroni exited after 10.0s with status 1 (2 of 2 runs) |
| libpq (`pg_sleep()`) | SIGKILL, Postgres orphaned | same | same | Postgres asked to stop after 10.0s, Patroni exited after 10.0s with status 1 (2 of 2 runs) |
With the guard, Postgres shut down cleanly 10.0s after SIGTERM, and the fast shutdown also terminated the backend running `pg_sleep()`.
- **Real SIGTERM without a hang:** same setup, SIGTERM at a random phase of the HA loop. 10 of 10 runs exited with status 0, and the guard didn't fire. Postgres was asked to stop after median 0.17s (max 0.57s), and Patroni exited after median 1.4s (max 1.72s).
- **Guard in C code:** `AbstractPatroniDaemon` with `loop_wait: 1`, the main thread blocked in C code (`pthread_mutex_lock()`), where `sigterm_handler()` can't run, then SIGTERM. On Python 3.8.20 and 3.14.10 the process exited with status 1 after 1.02s. With `_emergency_shutdown()` blocked too, SIGALRM killed it after 6.03s, also if the parent process had ignored or blocked SIGALRM. With 7fc7ce11 these two cases hung until the harness killed them after 15s.
- **Behave:** `standby_cluster.feature` and `basic_replication.feature` pass with etcd3, etcd (v2) and Raft, PostgreSQL 16 and Python 3.8.20. `When I shut down postgres-1` in `standby_cluster.feature` took 2.0s with each DCS (301s in the failed job).
- With timestamps written by instrumented handlers, 33 SIGTERMs arrived while the main loop was running, and in 32 cases the loop exited within 0.09s.
- The guard fired once, on Raft, during the teardown of `basic_replication.feature`. The leader `postgres-1` had lost the Raft quorum because two other members were already stopped, so its HA cycle didn't finish. The guard fired 1.0s after SIGTERM (`loop_wait` was 1, because the feature sets `ttl: 20`), and Postgres shut down cleanly 1.15s after SIGTERM, versus 3.4s with the SIGALRM fallback.
- **Unit tests and linters:**
- New unit tests, including ones where a real signal wakes up `dcs.watch()` and interrupts an etcd long-poll, and where a real SIGTERM makes the guard terminate a main loop that never finishes, with SIGALRM ignored by the parent. Others cover the guard's socket handling, `_emergency_exit()` and `Patroni._emergency_shutdown()` (fast shutdown, kill if that fails, paused cluster, exceptions, logging failures), `PatroniLogger.flush()`, and that the main thread never takes `_sigterm_lock`.
- `test_patroni.py`, `test_log.py`, `test_interrupt.py`, `test_api.py`, `test_ha.py`, `test_raft_controller.py` and the DCS tests (`test_etcd.py`, `test_etcd3.py`, `test_consul.py`, `test_kubernetes.py`, `test_zookeeper.py`, `test_raft.py`) pass on Python 3.8.20 and 3.14.10 (438 tests), with 100% coverage of `daemon.py`, `interrupt.py` and `log.py`.
- flake8, isort and pyright 1.1.411 report no errors.
## Earlier revisions
- The first revision blamed the mutex of the logging queue. That was wrong: there is no log call between SIGTERM and `signal_stop()`, so a leaked logging mutex can't keep Postgres running. The fix was the same; the docstring was corrected.
- Before fa1f6f1b, the handler released a `Lock` that a dedicated thread waited on, and that thread called `Ha.wakeup()`, because `threading.Event.set()` isn't safe in the handler. `SignalSafeEvent` made the thread unnecessary.
- In 7fc7ce11, `alarm(5)` had no effect if the parent process had ignored or blocked SIGALRM, and an exception while logging skipped `os._exit()`.
- Before 7fc7ce11 the fallback was a SIGALRM handler that raised `SystemExit` in the main thread, see Alternatives considered.
- Before c8c78a37 there was no fallback. An HA cycle that never finished kept Patroni running until the supervisor's SIGKILL, and Postgres was orphaned.