pg.ddx.io  patroni/patroni GitHub issues and pull requests  
help / color / mirror / Atom feed
[patroni/patroni] PR #3806: Alternative to #3805: wake up the HA loop on SIGTERM from the REST API thread
3+ messages / 1 participants
[nested] [flat]

* [patroni/patroni] PR #3806: Alternative to #3805: wake up the HA loop on SIGTERM from the REST API thread
@ 2026-11-18 20:07 "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
  0 siblings, 0 replies; 3+ messages in thread

From: CyberDem0n (@CyberDem0n) @ 2026-11-18 20:07 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

Alternative to #3805 for the same problem: on SIGTERM, Patroni sometimes hangs in `shutdown()` and leaves Postgres running. #3805 has the analysis of the failed behave job. In short, `sigterm_handler()` raises `SystemExit` in the main thread between arbitrary bytecode instructions. On Python 3.9 and older it can land right after `POP_BLOCK`, skip `__exit__()` and leak a lock, e.g. `Postgresql._state_lock`, on which `shutdown()` then deadlocks.

Compared to #3805, this PR needs no `SignalSafeEvent`: the REST API thread wakes up the HA loop. Both PRs have the same SIGALRM fallback for an HA loop that doesn't exit by itself.

## Fix

- **Startup:** unchanged. Until `abstract_main()` calls the new `enable_graceful_sigterm()` right before `run()`, the handler raises `SystemExit`.
- **Main loop:** the handler only sets `_received_sigterm`. It raises nothing and takes no locks; it no longer calls `api_sigterm()`, which takes the non-reentrant `_sigterm_lock`.
  - `RestApiServer.service_actions()`, which `serve_forever()` calls every 0.5s, calls `Ha.wakeup()` once the flag is set. The handler can't do that itself, because `threading.Event.set()` takes a lock.
  - `_run_cycle()` skips `schedule_next_run()` if SIGTERM arrived during the cycle, so the loop exits after the current cycle.
- **Fallback:** the handler also arms `signal.alarm(loop_wait)`. If the main loop is still running when SIGALRM arrives, `sigalrm_handler()` calls `sys.exit()`. That is what master does on SIGTERM, only `loop_wait` seconds later.
  - `run()` sets `_main_loop_running` and cancels the alarm with `alarm(0)` in `finally`. So the alarm is armed only while the loop runs, and SIGALRM doesn't interrupt `shutdown()`.
  - Cancelling is required: Python restores the default SIGALRM action during finalization, so an alarm still pending at exit would kill the process ("Alarm clock", exit status 142, reproduced on Python 3.8, 3.12 and 3.14).
  - Windows has no SIGALRM, so there is no fallback there.

## Trade-offs

| | master | #3805 | this PR |
|---|---|---|---|
| Exception raised inside the HA loop on SIGTERM | yes, at an arbitrary point | no, unless the fallback fires | no, unless the fallback fires |
| HA loop woken up by | the exception | the handler (`SignalSafeEvent`) | the REST API thread, within 0.5s |
| HA loop hung in a Python-level wait | exits | exits after `loop_wait` | exits after `loop_wait` |
| HA loop hung in libpq (`pg_sleep()`) | SIGKILL, Postgres orphaned | same | same |

- **The fallback is master's behaviour** (the same in both PRs), with master's risk of leaking a lock, but only when the HA cycle doesn't finish within `loop_wait` after SIGTERM. That includes legitimately slow cycles, e.g. DCS retries (up to `retry_timeout`) or a synchronous graceful demote, which are then interrupted instead of finishing.
- The fallback can't interrupt C code that retries on `EINTR`, e.g. a libpq query, or a deadlock that `shutdown()` would hit too.
- The fallback only matters if the supervisor's stop timeout exceeds `loop_wait` plus the time to stop Postgres. The sample systemd unit has `TimeoutSec=30`; Docker's default is 10s.
- **The wake-up depends on the REST API thread.** Without it, the loop exits when its sleep ends, up to `loop_wait` later.
- **etcd (v2) and Consul replicas**, as in #3805: the leader watch is an HTTP long-poll that `Ha.wakeup()` can't interrupt, so shutdown may start up to `loop_wait` later.
  - The long-poll can also end up to 0.5s (etcd) or `wait/16` (Consul) after the next cycle was due. So the fallback may occasionally fire in a healthy replica shutdown. This is from the code; behave doesn't exercise it.
  - It then interrupts a socket read, as master does on every SIGTERM. A margin, e.g. `alarm(loop_wait + 2)`, would avoid that.
- An escalated exit isn't logged and exits with status 0. The handler can't log, because logging takes locks.

## Testing

- **Latency model:** a real `socketserver` with `serve_forever(0.5)`, a real SIGTERM and a 2s nap; 40 trials each on Python 3.8.20 and 3.14.10.
  - This PR: the main loop exits after median 0.22s / 0.36s (max 0.50s), and the REST API stops after median 0.72s / 0.86s (max 1.00s).
  - #3805: the main loop exits within 0.02s, and the REST API stops within 0.49s.
- **End-to-end:** Python 3.12, etcd3, PostgreSQL 16, a single node (leader), a real SIGTERM at a random phase of the HA loop, 8 runs each, before either PR had the fallback:

  | | Postgres asked to stop (median / max) | Patroni exited (median / max) |
  |---|---|---|
  | master | 0.19s / 0.49s | 1.37s / 1.67s |
  | #3805 (fa1f6f1b) | 0.18s / 0.37s | 1.39s / 1.57s |
  | this PR | 0.72s / 0.94s | 1.89s / 2.12s |
  | no wake-up at all | 5.97s / 9.63s | 7.18s / 10.89s |

  With the fallback, Patroni exited after 1.7–2.2s (#3805: median 1.49s, max 1.72s in 10 runs).
- **Hung HA loop:** same setup, with a hang injected into `Ha.run_cycle()`, then SIGTERM, then SIGKILL after 30s like systemd. The results are in the table above. With this PR, Postgres was asked to stop after 10.0–10.5s, and Patroni exited after 11.2–11.7s with exit status 0. #3805 with the same fallback: 10.0s and 11.2s (2 of 2 runs).
- **Unit tests:** SIGTERM at startup and in the main loop, `api_sigterm()`, `service_actions()`, the alarm being armed and cancelled, and `sigalrm_handler()` before, during and after the loop.
  - `test_patroni.py`, `test_api.py` and `test_ha.py` pass on Python 3.8 and 3.14, with 100% coverage of `daemon.py`.
  - flake8 and isort report no errors.
- Not repeated from #3805: the bytecode-level fault injection and the behave runs. The behave runs on #3805 cover the shared fallback too: it fired once, on a Raft leader that had lost quorum during teardown, and Postgres shut down cleanly.


inline patch 1: patroni/__main__.py, patroni/api.py, patroni/daemon.py and 2 more (+105/-12 lines) download inline-1.patchdiff --git a/patroni/__main__.py b/patroni/__main__.py index 1ceaa2913..cc8210794 100644 --- a/patroni/__main__.py +++ b/patroni/__main__.py @@ -259,7 +259,9 @@ def _run_cycle(self) -> None: if self.postgresql.role != PostgresqlRole.UNINITIALIZED: self.config.save_cache() - self.schedule_next_run() + # Don't wait for the next cycle if SIGTERM was received while running the current one. + if not self.received_sigterm: + self.schedule_next_run() def _shutdown(self) -> None: """Perform shutdown of ``patroni`` daemon process. diff --git a/patroni/api.py b/patroni/api.py index ccc85e8f2..07271e7ec 100644 --- a/patroni/api.py +++ b/patroni/api.py @@ -1939,6 +1939,16 @@ def finish_request(self, request: Union[socket.socket, Tuple[bytes, socket.socke except OSError as e: logger.debug('Connection from %s:%s was reset: %r', client_address[0], client_address[1], e) + def service_actions(self) -> None: + """Wake up the HA loop if SIGTERM was received. + + Called by :func:`~socketserver.BaseServer.serve_forever` every 0.5 seconds. The SIGTERM handler can't do it + itself, because it must not acquire locks. + """ + super(RestApiServer, self).service_actions() + if self.patroni.received_sigterm: + self.patroni.ha.wakeup() + def shutdown(self) -> None: super(RestApiServer, self).shutdown() self._executor.shutdown(wait=True) diff --git a/patroni/daemon.py b/patroni/daemon.py index ecc20d152..ba212bf10 100644 --- a/patroni/daemon.py +++ b/patroni/daemon.py @@ -125,25 +125,51 @@ def api_sigterm(self) -> bool: def sigterm_handler(self, *_: Any) -> None: """Handle SIGTERM signals. - Terminate the daemon process through :func:`api_sigterm`. + Terminate the daemon process until :func:`enable_graceful_sigterm` is called. Afterwards only flag the daemon + as "SIGTERM received", the main loop exits and shuts down the daemon. If the main loop doesn't exit within + ``loop_wait`` seconds, for example because it hangs, :func:`sigalrm_handler` terminates the daemon process. + + .. note:: + Python runs signal handlers in the main thread between arbitrary bytecode instructions. An exception + raised here could leave a lock held by the interrupted code locked forever, and the shutdown would + deadlock on it. The handler must not acquire locks either, because the interrupted code could hold them. + """ + if not self._received_sigterm: + self._received_sigterm = True + if not self._graceful_sigterm: + sys.exit() + if self._main_loop_running and os.name != 'nt': + signal.alarm(self.config['loop_wait']) + + def sigalrm_handler(self, *_: Any) -> None: + """Handle SIGALRM signals. + + Terminate the daemon process if the main loop is still running, i.e. it didn't exit in time after SIGTERM. """ - if self.api_sigterm(): + if self._main_loop_running: sys.exit() + def enable_graceful_sigterm(self) -> None: + """Make :func:`sigterm_handler` only flag the daemon as "SIGTERM received" instead of terminating it.""" + self._graceful_sigterm = True + def setup_signal_handlers(self) -> None: """Set up daemon signal handlers. - Set up SIGHUP and SIGTERM signal handlers. + Set up SIGHUP, SIGALRM and SIGTERM signal handlers. .. note:: - SIGHUP is only handled in non-Windows environments. + SIGHUP and SIGALRM are only handled in non-Windows environments. """ self._received_sighup = False self._sigterm_lock = Lock() self._received_sigterm = False + self._graceful_sigterm = False + self._main_loop_running = False if os.name != 'nt': signal.signal(signal.SIGHUP, self.sighup_handler) + signal.signal(signal.SIGALRM, self.sigalrm_handler) signal.signal(signal.SIGTERM, self.sigterm_handler) @property @@ -177,13 +203,20 @@ def run(self) -> None: """ notify_systemd("READY=1") self.logger.start() - while not self.received_sigterm: - if self._received_sighup: - self._received_sighup = False - self.reload_config(True, self.config.reload_local_configuration()) - notify_systemd("READY=1") - - self._run_cycle() + self._main_loop_running = True + try: + while not self.received_sigterm: + if self._received_sighup: + self._received_sighup = False + self.reload_config(True, self.config.reload_local_configuration()) + notify_systemd("READY=1") + + self._run_cycle() + finally: + # SIGALRM must neither interrupt the shutdown nor kill the process after Python restores default handlers + self._main_loop_running = False + if os.name != 'nt': + signal.alarm(0) @abc.abstractmethod def _shutdown(self) -> None: @@ -238,6 +271,7 @@ def abstract_main(cls: Type[AbstractPatroniDaemon], configfile: str) -> None: logger.warning('Failed to set threading.stack_size(%s): %r', thread_stack_size, e) controller = cls(config, patroni_logger) + controller.enable_graceful_sigterm() try: controller.run() except KeyboardInterrupt: diff --git a/tests/test_api.py b/tests/test_api.py index 40bc1c44a..5b888ac5c 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -175,6 +175,8 @@ def sighup_handler(): def api_sigterm(): pass + received_sigterm = False + class MockRequest(object): @@ -848,6 +850,14 @@ def test_server_bind(self, mock_getfqdn): self.srv.server_bind() mock_getfqdn.assert_not_called() + @patch.object(MockHa, 'wakeup') + def test_service_actions(self, mock_wakeup): + self.srv.service_actions() + mock_wakeup.assert_not_called() + with patch.object(MockPatroni, 'received_sigterm', True): + self.srv.service_actions() + mock_wakeup.assert_called_once() + @patch.object(HTTPServer, '__init__', Mock()) def test_reload_config(self): bad_config = {'listen': 'foo'} diff --git a/tests/test_patroni.py b/tests/test_patroni.py index 00f16b2cd..d7ba3480a 100644 --- a/tests/test_patroni.py +++ b/tests/test_patroni.py @@ -203,6 +203,43 @@ def test_run(self): def test_sigterm_handler(self): self.assertRaises(SystemExit, self.p.sigterm_handler) + def test_api_sigterm(self): + self.assertTrue(self.p.api_sigterm()) + self.assertFalse(self.p.api_sigterm()) + self.p.sigterm_handler() # doesn't raise SystemExit, because SIGTERM was already received + + @patch('signal.alarm', create=True) + @patch('patroni.config.Config.save_cache', Mock()) + @patch('patroni.config.Config.set_dynamic_configuration', Mock(return_value=False)) + @patch.object(Postgresql, 'reload_config', Mock()) + def test_graceful_sigterm(self, mock_alarm): + self.p.enable_graceful_sigterm() + self.p.api.start = Mock() + self.p.logger.start = Mock() + self.p.ha.dcs.watch = Mock(side_effect=SleepException) + + def run_cycle(): + self.p.sigterm_handler() # doesn't raise SystemExit anymore + return 'no action' + + # the main loop exits after the HA cycle, during which SIGTERM was received, without waiting for the next one + self.p.ha.run_cycle = Mock(side_effect=run_cycle) + self.p.run() + self.assertTrue(self.p.received_sigterm) + self.p.ha.dcs.watch.assert_not_called() + # the alarm is armed on SIGTERM and canceled when the main loop exits + loop_wait = self.p.config['loop_wait'] + self.assertEqual(mock_alarm.call_args_list, [] if os.name == 'nt' else [((loop_wait,),), ((0,),)]) + + def test_sigalrm_handler(self): + self.p.sigalrm_handler() # doesn't raise SystemExit before the main loop has started + self.p.api.start = Mock() + self.p.logger.start = Mock() + # the main loop is stuck in the HA cycle when SIGALRM arrives + self.p.ha.run_cycle = Mock(side_effect=self.p.sigalrm_handler) + self.assertRaises(SystemExit, self.p.run) + self.p.sigalrm_handler() # doesn't raise SystemExit after the main loop has exited + def test_schedule_next_run(self): self.p.ha.cluster = Mock() self.p.ha.dcs.watch = Mock(return_value=True)
^ permalink raw [nested|flat] 3+ messages in thread

* Re: [patroni/patroni] PR #3806: Alternative to #3805: wake up the HA loop on SIGTERM from the REST API thread
@ 2026-11-18 21:05 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
  1 sibling, 0 replies; 3+ messages in thread

From: CyberDem0n (@CyberDem0n) @ 2026-11-18 21:05 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

@CyberDem0n renamed this.

^ permalink  raw  [nested|flat] 3+ messages in thread

* Re: [patroni/patroni] PR #3806: Alternative to #3805: wake up the HA loop on SIGTERM from the REST API thread
@ 2026-11-20 09:44 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
  1 sibling, 0 replies; 3+ messages in thread

From: CyberDem0n (@CyberDem0n) @ 2026-11-20 09:44 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

@CyberDem0n closed this.

^ permalink  raw  [nested|flat] 3+ messages in thread


end of thread, other threads:[~2026-11-20 09:44 UTC | newest]

Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-11-18 20:07 [patroni/patroni] PR #3806: Alternative to #3805: wake up the HA loop on SIGTERM from the REST API thread "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-18 21:05 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-20 09:44 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox