pg.ddx.io  patroni/patroni GitHub issues and pull requests  
help / color / mirror / Atom feed
[patroni/patroni] PR #3820: Don't wait for REST API clients when stopping Patroni
2+ messages / 2 participants
[nested] [flat]

* [patroni/patroni] PR #3820: Don't wait for REST API clients when stopping Patroni
@ 2026-11-20 07:35 "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
  0 siblings, 0 replies; 2+ messages in thread

From: CyberDem0n (@CyberDem0n) @ 2026-11-20 07:35 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

Fixes #3819

## Problem

`Patroni._shutdown()` stops the REST API before `Ha.shutdown()`, and the HA loop doesn't run in between. Since #3526, `RestApiServer.shutdown()` waits for the REST API thread pool. Any busy worker therefore delays the stop of PostgreSQL:

| REST API worker is... | `shutdown()` blocks on master for |
|---|---|
| waiting for the request of an idle HTTP client | `restapi.request_timeout` |
| the same, over TLS (`readline()`, then `unwrap()` waits for `close_notify`) | 2 × `request_timeout` |
| receiving a request from a slow client (one byte per `request_timeout`) | unbounded |
| running a long handler (synchronous `POST /restart`, `/switchover` polling, DCS retries) | the handler's duration |

Nothing updates the leader key while it waits. If the delay exceeds `ttl`, a replica is promoted while the old primary still accepts writes. Those writes are lost when the old primary is rewound.

## Reproduction

The reporter's 3-node docker setup was used, with `ttl: 30`, `loop_wait: 10` and `restapi.request_timeout: 60`. One idle TCP client was connected to the REST API of the node being stopped, and a writer ran against the leader.

| | master | this PR (HTTP) | this PR (HTTPS, the idle client sent one request first) |
|---|---|---|---|
| replica stop | 60.4 s | 1.3 s | 1.4 s |
| leader stop | 60.4 s | 1.5 s | 1.5 s |
| leader change while the old leader was still running | yes, 30 s after SIGTERM | no | no |
| acknowledged rows missing on the new leader | 455 of 951 | 0 of 61 | 0 of 63 |
| pg_rewind of the old leader | needed, the servers diverged | not needed | not needed |

## Changes

- `RestApiServer.shutdown()` doesn't wait for the thread pool (`wait=False`). Running handlers can still send their responses.
- Workers that wait for a client are woken up with the portable `Interrupter` from #3808. `RestApiServer` has one, and `shutdown()` interrupts it before it stops `serve_forever()`:
  - `RestApiHandler.setup()` replaces `rfile` with `interruptible_reader()`, so reading the request line, headers or body raises `RequestInterrupted`. It is a `BaseException`, so `handle_one_request()` doesn't answer `503` and `_read_json_content()` doesn't log an error. `finish_request()` logs it at DEBUG.
  - The TLS handshake in `process_request_thread()`, and `unwrap()` in `shutdown_request()`, which waits for the client's `close_notify`, run through the new `interrupt.interruptible_call()`. It retries a nonblocking socket operation until it succeeds, the socket's timeout expires, or the `Interrupter` is interrupted. The operation is always tried once, so the server's `close_notify` is still sent.
  - `_InterruptibleSocketIO.readinto()` now uses the same loop. `interruptible_reader()` and `interruptible_call()` are public, so `api.py` doesn't use private helpers. They raise the new `RequestInterrupted`. `WatchInterrupted` is now derived from it, and is still raised by `armed()` and by reads of HTTP responses received while an `Interrupter` was armed, which the DCS watches catch.
  - This works the same on all platforms, including Windows, and doesn't need connection tracking or cross-thread socket shutdown.
- A request that was already received but whose handler hasn't started when the shutdown begins is rejected with `503`, because Patroni no longer waits for handlers before it stops PostgreSQL.
- Handlers are no longer waited for before `Ha.shutdown()`, so a `POST /restart` or `/reinitialize` could otherwise schedule an action after it cancelled the async executor. That action could start PostgreSQL after it was stopped and the leader key was removed. `Ha.shutdown()` now calls the new `AsyncExecutor.shutdown()`, which:
  - refuses to schedule (`shutdown already in progress`, `503`) or start new tasks;
  - cancels the current task, as before;
  - waits for **all** running tasks.

  The executor now counts running tasks, because more than one can run at once. A forced `/reinitialize` can cancel a `/restart` that was scheduled but hadn't started yet, and that restart then still starts. This already happens on master. Before this change, the first task to finish released `cancel()`, and a task that started late could reset the cancellation. Now `cancel()` waits for every running task. The finished-event is set only when no task is running. The cancellation is reset under the executor lock, before the task starts.
- `Ha.shutdown()` takes the executor lock before it stops PostgreSQL, so a handler that holds the lock still delays the stop. Handlers no longer hold it while they wait for the DCS or PostgreSQL:
  - `/reinitialize` reads the DCS without the lock. It keeps the cluster in a local variable instead of `self.cluster`, which belongs to the HA loop. The HA loop takes the leader lock only while it holds the executor lock, so leadership is checked again under that lock: right before a forced reinitialize cancels the running task (the new `unless` argument of `AsyncExecutor.cancel()`), and right before the reinitialize is scheduled. A node that became the leader during the read isn't reinitialized, and its pending promotion isn't cancelled.
  - `AsyncExecutor.cancel()` still waits for the cancelled task without holding the lock, but it no longer unschedules a task that was scheduled meanwhile, for example a promotion by the HA loop.
  - `POST /restart` with a schedule queries PostgreSQL before it takes the lock. It and `DELETE /restart` change only the in-memory schedule under the lock, and publish the member data after releasing it. `touch_member()` reads the schedule once, because a request can replace it concurrently.
- After the executor is shut down, `POST /restart` with a schedule and `DELETE /restart` answer `503` (`shutdown already in progress`). The schedule exists only in memory, and no HA cycle would run to execute it.
- `Ha.shutdown()` remembers its own leader key before it stops PostgreSQL, and removes only that key. It used to read `self.cluster.leader` only when it removed the key, and Consul deletes the leader key by its version only: if the cluster had been refreshed meanwhile, the key of the next leader could be removed. A `/reinitialize` that ran during the shutdown could do that refresh. This is possible since #2810, and #3526 hid it, because it waits for the handlers before `Ha.shutdown()`.
- `Patroni._shutdown()` waits for the handlers that are still running after `Ha.shutdown()`, before it closes the global thread pool. A `/switchover` handler submits the checks of the candidates to that pool, and would otherwise fail with `503` (`RuntimeError: cannot schedule new futures after shutdown`) before it writes the failover key. The listener runs in the pool of the REST API as well, so the handlers aren't waited for if `RestApiServer.shutdown()` failed.

The first commit shut down the reading side of the connections (`SHUT_RD`) instead. That doesn't wake up the workers on Windows, so it was skipped there. The second commit replaces it with the `Interrupter`, and drops the `400` for request bodies shorter than `Content-Length` and the related `POST /reinitialize` change, because an interrupted read can no longer cut a request short. The third commit adds `RequestInterrupted`, so that `api.py` doesn't use `WatchInterrupted`. The fourth commit fixes problems found in review: handlers that held the executor lock during I/O, scheduled restarts during the shutdown, the removal of the leader key, and the global thread pool that was closed under a running `/switchover`.

## Testing

- New and updated tests:
  - `TestRestApiServer.test_shutdown`: real sockets with a busy, an idle and a truncated client. `shutdown()` returns quickly, and the idle and truncated clients are disconnected while the busy handler is still running. The busy client still gets `200`, and a request handled after the shutdown gets `503`.
  - `test_process_request_thread_ssl_handshake_timeout`: a real TLS connection without a ClientHello. The handshake times out, it is interrupted by the `Interrupter`, and the timeout is restored after a successful handshake.
  - `test_shutdown_request_interrupted`, and `TestInterruptibleCall` in `test_interrupt.py`
  - `TestHa.test_restart_after_shutdown`, which now covers scheduled restarts too, including a shutdown while PostgreSQL is queried. The `503` responses are tested in `test_do_POST_restart` and `test_do_DELETE_restart`.
  - `TestAsyncExecutor.test_shutdown` and `test_shutdown_waits_for_all_tasks`. The second uses threads: a cancelled restart starts next to a reinitialize, and `shutdown()` must wait for both.
  - `TestHa.test_reinitialize_new_leader`: the HA loop takes the leader lock while `/reinitialize` reads the DCS. Neither a normal nor a forced reinitialize is scheduled, a pending promotion isn't cancelled, and a promotion that is scheduled while `cancel()` waits stays scheduled.
  - `TestHa.test_scheduled_restart_does_not_delay_shutdown` uses threads: `Postgresql.stop()` must be called while a scheduled `POST /restart` or a `DELETE /restart` still publishes the member data.
  - `TestHa.test_shutdown`: `self.cluster` is replaced by the one of the next leader while PostgreSQL is stopped, and `delete_leader()` still gets our own leader key, both from the `on_shutdown` callback and after the stop.
  - `TestPatroni.test_shutdown`: the handlers are waited for only if `RestApiServer.shutdown()` succeeded.
- The unit suite, flake8, isort and pyright pass.
- A real REST API server with `request_timeout: 10`. The time is measured from the call of `shutdown()` until all workers exited, and the clients connected 0.5 s before it. This PR was run with the `poll()` code path and with the `select()` one that Windows uses. For the first commit, the Windows path was simulated by skipping `SHUT_RD`:

  | client | first commit, Linux | first commit, Windows | this PR, `poll()` and `select()` |
  |---|---|---|---|
  | idle | ≤ 0.5 s | 9.5 s (`request_timeout`) | ≤ 0.5 s |
  | partial `POST` body | ≤ 0.5 s | 9.5 s | ≤ 0.5 s |
  | slow client (one byte per second) | ≤ 0.5 s | unbounded (> 30 s) | ≤ 0.5 s |
  | TLS before the handshake | ≤ 0.5 s | 1.5 s (`handshake_timeout`) | ≤ 0.5 s |
  | idle TLS | ≤ 0.5 s | 19.5 s (2 × `request_timeout`) | ≤ 0.5 s |
  | TLS without `close_notify` | ≤ 0.5 s | 9.5 s | ≤ 0.5 s |

  The remaining ≤ 0.5 s is `serve_forever()`'s poll interval, i.e. the time `shutdown()` itself takes. With this PR, every client sees the connection closed.
- The e2e runs above, on the last commit.
- Timed reproductions of the review findings, with a mocked DCS and a real REST API server on the loopback interface:
  - While `/reinitialize`, a scheduled `POST /restart` or a `DELETE /restart` is blocked in the DCS, `Postgresql.stop()` is still called immediately.
  - `delete_leader()` gets our own leader key (version 41), even though the cached cluster already has the key of the next leader (version 42).
  - A scheduled restart during the shutdown gets `503`. A `/switchover` during the shutdown gets `200`, and the failover key is written.
- Live checks of the last commit with the same docker setup: a scheduled `POST /restart` got `202` and then `409`, and the schedule was published in the DCS. `DELETE /restart` got `200` and then `404`, and removed it from the DCS. `POST /reinitialize` got `503` (`I am the leader, can not reinitialize`) on the leader, and `200` on a replica with and without `force`, after which the replica was streaming again.
- Live checks of the third commit with the same docker setup, and DEBUG logging for `patroni.api` and `patroni.dcs.etcd3`:
  - A replica with a TLS REST API, `request_timeout` and `handshake_timeout` set to 60 s, and five idle clients: one without a ClientHello, one without a request, one with partial headers, a `PATCH /config` that sent 10 of its 51 body bytes, and one that got its response but never sent `close_notify`. `docker stop` took 1.7 s, and all five clients saw EOF within 40 ms of the SIGTERM. The partial `PATCH` wasn't applied. The interrupted connections were logged only at DEBUG, without tracebacks.
  - etcd v3: `docker pause` of the etcd node used by a replica and by the leader. On both nodes the read timed out, another etcd node was selected, and the cache watch was interrupted ("Watch request interrupted") and created again within 0.1 s. A `PATCH /config` sent afterwards was applied on all three nodes within 0.2 s.
  - etcd v2: a `PATCH /config` sent to a replica during its long-poll started the next HA cycle within 0.11 s, 4 times out of 4. Otherwise, the cycles were 10 s apart.
- The behave suite on the third commit, with etcd v2, etcd v3 and Consul (Python 3.13, PostgreSQL 17): all features pass, except `citus` (no extension in the image) and `bootstrap labels` (Kubernetes only), which were skipped.

## Notes

- After `Ha.shutdown()`, Patroni still waits for the handlers that are running, for example a `/switchover` that polls, before it exits. This no longer affects PostgreSQL or the leader key.
- A handler that publishes the member data can still delay the stop of a synchronous standby by one update of the member key, because the standby takes `_member_state_lock` in `while_not_sync_standby()` before it stops PostgreSQL. The primary doesn't.
- Outside of shutdown, a forced `/reinitialize` and a pending `/restart` can still run at the same time. This race already exists and isn't addressed here; this PR only makes sure that shutdown waits for both.
- `POST /reinitialize` with an invalid JSON body sends `400`, then still reinitializes with default options and writes a second response. This already happens on master and isn't fixed here anymore.
- The docs say that `restapi.request_timeout` bounds how long a connection holds a worker thread. It applies to each `recv()`, so it doesn't bound a slow client. This was already the case and isn't changed here.
- #3805 conflicts with this PR in the module docstring and the imports of `patroni/interrupt.py`, and in the imports of `tests/test_interrupt.py`. Whichever is merged second needs a trivial rebase.


^ permalink  raw  [nested|flat] 2+ messages in thread

* Re: [patroni/patroni] PR #3820: Don't wait for REST API clients when stopping Patroni
@ 2026-11-20 10:13 "codecov[bot] (bot) (@codecov[bot])" <noreply+codecov[bot]@github.com>
  0 siblings, 0 replies; 2+ messages in thread

From: codecov[bot] (bot) (@codecov[bot]) @ 2026-11-20 10:13 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

## [Codecov](https://app.codecov.io/gh/patroni/patroni/pull/3820?dropdown=coverage&src=pr&el=h1&utm_m...) Report
:white_check_mark: All modified and coverable lines are covered by tests.

:loudspeaker: Thoughts on this report? [Let us know!](https://github.com/codecov/feedback/issues/255)

^ permalink  raw  [nested|flat] 2+ messages in thread


end of thread, other threads:[~2026-11-20 10:13 UTC | newest]

Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-11-20 07:35 [patroni/patroni] PR #3820: Don't wait for REST API clients when stopping Patroni "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-20 10:13 Re: [patroni/patroni] PR #3820: Don't wait for REST API clients when stopping Patroni "codecov[bot] (bot) (@codecov[bot])" <noreply+codecov[bot]@github.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox