From: CyberDem0n (@CyberDem0n) <noreply+CyberDem0n@github.com>
To: patroni/patroni <noreply+patroni-patroni@github.com>
Subject: [patroni/patroni] PR #3808: Interrupt DCS watch requests with a portable Interrupter
Date: Thu, 19 Nov 2026 13:10:46 +0000
Message-ID: <gh-patroni-patroni-3808@github.com> (raw)
## Problem
etcd3 and Kubernetes keep the cluster state in a cache, which a background thread updates from a watch stream (`KVCache`, `ObjectCache`). When the cache could be stale, `kill_stream()` makes that thread start over. It calls `shutdown(SHUT_RDWR)` and `close()` on the socket of the watch request from another thread.
- etcd3 calls it when the etcd node is stale, when switching to another etcd node and when a transaction fails.
- Kubernetes calls it on `409 Conflict` in `patch_or_create()` and `attempt_to_acquire_leader()`.
This has several problems:
- **Windows:** `shutdown()` doesn't wake up the reader.
- etcd3 works around it with `read_timeout=ttl` for the watch request. So `kill_stream()` takes effect only up to `ttl` seconds later, and idle watches are re-established every `ttl` seconds.
- Kubernetes has no workaround.
- **Every `kill_stream()` logs an ERROR**, e.g. `watchprefix failed: ProtocolError('Response ended prematurely')` (etcd3) or `ObjectCache.run ProtocolError('Response ended prematurely')` (Kubernetes).
- **pyOpenSSL**, if it is injected into urllib3 (python-etcd does that on import if pyOpenSSL is installed): etcd3 already shuts down `WrappedSocket.socket` for that reason, but Kubernetes calls `WrappedSocket.shutdown()` and `close()`.
- urllib3 1.26: `WrappedSocket.shutdown()` takes no arguments. The `TypeError` is logged at DEBUG, and the stream isn't interrupted at all.
- urllib3 2.x: the reader fails with `ValueError('file descriptor cannot be a negative integer (-1)')`, which is logged as ERROR, and the watch is re-established only up to 1s later.
etcd (v2) and Consul have a related problem:
- Replicas wait for changes of the leader key with a long-poll request in the main thread.
- `Ha.wakeup()` only sets `dcs.event`, which doesn't interrupt that request. For example, a replica applies a `PATCH /config` sent to it, or reacts to the end of a `POST /restart`, only when the long-poll returns, up to `loop_wait` seconds later.
- Only an exception raised by a signal handler interrupts that request. This is how the SIGTERM handler stops Patroni today.
- #3805 stops raising `SystemExit` from the handler, because that can leak locks, and calls `Ha.wakeup()` instead. Then a replica starts to shut down only when the long-poll returns, up to `loop_wait` seconds after SIGTERM.
## Fix
New module `patroni/interrupt.py`:
- **`Interrupter`**: `interrupt()` sets a flag and writes a byte to a non-blocking `socketpair()` to wake up blocked reads.
- It doesn't block, take locks or raise, so it is safe to call from other threads and from signal handlers, on every platform.
- The interruption is sticky until `clear()`, which also discards the bytes.
- Only the flag means that it was interrupted. Windows emulates `socketpair()` over loopback TCP, so a byte can become readable later, even after `clear()`.
- `is_set()` checks the flag, so it doesn't depend on when the byte becomes readable.
- Discarding bytes can't lose an `interrupt()`. It stops once the flag is set. If it has already consumed the new byte, it writes the byte again, so the socket stays readable while interrupted.
- **HTTP responses received while an `Interrupter` is `armed()`** in the current thread wait for their socket and for the interrupter before every read.
- They use `poll()`, or `select()` where `poll()` isn't available (Windows).
- Once interrupted, the read raises `WatchInterrupted`.
- A byte that becomes readable after `clear()` doesn't interrupt the read. The read discards it and keeps waiting, without extending its timeout.
- The response stays interruptible after `armed()` is left, so a watch stream only needs to be armed while the request is sent.
- `WatchInterrupted` is derived from `BaseException`, so the `except Exception` clauses of python-etcd, python-consul and Patroni's retries don't catch it.
- urllib3 closes the connection, as on any error during a read, so it isn't reused.
- The socket timeout still applies, i.e. `ReadTimeoutError` is raised as before.
- Supported sockets are `socket.socket`, `ssl.SSLSocket` and pyOpenSSL's `WrappedSocket`. Other socket types are read as before, without interruption.
- Decrypted data that TLS has buffered is read without waiting.
- The read itself doesn't block, because a readable socket doesn't guarantee a complete TLS record.
- **`make_interruptible(pool_manager)`** replaces `pool_classes_by_scheme` of the pool manager with connection pools whose connections use the interruptible `response_class`.
- Nothing is monkey-patched. Only the pool managers of Patroni's etcd, etcd3, Consul and Kubernetes clients are affected.
- Its HTTPS pool class is public as `InterruptibleHTTPSConnectionPool`. Custom HTTPS pools of such pool managers must be based on it to keep responses interruptible.
- etcd's SAN-only `EtcdHTTPSConnectionPool` from #3608 is based on it. `_configure_tls()` installs it after `make_interruptible()` with urllib3 < 2 and `hostname_checks_common_name: false`.
Usage:
- **etcd3 and Kubernetes:** `KVCache` and `ObjectCache` derive from the new `WatchStreamThread`. It owns an `Interrupter`, the readiness flag of the cache and the `Condition` that guards the flag. Both Kubernetes caches share one `Condition`.
- `kill_stream()` marks the cache as not ready and calls `interrupt()`, both while holding the condition. Once it returns, `get_cluster()` waits for a fresh list.
- `_build_cache()` clears the interrupter before the list request. Then, while holding the condition, it either discards the result and lists again, if `kill_stream()` was called in the meantime, or publishes it, marks the cache as ready and wakes up all waiting readers.
- `_do_watch()` arms the interrupter while sending the watch request, and finishes the response in `finally`.
- `_build_cache()` logs `Watch request interrupted` at DEBUG.
- etcd3: `get_cluster()` waits for the cache with the new `KVCache.snapshot()`, which copies it while holding the condition. It replaces `_wait_cache()`.
- etcd3: when a request of the `KVCache` thread itself switches to another etcd node, it doesn't interrupt itself. Its next watch request goes to the new node anyway.
- Exception: if the list request succeeds only after a retry, and the membership refresh that follows selects a different node, the cache lists again from that node. Otherwise it could watch a restored cluster from the old cluster's revision.
- `start_watcher()` registers the cache before starting its thread, so this also applies to the first list.
- Removed: `_response`/`_response_lock` and the `shutdown()` code.
- Also removed, in etcd3: the Windows-only `read_timeout=ttl`, the then unused `read_timeout` parameter of `watchrange()`/`watchprefix()`, and suppression of `ReadTimeoutError`/`ProtocolError`.
- **etcd (v2) and Consul:** `Ha.wakeup()` calls the new `AbstractDCS.wakeup()`, which makes `watch()` return `True` right away, also if `watch()` starts waiting after the call. That `watch()` consumes the wakeup.
- The default implementation sets `dcs.event`, like `Ha.wakeup()` did, because the other DCS implementations wait for it.
- etcd (v2) and Consul derive from the new `AbstractLongPollingDCS`, which owns an `Interrupter`. Its `wakeup()` calls `interrupt()` after setting the event.
- Their `watch()` arms the interrupter around the long-poll.
- `watch()` clears the interrupter before it checks the event. This drops a stale interruption, but can't miss a new `wakeup()`. On `WatchInterrupted`, `watch()` clears the event and returns `True`.
- With #3805, the SIGTERM handler wakes up such replicas through `Ha.wakeup()` without further changes, because #3805's `SignalSafeEvent` makes `dcs.event` safe to set from a signal handler.
### Behavior changes
- `kill_stream()` logs at DEBUG instead of ERROR.
- etcd (v2) and Consul: `Ha.wakeup()` also wakes up replicas that wait in the long-poll. The interrupted connection is closed, and the next long-poll opens a new one.
- etcd3 on Windows: the watch request no longer has a read timeout, as on other platforms.
- Idle watches aren't re-established every `ttl` seconds anymore.
- TCP keepalive still detects dead connections.
- etcd3 and Kubernetes: a `kill_stream()` during the list request makes the cache thread discard the result and list again. On master the `kill_stream()` was discarded instead.
- etcd3 and Kubernetes: once `kill_stream()` returns, `get_cluster()` waits for a fresh list. On master it could still return the old snapshot until the cache thread noticed the closed socket, e.g. up to `ttl` seconds later with etcd3 on Windows.
- etcd3 and Kubernetes: a fresh list wakes up all threads that wait for it, e.g. the HA loop and REST API handlers. On master only one of them was woken up, and the others waited until their timeout (`retry_timeout`).
- As on master:
- A `kill_stream()` before the list request is discarded.
- A `kill_stream()` while the watch request is being sent takes effect when the response is read.
- Each `Interrupter` holds a `socketpair()`, i.e. two sockets (loopback TCP on Windows).
- etcd (v2) and Consul: one per DCS object.
- etcd3: one.
- Kubernetes: two.
## Alternatives considered
- **Raising an exception from the SIGTERM handler only while in the long-poll:**
- It only helps etcd (v2) and Consul on SIGTERM, not `Ha.wakeup()` or `kill_stream()` from other threads.
- It could still leak a lock inside urllib3.
- **An `Interrupter`-based `dcs.event`**, so that setting the event interrupts the long-poll: it would change the event of every DCS implementation and conflict with #3805's `SignalSafeEvent`.
- **Read timeouts**, like the etcd3 workaround on Windows: they delay the interruption by up to the timeout, and shorter timeouts mean more reconnects of idle watches.
- **Monkey-patching `http.client` or urllib3** would affect every HTTP client in the process. `make_interruptible()` changes only Patroni's pool managers.
## Testing
- **Unit tests** use real sockets: `HangingHTTPServer` sends response headers or a chunk and then keeps the connection open.
- `tests/test_interrupt.py` covers:
- interrupting a long-poll that waits for the headers or the body;
- interrupting a stream;
- the sticky interruption, also before the wakeup byte is readable;
- a wakeup byte that becomes readable only after `clear()`: the next read discards it and times out instead of being interrupted;
- an `interrupt()` while a read discards a stale byte: the read is interrupted, and the socket stays readable;
- read timeouts.
- `test_kill_stream()` in `test_etcd3.py` and `test_kubernetes.py` interrupts `_do_watch()` through Patroni's own pool managers. `test_kill_stream_before_watch()` calls `kill_stream()` during the list request and before the watch request.
- `test_kill_stream_while_publishing()` in both files calls `kill_stream()` right before `_build_cache()` publishes the list, and checks that it isn't published. `test_readers_wait_after_kill_stream()` checks that two concurrent `get_cluster()` calls wait for a fresh list after `kill_stream()`, and that the list wakes up both.
- etcd3:
- `test_watch_after_list_failover()`: a list that succeeds after a retry is watched on the responding node, or listed again if the membership refresh selects another node.
- `test__restart_watcher_from_kv_cache()`: the `KVCache` thread doesn't interrupt itself when it switches to another node.
- `test_snapshot_timeout()` and `test_start_watcher_registers_before_start()`.
- `test_wakeup()` in `test_etcd.py` and `test_consul.py` interrupts the long-poll from another thread in the same way, and checks that the next `watch()` long-polls again. `test_watch()` checks that a `wakeup()` before `watch()` skips the long-poll.
- `test_cn_fallback_false_on_urllib3_1x_keeps_responses_interruptible()` in `test_etcd.py`: the SAN-only pool installed by `_configure_tls()` keeps the interruptible connection class.
- CI runs them on Linux, macOS and Windows (`select()`).
- The touched tests also pass with urllib3 1.26.20 (Python 3.12) and 1.24.3 (Python 3.8).
- flake8 and isort are clean, and pyright reports no errors in the changed files.
- **`kill_stream()`** against real etcd 3.4.23 and k3s:
- Linux, Python 3.12, urllib3 2.8 unless noted.
- `kill_stream()` was called while the watch was idle.
| | master | this PR |
|---|---|---|
| etcd3: http, https and https + pyOpenSSL, 30 kills each | ERROR on every kill: `watchprefix failed: ProtocolError('Response ended prematurely')` or `ProtocolError("Connection broken: ConnectionResetError(104, ...)")` | DEBUG. Stream dropped ≤1.4ms after the kill, watch re-established ≤54ms |
| Kubernetes: https, 40 kills | ERROR on every kill: `ObjectCache.run ProtocolError('Response ended prematurely')` | DEBUG. Dropped ≤0.6ms, re-established ≤12ms |
| Kubernetes: https + pyOpenSSL, 40 kills | ERROR on every kill: `ObjectCache.run ValueError('file descriptor cannot be a negative integer (-1)')`. Re-established up to 1.01s later. The pool created 43 connections | DEBUG. Dropped ≤3.4ms, re-established ≤14ms. 3 connections |
| urllib3 1.26 + pyOpenSSL 24: etcd3 and Kubernetes, 15 kills each (master: 5) | Kubernetes: the stream wasn't dropped within 30s in 5 of 5 tries (`TypeError` from `WrappedSocket.shutdown()`). etcd3: ERROR on every kill | DEBUG. Dropped ≤0.6ms, re-established ≤51ms |
- **SIGTERM during the etcd (v2) or Consul long-poll:**
- Setup:
- A child process runs `dcs.watch()` as a replica with a 3s timeout.
- Its SIGTERM handler calls `dcs.wakeup()`.
- SIGTERM came either from another process at a random time, or from an in-process timer within 4ms. The timer hits the request setup, e.g. `putheader()`, `makefile()` or `_read_status()`.
- Each combination of etcd (v2) or Consul with http, https or https + pyOpenSSL ran 100 times with urllib3 2.8 and 50 times with urllib3 1.26.
- This PR:
- `watch()` returned at most 3ms after every signal (median 0.2-0.3ms).
- Afterwards no lock of the pool manager or the connection pools was held.
- Requests from another thread worked.
- The number of open fds was stable.
- #3805 alone (12 runs each, http): `watch()` returned only when the long-poll ended, median 3.0s (Consul) and 3.3s (etcd).
- **End-to-end** with #3805, without other changes (its `Patroni._wakeup()` calls `Ha.wakeup()`):
- PostgreSQL 16, `synchronous_mode: true`, `loop_wait: 10`.
- SIGTERM was sent to the synchronous standby at a random time, 5 runs each.
| SIGTERM → `Ha.shutdown()` | #3805 | #3805 + this PR |
|---|---|---|
| etcd (v2) | median 4.38s, max 9.32s | median 0.004s, max 0.006s |
| Consul | median 2.97s, max 7.16s | median 0.004s, max 0.005s |
All runs:
- exited with status 0;
- shut Postgres down in recovery, after the primary released the standby from synchronous replication;
- logged no errors.
- **Late wakeup bytes**, as with loopback TCP: a stress test delayed every byte by 0-3ms and called `_InterruptibleSocketIO.readinto()` directly, with `poll()` and with `select()`.
- Each round called `clear()`, read for 10ms without an `interrupt()`, and then read while another thread called `interrupt()` 1-3 times.
- 3,600 rounds: no `interrupt()` was missed, and no read without one was interrupted.
- For comparison, when a readable socket counted as an interruption, as in an earlier revision of this PR, 501 of 600 reads without an `interrupt()` were interrupted.
- **Wine** (Windows CPython 3.12) uses the `select()` path.
- The unit tests pass, and `kill_stream()` against etcd3 and k3s behaves as on Linux.
- However, master's `shutdown()` works under Wine as well, so the Windows CI job is the real check.