pg.ddx.io  patroni/patroni GitHub issues and pull requests  
help / color / mirror / Atom feed
From: arajkumar (@arajkumar) <noreply+arajkumar@github.com>
To: patroni/patroni <noreply+patroni-patroni@github.com>
Subject: [patroni/patroni] issue #3819: Graceful stop waits for idle REST API connections, so the leader key can expire while PostgreSQL still accepts writes
Date: Fri, 20 Nov 2026 05:59:43 +0000
Message-ID: <gh-patroni-patroni-3819@github.com> (raw)

### What happened?

A graceful stop (SIGTERM) shuts down the REST API before PostgreSQL, and since #3526 `RestApiServer.shutdown()` calls `self._executor.shutdown(wait=True)`. So the stop waits for each REST API worker, and a worker waits for as long as its client keeps the connection open. `py-spy dump` shows two cases:

1. A connection with no request yet (HTTP or HTTPS): the worker blocks in `rfile.readline()` in `handle_one_request`.
2. HTTPS only: the client got a response but keeps the connection open and does not read from it (an idle pooled connection). The worker blocks in `SSLSocket.unwrap()` in `shutdown_request`, until the client sends close_notify.

While Patroni waits, the HA loop does not run. On the leader, the leader key expires, a replica promotes, and the old primary still accepts writes. When the old leader starts again, `pg_rewind` removes the transactions that it committed after the promotion. Clients got a commit acknowledgement for them.

Affected: 4.0.8 to 4.0.11, 4.1.1 to 4.1.5 (4.0.7 and 4.1.0 used daemon threads).

#3706 (on master, `REL_4_1` and `REL_4_0`, not released yet) bounds this wait with `restapi.request_timeout`, default 5 s. With the default, master is fine in our test. But the stop still waits for clients, up to `request_timeout`. With `request_timeout: 60` on master we get the same data loss.

### How can we reproduce it (as minimally and precisely as possible)?

The script [`repro.sh`](https://gist.github.com/arajkumar/527f6a7d35c80f27d6871909aeb4d3a5) builds the `Dockerfile` of this repository at a pinned ref and uses its `docker-compose.yml` (etcd and three Patroni nodes). It stops a replica, then the leader under a writer, each with one idle client on the REST API for 60 s. Then it counts the acknowledged rows that are missing on the new leader. It needs docker with `docker compose`, git and bash.

```
curl -LO https://gist.githubusercontent.com/arajkumar/527f6a7d35c80f27d6871909aeb4d3a5/raw/repro.sh
bash repro.sh                                  # v4.1.5, HTTP, no request on the idle connection
RESTAPI_TLS=1 IDLE_REQUEST=1 bash repro.sh     # v4.1.5, HTTPS, idle after one request
PATRONI_REF=master MODES=stock bash repro.sh   # master
```

Results (ttl 30, loop_wait 10, two runs per row unless noted):

| Version, case | Replica stop, idle client | Leader stop | Lost acknowledged rows |
|---|---|---|---|
| 4.1.5, HTTP, no request | 60.4 s, 60.6 s | 60.6 s, 60.5 s | 540 of 963, 469 of 944 |
| 4.1.5, HTTPS, idle after one request | 60.5 s, 60.5 s | 60.5 s, 60.5 s | 425 of 962, 401 of 940 |
| 4.1.5, no client | 1.8 s, 1.9 s | | |
| master 522f991, both cases above | 5.3 s to 5.4 s | 5.5 s to 6.2 s | 0 |
| master 522f991, HTTP, `request_timeout: 60` (one run) | 60.4 s | 60.5 s | 464 of 962 |

In each case with lost rows, a replica promoted 23 s to 43 s after SIGTERM, and the old leader log shows `pg_rewind: servers diverged at WAL location ... on timeline 1`.

<details>
<summary>py-spy dump during the stop (4.1.5)</summary>

```
Thread 7 (idle): "MainThread"
    Thread.join (threading.py:1094)
    ThreadPoolExecutor.shutdown (concurrent/futures/thread.py:239)
    RestApiServer.shutdown (patroni/api.py:1854)
    Patroni._shutdown (patroni/__main__.py:248)
Thread 17 (idle): "RestAPI_0"                      # case 1
    SocketIO.readinto (socket.py:719)
    BaseHTTPRequestHandler.handle_one_request (http/server.py:404)
    RestApiHandler.handle_one_request (patroni/api.py:1458)
Thread 20 (idle): "RestAPI_3"                      # case 2
    SSLSocket.unwrap (ssl.py:1348)
    RestApiServer.shutdown_request (patroni/api.py:1869)
    ThreadingMixIn.process_request_thread (socketserver.py:701)
```

</details>

### What did you expect to happen?

A graceful stop does not depend on REST API clients. On the leader, the time from SIGTERM to the release of the leader key must fit in about `ttl - loop_wait` minus the PostgreSQL stop time, and each second of wait for clients uses part of it. `request_timeout` has no relation to `ttl`. Some options:

- In `RestApiServer.shutdown()`, close the connections that have no request in progress (cases 1 and 2), and do not wait for them.
- Wait for workers that still run a handler only for a short time, then continue with `Ha.shutdown()`.
- Log a warning when `request_timeout` is close to `ttl - loop_wait`.

Also, the release note for #3706 can say that it prevents this slow stop and data loss on the leader.

### Patroni/PostgreSQL/DCS version

- Patroni: 4.1.5, and master 522f991
- PostgreSQL: 17
- DCS: etcd (all from the repository Dockerfile)

### Patroni configuration file

```yaml
Default config of the repository docker-compose.yml. The HTTPS case adds
PATRONI_RESTAPI_CERTFILE, PATRONI_RESTAPI_KEYFILE and PATRONI_CTL_INSECURE=true.
```

### patronictl show-config

```yaml
ttl: 30
loop_wait: 10
retry_timeout: 10
maximum_lag_on_failover: 1048576
postgresql:
  use_pg_rewind: true
```

### Patroni log files

```shell
The script prints the pg_rewind lines of the old leader.
```

### PostgreSQL log files

```shell
Not needed.
```

### Have you tried to use GitHub issue search?

- [x] Yes

### Anything else we need to know?

Workaround on 4.1.5: set `patroni.api.RestApiHandler.timeout = 5` in a wrapper entry point before `main()`. It does not bound the TLS handshake. `restapi.handshake_timeout` from #3706 does.


Message-ID: <gh-patroni-patroni-3819@github.com>
Permalink:  ../gh-patroni-patroni-3819@github.com/

 · 


This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox