From: shriharidanawe (@shriharidanawe) <noreply+shriharidanawe@github.com>
To: patroni/patroni <noreply+patroni-patroni@github.com>
Subject: [patroni/patroni] issue #3802: Election deadlock: members defer to a peer that is ahead on LSN but on an older timeline (check_timeline: true), nobody promotes (4.1.5, Kubernetes Endpoints, synchronous_mode)
Date: Wed, 18 Nov 2026 04:43:31 +0000
Message-ID: <gh-patroni-patroni-3802@github.com> (raw)
### What happened?
In a three-member cluster with `synchronous_mode: true`, `check_timeline: true` and `maximum_lag_on_failover: 1048576` (the value from the sample `postgres0.yml`), the cluster can end up with no leader and no member able to take the leader lock. It stays that way until someone restarts a member manually. The lag threshold plays no role here: both members that could be promoted pass the lag check (the log has no `My wal position exceeds maximum replication lag` line); they are stopped later, by the comparison with their peers.
The state it ends in (the run attached, still stuck 6 minutes after the last member reconnected):
```
+ Cluster: robin-patroni (7709344756141768815) ------+----+-------------+-----+------------+-----+
| Member | Host | Role | State | TL | Receive LSN | Lag | Replay LSN | Lag |
+-----------------+--------------+---------+---------+----+-------------+-----+------------+-----+
| robin-patroni-0 | 192.168.0.56 | Replica | running | 3 | 0/7000000 | 0 | 0/749CA58 | 0 |
| robin-patroni-1 | 192.168.1.83 | Leader | running | 4 | | | | |
| robin-patroni-2 | 192.168.2.77 | Replica | running | 4 | 0/60126A8 | 0 | 0/60126A8 | 0 |
+-----------------+--------------+---------+---------+----+-------------+-----+------------+-----+
```
`robin-patroni-1` is shown as "Leader" only because the leader key still names it; its `renewTime` is stale and Postgres is in recovery. `robin-patroni-0` is the old leader. It is on timeline 3, but it is **ahead on LSN** because it wrote WAL while it was cut off, and that WAL never reached the other members.
How the members behave, every HA loop (5 s), indefinitely:
- `robin-patroni-1` and `robin-patroni-2` (timeline 4) run the election, compare their LSN with `robin-patroni-0`, find it higher, and stop. They never look at the timeline of `robin-patroni-0`:
```
INFO: Got response from robin-patroni-0 http://192.168.0.56:8008/patroni: {"state": "running", ..., "role": "replica", ...}
INFO: Wal position of robin-patroni-0 is ahead of my wal position
INFO: following a different leader because i am not the healthiest node
```
- `robin-patroni-0` (timeline 3) rejects itself because of `check_timeline`:
```
INFO: My timeline 3 is behind last known cluster timeline 4
INFO: following a different leader because i am not the healthiest node
```
So the members that could be promoted wait for a member that can never be promoted. In `patroni/ha.py` (4.1.5), `_is_healthiest_node()` applies `check_timeline` to the local node only (lines 1313-1319). The loop over the other members (lines 1348-1359) compares `my_wal_position < st.wal_position` and returns False. `st.timeline` is never read there.
In our runs the members got into the race with an empty synchronous record: the `<scope>-sync` Endpoints object was missing. With Kubernetes Endpoints as the DCS, that object has no Service, and the kube-controller-manager endpoints controller deletes it when the controller-manager lease changes hands. Patroni recreates it only from a running leader. In the reproduction no leader exists at that moment, so it stays missing. With the record absent, `sync_mode_is_active()` is False and all three members race, which exposes the peer comparison above.
**Where we first saw it.** This happened on a production cluster (an older Patroni release; Kubernetes Endpoints as DCS, `synchronous_mode`, three members). All three nodes lost connectivity to the kube-apiserver for about 16 minutes, without rebooting. During that time the members demoted and promoted in turn, and the `kube-controller-manager` lease changed holders 8 times in 12 minutes. Each new lease holder's endpoints controller deleted the Service-less `<scope>-sync` Endpoints object. The second delete happened while no Patroni leader existed, so nothing recreated it. When the API server came back, all three members raced with an empty synchronous record. The end state was the same as above: the member on the newest timeline deferred to a member that was on an older timeline but ahead on LSN, and the cluster had no primary for 2 h 27 min until a member was restarted by hand. The audit-log events are in `production_endpoint_and_lease_events.txt`. The reproduction in this issue is on 4.1.5.
### How can we reproduce it (as minimally and precisely as possible)?
Cluster: three members `robin-patroni-0/1/2`, Patroni 4.1.5, PostgreSQL 15.19, Kubernetes Endpoints as DCS (`kubernetes.use_endpoints: true`), `synchronous_mode: true`, `synchronous_mode_strict: true`, `check_timeline: true`, `maximum_lag_on_failover: 1048576`, `use_pg_rewind: true`, `loop_wait 5`, `ttl 20`, `retry_timeout 7`. Start with a healthy cluster: leader Y (`robin-patroni-0`), sync standby X (`robin-patroni-1`), third member Z (`robin-patroni-2`).
The attached script `repro_deadlock_endpoints_controller.sh` runs all of the steps below (needs `kubectl`, `sshpass` and SSH access to the control-plane node that holds the `kube-controller-manager` lease). The manual steps:
1. **Isolate the leader Y** from the DCS and its peers (a Kubernetes NetworkPolicy with deny-all ingress and egress, selecting a label that is added to the pod). While it is cut off, write WAL on Y that the others never get:
```
psql -c "set synchronous_commit=local; insert into repro_t(pad) select repeat('x',1000) from generate_series(1,2000)"
```
Y logs `demoting self because DCS is not accessible and I was a leader`. After that, terminate its walsenders so its Postgres shuts down quickly (otherwise the shutdown hangs on the walsenders and the liveness probe restarts the container, which changes the scenario):
```
select pg_terminate_backend(pid) from pg_stat_replication;
```
2. **X promotes** to timeline 4 (`patronictl list`). Y is on timeline 3 and ahead on LSN.
3. **Isolate X and Z** the same way. Wait until all three Postgres instances are in recovery and the leader key is older than `ttl` (no leader).
4. **Fail the `kube-controller-manager` lease over** to another control-plane node (we moved the static pod manifest of the current lease holder out of `/etc/kubernetes/manifests` for about 20 seconds, then put it back). The new lease holder's endpoints controller deletes `<scope>-sync` (it has no Service). This is what happened in production, where the lease changed holders repeatedly while the API server was unreachable. Check: `kubectl -n <ns> get endpoints <scope>-sync` returns NotFound and stays NotFound. (Deleting the object by hand with `kubectl delete endpoints <scope>-sync` should have the same effect for the purpose of this report; we did not run that variant separately.)
5. **Reconnect Y first**, and wait until it logs `following a different leader because i am not the healthiest node`.
6. **Reconnect X and Z together.**
Result: from this point every HA loop logs the lines shown in "What happened". Nothing promotes. In the run attached here the cluster was still stuck 6 minutes after step 6 (04:02:34 to 04:08:42 UTC) and no pod had restarted.
Timeline of the attached run (UTC, 2026-11-18; `maximum_lag_on_failover` set with `patronictl edit-config -s maximum_lag_on_failover=1048576` before the run):
| Time | Event |
|---|---|
| 03:59:11 | Y (`robin-patroni-0`) isolated; WAL written on it |
| 03:59:29 | Y demotes ("demoting self because DCS is not accessible") |
| 03:59:37 | X (`robin-patroni-1`) promotes, timeline 4 |
| 04:00:05 | X and Z isolated; X demotes at 04:00:20; all three demoted by 04:01:33 |
| 04:01:52 | `kube-controller-manager` lease moves to another node (transitions 10 to 11) |
| 04:01:54 | `<scope>-sync` deleted by the endpoints controller; not recreated |
| 04:01:56 | Y reconnected alone; it logs `following a different leader because i am not the healthiest node` |
| 04:02:34 | X and Z reconnected; the deadlock starts |
### What did you expect to happen?
One of the members on the newest timeline (`robin-patroni-1` or `robin-patroni-2`) takes the leader lock and promotes. The old leader (`robin-patroni-0`) then rewinds or is reinitialised and rejoins as a replica.
A member should not defer its own promotion to a peer that cannot be promoted. With `check_timeline: true`, a peer whose timeline is behind the cluster timeline is not a valid candidate (the member itself applies exactly this rule to itself in lines 1313-1319, and `is_failover_possible()` applies it to peers: `ha.py:1414`, `elif self.check_timeline() and (not st.timeline or st.timeline < cluster_timeline)`). The peer loop in `_is_healthiest_node()` should apply the same rule, so the decision compares timeline and LSN, not LSN alone.
### Patroni/PostgreSQL/DCS version
- Patroni version: 4.1.5
- PostgreSQL version: 15.19
- DCS (and its version): Kubernetes API (Endpoints, `use_endpoints: true`), Kubernetes v1.34.1 (GKE-based), three control-plane nodes
### Patroni configuration file
```yaml
bootstrap:
dcs:
failsafe_mode: true
loop_wait: 5
maximum_lag_on_failover: 33554432
postgresql:
parameters:
archive_mode: 'off'
archive_timeout: '0'
autovacuum_analyze_scale_factor: 0.02
autovacuum_max_workers: 5
autovacuum_vacuum_scale_factor: 0.05
checkpoint_completion_target: 0.9
hot_standby: 'on'
huge_pages: 'off'
log_autovacuum_min_duration: 0
log_checkpoints: 'on'
log_connections: 'on'
log_disconnections: 'on'
log_line_prefix: '%t [%p]: [%l-1] %c %x %d %u %a %h '
log_lock_waits: 'on'
log_min_duration_statement: 500
log_statement: ddl
log_temp_files: 0
max_connections: '400'
max_replication_slots: 10
max_wal_senders: 10
ssl_ciphers: HIGH:MEDIUM:-3DES:!aNULL
synchronous_standby_names: '*'
tcp_keepalives_count: '2'
tcp_keepalives_idle: '10'
tcp_keepalives_interval: '5'
track_functions: all
wal_compression: 'on'
wal_keep_segments: '100'
wal_keep_size: 2GB
wal_level: hot_standby
wal_log_hints: 'on'
use_pg_rewind: true
use_slots: true
retry_timeout: 7
slots:
robin_patroni_0:
type: physical
robin_patroni_1:
type: physical
robin_patroni_2:
type: physical
synchronous_mode: true
synchronous_mode_strict: true
ttl: 20
initdb:
- auth-host: scram-sha-256
- auth-local: trust
- data-checksums
- encoding: UTF8
- locale: en_US.UTF-8
post_init: /scripts/post_init.sh "zalandos"
kubernetes:
bootstrap_labels:
critical-operation: 'true'
bypass_api_service: true
labels:
application: spilo
leader_label_value: master
pod_ip: 192.168.0.112
port: tcp://10.96.0.1:443
port_443_tcp: tcp://10.96.0.1:443
port_443_tcp_addr: 10.96.0.1
port_443_tcp_port: '443'
port_443_tcp_proto: tcp
ports:
- name: postgresql
port: 5432
role_label: spilo-role
scope_label: cluster-name
service_host: 10.96.0.1
service_port: '443'
service_port_https: '443'
standby_leader_label_value: master
use_endpoints: true
namespace: robinio
postgresql:
authentication:
replication:
password: <redacted>
username: replicator
superuser:
password: <redacted>
username: postgres
basebackup_fast_xlog:
command: /scripts/basebackup.sh
retries: 2
bin_dir: /usr/lib/postgresql/15/bin
callbacks:
on_role_change: /usr/bin/python3 /home/postgres/pgdata/pgroot/callback/rename_partial_wal.py
connect_address: 192.168.0.112:5432
create_replica_method:
- basebackup_fast_xlog
data_dir: /home/postgres/pgdata/pgroot/data
listen: '*:5432'
name: robin-patroni-0
parameters:
archive_command: /bin/true
bg_mon.history_buckets: 120
bg_mon.listen_address: '::'
extwlist.custom_path: /scripts
extwlist.extensions: btree_gin,btree_gist,citext,extra_window_functions,first_last_agg,hll,hstore,hypopg,intarray,ltree,pgcrypto,pgq,pgq_node,pg_trgm,postgres_fdw,roaringbitmap,tablefunc,uuid-ossp,vector,timescaledb,pg_partman
log_destination: csvlog
log_directory: ../pg_log
log_file_mode: '0644'
log_filename: postgresql-%u.log
log_rotation_age: 1d
log_truncate_on_rotation: 'on'
logging_collector: 'on'
output_plugin_libraries: pgoutput, test_decoding, pglogical_output, wal2json, decoderbufs
pg_stat_statements.track_utility: 'off'
shared_buffers: 1024MB
shared_preload_libraries: bg_mon,pg_stat_statements,pgextwlist,pg_auth_mon,set_user,timescaledb,pg_cron,pg_stat_kcache
ssl: 'on'
ssl_cert_file: /run/certs/server.crt
ssl_key_file: /run/certs/server.key
pg_hba:
- local all all trust
- hostssl all +zalandos 127.0.0.1/32 pam
- host all all 127.0.0.1/32 md5
- hostssl all +zalandos ::1/128 pam
- host all all ::1/128 md5
- local replication replicator trust
- hostssl replication replicator all md5
- hostnossl all all all reject
- hostssl all +zalandos all pam
- hostssl all all all md5
pgpass: /run/postgresql/pgpass
use_unix_socket: true
use_unix_socket_repl: true
restapi:
connect_address: 192.168.0.112:8008
listen: :8008
scope: robin-patroni
```
### patronictl show-config
```yaml
check_timeline: true
failsafe_mode: true
loop_wait: 5
maximum_lag_on_failover: 1048576
postgresql:
callbacks:
on_role_change: /home/postgres/pgdata/pgroot/callback/rename_partial_wal.py
parameters:
archive_mode: 'off'
archive_timeout: '0'
autovacuum_analyze_scale_factor: 0.02
autovacuum_max_workers: 5
autovacuum_vacuum_scale_factor: 0.05
checkpoint_completion_target: 0.9
hot_standby: 'on'
huge_pages: 'off'
log_autovacuum_min_duration: 0
log_checkpoints: 'on'
log_connections: 'on'
log_disconnections: 'on'
log_line_prefix: '%t [%p]: [%l-1] %c %x %d %u %a %h '
log_lock_waits: 'on'
log_min_duration_statement: 500
log_statement: ddl
log_temp_files: 0
max_connections: '400'
max_replication_slots: 10
max_wal_senders: 10
ssl_ciphers: HIGH:MEDIUM:-3DES:!aNULL
synchronous_standby_names: '*'
tcp_keepalives_count: '2'
tcp_keepalives_idle: '10'
tcp_keepalives_interval: '5'
track_functions: all
wal_compression: 'on'
wal_keep_segments: '100'
wal_keep_size: 2GB
wal_level: hot_standby
wal_log_hints: 'on'
remove_data_directory_on_diverged_timelines: true
remove_data_directory_on_rewind_failure: true
use_pg_rewind: true
use_slots: true
retry_timeout: 7
slots:
robin_patroni_0:
type: physical
robin_patroni_1:
type: physical
robin_patroni_2:
type: physical
synchronous_mode: true
synchronous_mode_strict: true
ttl: 20
```
### Patroni log files
```shell
Old leader `robin-patroni-0` loses the DCS and demotes (03:59:29), then `robin-patroni-1` promotes (03:59:37) and later loses the DCS too (04:00:20):
robin-patroni-0: 2026-11-18 03:59:29,578 INFO: demoting self because DCS is not accessible and I was a leader
robin-patroni-0: 2026-11-18 03:59:32,518 INFO: demoted self because DCS is not accessible and I was a leader
robin-patroni-1: 2026-11-18 03:59:37,172 INFO: promoted self to leader by acquiring session lock
robin-patroni-1: 2026-11-18 03:59:40,315 INFO: Synchronous standby status assigned to ['robin-patroni-2']
robin-patroni-1: 2026-11-18 04:00:20,431 INFO: demoting self because DCS is not accessible and I was a leader
The stuck state, one HA loop per member (this repeats every 5 s):
robin-patroni-1: 2026-11-18 04:03:20,816 INFO: Got response from robin-patroni-0 http://192.168.0.56:8008/patroni: {"state": "running", ..., "role": "replica", ...}
robin-patroni-1: 2026-11-18 04:03:20,817 INFO: Wal position of robin-patroni-0 is ahead of my wal position
robin-patroni-1: 2026-11-18 04:03:20,917 INFO: following a different leader because i am not the healthiest node
robin-patroni-2: 2026-11-18 04:03:23,919 INFO: Got response from robin-patroni-0 http://192.168.0.56:8008/patroni: {"state": "running", ..., "role": "replica", ...}
robin-patroni-2: 2026-11-18 04:03:23,938 INFO: Wal position of robin-patroni-0 is ahead of my wal position
robin-patroni-2: 2026-11-18 04:03:23,941 INFO: following a different leader because i am not the healthiest node
robin-patroni-0: 2026-11-18 04:03:23,336 INFO: My timeline 3 is behind last known cluster timeline 4
robin-patroni-0: 2026-11-18 04:03:23,340 INFO: following a different leader because i am not the healthiest node
(`robin-patroni-2` also logs `Wal position of robin-patroni-1 is ahead of my wal position` in some loops, because `robin-patroni-1`, the previous leader on timeline 4, reports a slightly higher LSN. Those two members are on the same timeline, so that is a legitimate tie-break; what blocks the cluster is `robin-patroni-0`.)
Counts in the attached logs: "Wal position of robin-patroni-0 is ahead of my wal position" 36 times on `robin-patroni-1` and 28 times on `robin-patroni-2`; "My timeline 3 is behind last known cluster timeline 4" 39 times on `robin-patroni-0`; "exceeds maximum replication lag" 0 times. Full logs are attached: `patroni-log-robin-patroni-0.log`, `patroni-log-robin-patroni-1.log`, `patroni-log-robin-patroni-2.log`.
DCS state while stuck: the leader endpoint still names `robin-patroni-1` with `renewTime` 2026-11-18T04:00:05 (stale), and `robin-patroni-sync` does not exist:
$ kubectl -n robinio get endpoints robin-patroni-sync
Error from server (NotFound): endpoints "robin-patroni-sync" not found
(`lease_and_dcs_watch.log` shows the `kube-controller-manager` lease change at 04:01:52 and the sync Endpoints object disappearing at 04:01:54.)
```
### PostgreSQL log files
```shell
Postgres log excerpts (csvlog, UTC), full files attached as `postgresql-log-robin-patroni-{0,1,2}.csv` (03:59 to 04:09):
robin-patroni-0 (old leader)
03:59:29.597 received fast shutdown request
03:59:29.892 database system is shut down
03:59:31.519 entering standby mode
03:59:31.522 waiting for WAL to become available at 0/7002000 <- it has WAL beyond the point where timeline 4 forked
robin-patroni-1 (new leader)
03:59:37.185 received promote request
03:59:37.188 selected new timeline ID: 4
03:59:37.536 standby "robin-patroni-2" is now a synchronous standby with priority 1
04:00:20.452 received fast shutdown request
04:00:22.227 entering standby mode
04:00:22.229 waiting for WAL to become available at 0/6002000
robin-patroni-2
03:59:35.157 WAL receiver process shutdown requested
03:59:37.205 started streaming WAL from primary at 0/5000000 on timeline 3
```
### Have you tried to use GitHub issue search?
- [x] Yes
### Anything else we need to know?
- **Where in the code (4.1.5, `patroni/ha.py`):** `_is_healthiest_node()` checks the local timeline at lines 1313-1319, then loops over peers at lines 1348-1359 and returns False as soon as `my_wal_position < st.wal_position`. The only exemption is "the former leader being ahead of us" and only when `sync_mode_is_active()` and the peer is the record's leader (line 1358). `is_failover_possible()` already uses peer timelines (line 1414).
- **Why the synchronous record matters:** with `synchronous_mode` and a non-empty record, only the record's leader and sync standbys take part in the election, which hid this in practice. With the record missing (here because the Endpoints object was deleted), all members race and the peer comparison is the only protection. The election logic is independent of how the record became empty.
- **Kubernetes Endpoints trigger:** only `<scope>-config` gets a headless Service (`_create_config_service`, with a comment that it would otherwise be "cleaned" by the k8s master). `<scope>-sync` has no Service. On a controller-manager lease change, the endpoints controller deletes Service-less Endpoints objects. We confirmed it by watching the object while failing the lease over with no leader running: it was deleted within about 2 seconds and not recreated (`lease_and_dcs_watch.log`). Only the leader writes the sync object (`process_sync_replication()` and the promote path in `ha.py`).
- **Production occurrence and what deleted the Endpoints object:** the audit log shows `endpoints.delete robin-patroni-sync` by `system:serviceaccount:kube-system:endpoint-controller` at 08:57:35 and 08:59:37 (UTC, 2026-11-14), 0.4 s and 2.2 s after a new `kube-controller-manager` lease holder started. The first delete was repaired within 2 s by the member that had just promoted (it created the object again); the second happened with no leader and the object stayed missing until 11:26:54, after a manual restart. The lease changed holders 8 times between 08:57:35 and 09:09:02 (transitions 113 to 120), while the nodes had lost the API server. Details: `production_endpoint_and_lease_events.txt`. We can provide the members' logs from that occurrence on request.
- **Proposed fix:** in the peer loop of `_is_healthiest_node()`, when `check_timeline` is on, skip a peer whose timeline is behind the cluster timeline (same condition as in `is_failover_possible()`), so a peer that cannot be promoted never makes a valid candidate defer. We have not tested a patch yet and are happy to open a PR.
- **Workaround:** restart the member that is ahead on LSN (here `robin-patroni-2`). Once it no longer answers as an "ahead" replica, the other members promote at their next loop. In the production occurrence, deleting the pod of that member made the member on the newest timeline promote about 0.2 s after that member's Postgres stopped (not tested separately in this reproduction).
- Reproduced without any manual change to the DCS and without pod restarts. The script and the full logs are attached.