pg.ddx.io  patroni/patroni GitHub issues and pull requests  
help / color / mirror / Atom feed
From: shriharidanawe (@shriharidanawe) <noreply+shriharidanawe@github.com>
To: patroni/patroni <noreply+patroni-patroni@github.com>
Subject: [patroni/patroni] issue #3802: Election deadlock: members defer to a peer that is ahead on LSN but on an older timeline (check_timeline: true), nobody promotes (4.1.5, Kubernetes Endpoints, synchronous_mode)
Date: Wed, 18 Nov 2026 04:43:31 +0000
Message-ID: <gh-patroni-patroni-3802@github.com> (raw)

### What happened?

In a three-member cluster with `synchronous_mode: true`, `check_timeline: true` and `maximum_lag_on_failover: 1048576` (the value from the sample `postgres0.yml`), the cluster can end up with no leader and no member able to take the leader lock. It stays that way until someone restarts a member manually. The lag threshold plays no role here: both members that could be promoted pass the lag check (the log has no `My wal position exceeds maximum replication lag` line); they are stopped later, by the comparison with their peers.

The state it ends in (the run attached, still stuck 6 minutes after the last member reconnected):

```
+ Cluster: robin-patroni (7709344756141768815) ------+----+-------------+-----+------------+-----+
| Member          | Host         | Role    | State   | TL | Receive LSN | Lag | Replay LSN | Lag |
+-----------------+--------------+---------+---------+----+-------------+-----+------------+-----+
| robin-patroni-0 | 192.168.0.56 | Replica | running |  3 |   0/7000000 |   0 |  0/749CA58 |   0 |
| robin-patroni-1 | 192.168.1.83 | Leader  | running |  4 |             |     |            |     |
| robin-patroni-2 | 192.168.2.77 | Replica | running |  4 |   0/60126A8 |   0 |  0/60126A8 |   0 |
+-----------------+--------------+---------+---------+----+-------------+-----+------------+-----+
```

`robin-patroni-1` is shown as "Leader" only because the leader key still names it; its `renewTime` is stale and Postgres is in recovery. `robin-patroni-0` is the old leader. It is on timeline 3, but it is **ahead on LSN** because it wrote WAL while it was cut off, and that WAL never reached the other members.

How the members behave, every HA loop (5 s), indefinitely:

- `robin-patroni-1` and `robin-patroni-2` (timeline 4) run the election, compare their LSN with `robin-patroni-0`, find it higher, and stop. They never look at the timeline of `robin-patroni-0`:

```
INFO: Got response from robin-patroni-0 http://192.168.0.56:8008/patroni: {"state": "running", ..., "role": "replica", ...}
INFO: Wal position of robin-patroni-0 is ahead of my wal position
INFO: following a different leader because i am not the healthiest node
```

- `robin-patroni-0` (timeline 3) rejects itself because of `check_timeline`:

```
INFO: My timeline 3 is behind last known cluster timeline 4
INFO: following a different leader because i am not the healthiest node
```

So the members that could be promoted wait for a member that can never be promoted. In `patroni/ha.py` (4.1.5), `_is_healthiest_node()` applies `check_timeline` to the local node only (lines 1313-1319). The loop over the other members (lines 1348-1359) compares `my_wal_position < st.wal_position` and returns False. `st.timeline` is never read there.

In our runs the members got into the race with an empty synchronous record: the `<scope>-sync` Endpoints object was missing. With Kubernetes Endpoints as the DCS, that object has no Service, and the kube-controller-manager endpoints controller deletes it when the controller-manager lease changes hands. Patroni recreates it only from a running leader. In the reproduction no leader exists at that moment, so it stays missing. With the record absent, `sync_mode_is_active()` is False and all three members race, which exposes the peer comparison above.

**Where we first saw it.** This happened on a production cluster (an older Patroni release; Kubernetes Endpoints as DCS, `synchronous_mode`, three members). All three nodes lost connectivity to the kube-apiserver for about 16 minutes, without rebooting. During that time the members demoted and promoted in turn, and the `kube-controller-manager` lease changed holders 8 times in 12 minutes. Each new lease holder's endpoints controller deleted the Service-less `<scope>-sync` Endpoints object. The second delete happened while no Patroni leader existed, so nothing recreated it. When the API server came back, all three members raced with an empty synchronous record. The end state was the same as above: the member on the newest timeline deferred to a member that was on an older timeline but ahead on LSN, and the cluster had no primary for 2 h 27 min until a member was restarted by hand. The audit-log events are in `production_endpoint_and_lease_events.txt`. The reproduction in this issue is on 4.1.5.

### How can we reproduce it (as minimally and precisely as possible)?

Cluster: three members `robin-patroni-0/1/2`, Patroni 4.1.5, PostgreSQL 15.19, Kubernetes Endpoints as DCS (`kubernetes.use_endpoints: true`), `synchronous_mode: true`, `synchronous_mode_strict: true`, `check_timeline: true`, `maximum_lag_on_failover: 1048576`, `use_pg_rewind: true`, `loop_wait 5`, `ttl 20`, `retry_timeout 7`. Start with a healthy cluster: leader Y (`robin-patroni-0`), sync standby X (`robin-patroni-1`), third member Z (`robin-patroni-2`).

The attached script `repro_deadlock_endpoints_controller.sh` runs all of the steps below (needs `kubectl`, `sshpass` and SSH access to the control-plane node that holds the `kube-controller-manager` lease). The manual steps:

1. **Isolate the leader Y** from the DCS and its peers (a Kubernetes NetworkPolicy with deny-all ingress and egress, selecting a label that is added to the pod). While it is cut off, write WAL on Y that the others never get:
   ```
   psql -c "set synchronous_commit=local; insert into repro_t(pad) select repeat('x',1000) from generate_series(1,2000)"
   ```
   Y logs `demoting self because DCS is not accessible and I was a leader`. After that, terminate its walsenders so its Postgres shuts down quickly (otherwise the shutdown hangs on the walsenders and the liveness probe restarts the container, which changes the scenario):
   ```
   select pg_terminate_backend(pid) from pg_stat_replication;
   ```
2. **X promotes** to timeline 4 (`patronictl list`). Y is on timeline 3 and ahead on LSN.
3. **Isolate X and Z** the same way. Wait until all three Postgres instances are in recovery and the leader key is older than `ttl` (no leader).
4. **Fail the `kube-controller-manager` lease over** to another control-plane node (we moved the static pod manifest of the current lease holder out of `/etc/kubernetes/manifests` for about 20 seconds, then put it back). The new lease holder's endpoints controller deletes `<scope>-sync` (it has no Service). This is what happened in production, where the lease changed holders repeatedly while the API server was unreachable. Check: `kubectl -n <ns> get endpoints <scope>-sync` returns NotFound and stays NotFound. (Deleting the object by hand with `kubectl delete endpoints <scope>-sync` should have the same effect for the purpose of this report; we did not run that variant separately.)
5. **Reconnect Y first**, and wait until it logs `following a different leader because i am not the healthiest node`.
6. **Reconnect X and Z together.**

Result: from this point every HA loop logs the lines shown in "What happened". Nothing promotes. In the run attached here the cluster was still stuck 6 minutes after step 6 (04:02:34 to 04:08:42 UTC) and no pod had restarted.

Timeline of the attached run (UTC, 2026-11-18; `maximum_lag_on_failover` set with `patronictl edit-config -s maximum_lag_on_failover=1048576` before the run):

| Time | Event |
|---|---|
| 03:59:11 | Y (`robin-patroni-0`) isolated; WAL written on it |
| 03:59:29 | Y demotes ("demoting self because DCS is not accessible") |
| 03:59:37 | X (`robin-patroni-1`) promotes, timeline 4 |
| 04:00:05 | X and Z isolated; X demotes at 04:00:20; all three demoted by 04:01:33 |
| 04:01:52 | `kube-controller-manager` lease moves to another node (transitions 10 to 11) |
| 04:01:54 | `<scope>-sync` deleted by the endpoints controller; not recreated |
| 04:01:56 | Y reconnected alone; it logs `following a different leader because i am not the healthiest node` |
| 04:02:34 | X and Z reconnected; the deadlock starts |

### What did you expect to happen?

One of the members on the newest timeline (`robin-patroni-1` or `robin-patroni-2`) takes the leader lock and promotes. The old leader (`robin-patroni-0`) then rewinds or is reinitialised and rejoins as a replica.

A member should not defer its own promotion to a peer that cannot be promoted. With `check_timeline: true`, a peer whose timeline is behind the cluster timeline is not a valid candidate (the member itself applies exactly this rule to itself in lines 1313-1319, and `is_failover_possible()` applies it to peers: `ha.py:1414`, `elif self.check_timeline() and (not st.timeline or st.timeline < cluster_timeline)`). The peer loop in `_is_healthiest_node()` should apply the same rule, so the decision compares timeline and LSN, not LSN alone.


### Patroni/PostgreSQL/DCS version

-   Patroni version: 4.1.5
-   PostgreSQL version: 15.19
-   DCS (and its version): Kubernetes API (Endpoints, `use_endpoints: true`), Kubernetes v1.34.1 (GKE-based), three control-plane nodes


### Patroni configuration file

```yaml
bootstrap:
  dcs:
    failsafe_mode: true
    loop_wait: 5
    maximum_lag_on_failover: 33554432
    postgresql:
      parameters:
        archive_mode: 'off'
        archive_timeout: '0'
        autovacuum_analyze_scale_factor: 0.02
        autovacuum_max_workers: 5
        autovacuum_vacuum_scale_factor: 0.05
        checkpoint_completion_target: 0.9
        hot_standby: 'on'
        huge_pages: 'off'
        log_autovacuum_min_duration: 0
        log_checkpoints: 'on'
        log_connections: 'on'
        log_disconnections: 'on'
        log_line_prefix: '%t [%p]: [%l-1] %c %x %d %u %a %h '
        log_lock_waits: 'on'
        log_min_duration_statement: 500
        log_statement: ddl
        log_temp_files: 0
        max_connections: '400'
        max_replication_slots: 10
        max_wal_senders: 10
        ssl_ciphers: HIGH:MEDIUM:-3DES:!aNULL
        synchronous_standby_names: '*'
        tcp_keepalives_count: '2'
        tcp_keepalives_idle: '10'
        tcp_keepalives_interval: '5'
        track_functions: all
        wal_compression: 'on'
        wal_keep_segments: '100'
        wal_keep_size: 2GB
        wal_level: hot_standby
        wal_log_hints: 'on'
      use_pg_rewind: true
      use_slots: true
    retry_timeout: 7
    slots:
      robin_patroni_0:
        type: physical
      robin_patroni_1:
        type: physical
      robin_patroni_2:
        type: physical
    synchronous_mode: true
    synchronous_mode_strict: true
    ttl: 20
  initdb:
  - auth-host: scram-sha-256
  - auth-local: trust
  - data-checksums
  - encoding: UTF8
  - locale: en_US.UTF-8
  post_init: /scripts/post_init.sh "zalandos"
kubernetes:
  bootstrap_labels:
    critical-operation: 'true'
  bypass_api_service: true
  labels:
    application: spilo
  leader_label_value: master
  pod_ip: 192.168.0.112
  port: tcp://10.96.0.1:443
  port_443_tcp: tcp://10.96.0.1:443
  port_443_tcp_addr: 10.96.0.1
  port_443_tcp_port: '443'
  port_443_tcp_proto: tcp
  ports:
  - name: postgresql
    port: 5432
  role_label: spilo-role
  scope_label: cluster-name
  service_host: 10.96.0.1
  service_port: '443'
  service_port_https: '443'
  standby_leader_label_value: master
  use_endpoints: true
namespace: robinio
postgresql:
  authentication:
    replication:
      password: <redacted>
      username: replicator
    superuser:
      password: <redacted>
      username: postgres
  basebackup_fast_xlog:
    command: /scripts/basebackup.sh
    retries: 2
  bin_dir: /usr/lib/postgresql/15/bin
  callbacks:
    on_role_change: /usr/bin/python3 /home/postgres/pgdata/pgroot/callback/rename_partial_wal.py
  connect_address: 192.168.0.112:5432
  create_replica_method:
  - basebackup_fast_xlog
  data_dir: /home/postgres/pgdata/pgroot/data
  listen: '*:5432'
  name: robin-patroni-0
  parameters:
    archive_command: /bin/true
    bg_mon.history_buckets: 120
    bg_mon.listen_address: '::'
    extwlist.custom_path: /scripts
    extwlist.extensions: btree_gin,btree_gist,citext,extra_window_functions,first_last_agg,hll,hstore,hypopg,intarray,ltree,pgcrypto,pgq,pgq_node,pg_trgm,postgres_fdw,roaringbitmap,tablefunc,uuid-ossp,vector,timescaledb,pg_partman
    log_destination: csvlog
    log_directory: ../pg_log
    log_file_mode: '0644'
    log_filename: postgresql-%u.log
    log_rotation_age: 1d
    log_truncate_on_rotation: 'on'
    logging_collector: 'on'
    output_plugin_libraries: pgoutput, test_decoding, pglogical_output, wal2json, decoderbufs
    pg_stat_statements.track_utility: 'off'
    shared_buffers: 1024MB
    shared_preload_libraries: bg_mon,pg_stat_statements,pgextwlist,pg_auth_mon,set_user,timescaledb,pg_cron,pg_stat_kcache
    ssl: 'on'
    ssl_cert_file: /run/certs/server.crt
    ssl_key_file: /run/certs/server.key
  pg_hba:
  - local   all             all                                   trust
  - hostssl all             +zalandos    127.0.0.1/32       pam
  - host    all             all                127.0.0.1/32       md5
  - hostssl all             +zalandos    ::1/128            pam
  - host    all             all                ::1/128            md5
  - local   replication     replicator                    trust
  - hostssl replication     replicator all                md5
  - hostnossl all           all                all                reject
  - hostssl all             +zalandos    all                pam
  - hostssl all             all                all                md5
  pgpass: /run/postgresql/pgpass
  use_unix_socket: true
  use_unix_socket_repl: true
restapi:
  connect_address: 192.168.0.112:8008
  listen: :8008
scope: robin-patroni
```

### patronictl show-config

```yaml
check_timeline: true
failsafe_mode: true
loop_wait: 5
maximum_lag_on_failover: 1048576
postgresql:
  callbacks:
    on_role_change: /home/postgres/pgdata/pgroot/callback/rename_partial_wal.py
  parameters:
    archive_mode: 'off'
    archive_timeout: '0'
    autovacuum_analyze_scale_factor: 0.02
    autovacuum_max_workers: 5
    autovacuum_vacuum_scale_factor: 0.05
    checkpoint_completion_target: 0.9
    hot_standby: 'on'
    huge_pages: 'off'
    log_autovacuum_min_duration: 0
    log_checkpoints: 'on'
    log_connections: 'on'
    log_disconnections: 'on'
    log_line_prefix: '%t [%p]: [%l-1] %c %x %d %u %a %h '
    log_lock_waits: 'on'
    log_min_duration_statement: 500
    log_statement: ddl
    log_temp_files: 0
    max_connections: '400'
    max_replication_slots: 10
    max_wal_senders: 10
    ssl_ciphers: HIGH:MEDIUM:-3DES:!aNULL
    synchronous_standby_names: '*'
    tcp_keepalives_count: '2'
    tcp_keepalives_idle: '10'
    tcp_keepalives_interval: '5'
    track_functions: all
    wal_compression: 'on'
    wal_keep_segments: '100'
    wal_keep_size: 2GB
    wal_level: hot_standby
    wal_log_hints: 'on'
  remove_data_directory_on_diverged_timelines: true
  remove_data_directory_on_rewind_failure: true
  use_pg_rewind: true
  use_slots: true
retry_timeout: 7
slots:
  robin_patroni_0:
    type: physical
  robin_patroni_1:
    type: physical
  robin_patroni_2:
    type: physical
synchronous_mode: true
synchronous_mode_strict: true
ttl: 20
```

### Patroni log files

```shell
Old leader `robin-patroni-0` loses the DCS and demotes (03:59:29), then `robin-patroni-1` promotes (03:59:37) and later loses the DCS too (04:00:20):


robin-patroni-0: 2026-11-18 03:59:29,578 INFO: demoting self because DCS is not accessible and I was a leader
robin-patroni-0: 2026-11-18 03:59:32,518 INFO: demoted self because DCS is not accessible and I was a leader
robin-patroni-1: 2026-11-18 03:59:37,172 INFO: promoted self to leader by acquiring session lock
robin-patroni-1: 2026-11-18 03:59:40,315 INFO: Synchronous standby status assigned to ['robin-patroni-2']
robin-patroni-1: 2026-11-18 04:00:20,431 INFO: demoting self because DCS is not accessible and I was a leader


The stuck state, one HA loop per member (this repeats every 5 s):


robin-patroni-1: 2026-11-18 04:03:20,816 INFO: Got response from robin-patroni-0 http://192.168.0.56:8008/patroni: {"state": "running", ..., "role": "replica", ...}
robin-patroni-1: 2026-11-18 04:03:20,817 INFO: Wal position of robin-patroni-0 is ahead of my wal position
robin-patroni-1: 2026-11-18 04:03:20,917 INFO: following a different leader because i am not the healthiest node

robin-patroni-2: 2026-11-18 04:03:23,919 INFO: Got response from robin-patroni-0 http://192.168.0.56:8008/patroni: {"state": "running", ..., "role": "replica", ...}
robin-patroni-2: 2026-11-18 04:03:23,938 INFO: Wal position of robin-patroni-0 is ahead of my wal position
robin-patroni-2: 2026-11-18 04:03:23,941 INFO: following a different leader because i am not the healthiest node

robin-patroni-0: 2026-11-18 04:03:23,336 INFO: My timeline 3 is behind last known cluster timeline 4
robin-patroni-0: 2026-11-18 04:03:23,340 INFO: following a different leader because i am not the healthiest node


(`robin-patroni-2` also logs `Wal position of robin-patroni-1 is ahead of my wal position` in some loops, because `robin-patroni-1`, the previous leader on timeline 4, reports a slightly higher LSN. Those two members are on the same timeline, so that is a legitimate tie-break; what blocks the cluster is `robin-patroni-0`.)

Counts in the attached logs: "Wal position of robin-patroni-0 is ahead of my wal position" 36 times on `robin-patroni-1` and 28 times on `robin-patroni-2`; "My timeline 3 is behind last known cluster timeline 4" 39 times on `robin-patroni-0`; "exceeds maximum replication lag" 0 times. Full logs are attached: `patroni-log-robin-patroni-0.log`, `patroni-log-robin-patroni-1.log`, `patroni-log-robin-patroni-2.log`.

DCS state while stuck: the leader endpoint still names `robin-patroni-1` with `renewTime` 2026-11-18T04:00:05 (stale), and `robin-patroni-sync` does not exist:


$ kubectl -n robinio get endpoints robin-patroni-sync
Error from server (NotFound): endpoints "robin-patroni-sync" not found


(`lease_and_dcs_watch.log` shows the `kube-controller-manager` lease change at 04:01:52 and the sync Endpoints object disappearing at 04:01:54.)
```

### PostgreSQL log files

```shell
Postgres log excerpts (csvlog, UTC), full files attached as `postgresql-log-robin-patroni-{0,1,2}.csv` (03:59 to 04:09):


robin-patroni-0 (old leader)
03:59:29.597  received fast shutdown request
03:59:29.892  database system is shut down
03:59:31.519  entering standby mode
03:59:31.522  waiting for WAL to become available at 0/7002000     <- it has WAL beyond the point where timeline 4 forked

robin-patroni-1 (new leader)
03:59:37.185  received promote request
03:59:37.188  selected new timeline ID: 4
03:59:37.536  standby "robin-patroni-2" is now a synchronous standby with priority 1
04:00:20.452  received fast shutdown request
04:00:22.227  entering standby mode
04:00:22.229  waiting for WAL to become available at 0/6002000

robin-patroni-2
03:59:35.157  WAL receiver process shutdown requested
03:59:37.205  started streaming WAL from primary at 0/5000000 on timeline 3
```

### Have you tried to use GitHub issue search?

- [x] Yes

### Anything else we need to know?

- **Where in the code (4.1.5, `patroni/ha.py`):** `_is_healthiest_node()` checks the local timeline at lines 1313-1319, then loops over peers at lines 1348-1359 and returns False as soon as `my_wal_position < st.wal_position`. The only exemption is "the former leader being ahead of us" and only when `sync_mode_is_active()` and the peer is the record's leader (line 1358). `is_failover_possible()` already uses peer timelines (line 1414).
- **Why the synchronous record matters:** with `synchronous_mode` and a non-empty record, only the record's leader and sync standbys take part in the election, which hid this in practice. With the record missing (here because the Endpoints object was deleted), all members race and the peer comparison is the only protection. The election logic is independent of how the record became empty.
- **Kubernetes Endpoints trigger:** only `<scope>-config` gets a headless Service (`_create_config_service`, with a comment that it would otherwise be "cleaned" by the k8s master). `<scope>-sync` has no Service. On a controller-manager lease change, the endpoints controller deletes Service-less Endpoints objects. We confirmed it by watching the object while failing the lease over with no leader running: it was deleted within about 2 seconds and not recreated (`lease_and_dcs_watch.log`). Only the leader writes the sync object (`process_sync_replication()` and the promote path in `ha.py`).
- **Production occurrence and what deleted the Endpoints object:** the audit log shows `endpoints.delete robin-patroni-sync` by `system:serviceaccount:kube-system:endpoint-controller` at 08:57:35 and 08:59:37 (UTC, 2026-11-14), 0.4 s and 2.2 s after a new `kube-controller-manager` lease holder started. The first delete was repaired within 2 s by the member that had just promoted (it created the object again); the second happened with no leader and the object stayed missing until 11:26:54, after a manual restart. The lease changed holders 8 times between 08:57:35 and 09:09:02 (transitions 113 to 120), while the nodes had lost the API server. Details: `production_endpoint_and_lease_events.txt`. We can provide the members' logs from that occurrence on request.
- **Proposed fix:** in the peer loop of `_is_healthiest_node()`, when `check_timeline` is on, skip a peer whose timeline is behind the cluster timeline (same condition as in `is_failover_possible()`), so a peer that cannot be promoted never makes a valid candidate defer. We have not tested a patch yet and are happy to open a PR.
- **Workaround:** restart the member that is ahead on LSN (here `robin-patroni-2`). Once it no longer answers as an "ahead" replica, the other members promote at their next loop. In the production occurrence, deleting the pod of that member made the member on the newest timeline promote about 0.2 s after that member's Postgres stopped (not tested separately in this reproduction).
- Reproduced without any manual change to the DCS and without pod restarts. The script and the full logs are attached.


view thread (4+ messages)  latest in thread

Message-ID: <gh-patroni-patroni-3802@github.com>
Permalink:  ../gh-patroni-patroni-3802@github.com/

 · 


This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox