pg.ddx.io patroni/patroni GitHub issues and pull requests
help / color / mirror / Atom feed[patroni/patroni] PR #3803: Ignore peers on an older timeline when choosing the healthiest node
15+ messages / 4 participants
[nested] [flat]
* [patroni/patroni] PR #3803: Ignore peers on an older timeline when choosing the healthiest node
@ 2026-11-18 06:41 "sgdanawe (@sgdanawe)" <noreply+sgdanawe@github.com>
0 siblings, 0 replies; 15+ messages in thread
From: sgdanawe (@sgdanawe) @ 2026-11-18 06:41 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
When check_timeline is enabled, a member whose timeline is behind the cluster timeline disqualifies itself and never promotes. The peer loop in _is_healthiest_node() nevertheless compared only the WAL position: a member that was ahead on LSN, because of the WAL it had written on an abandoned timeline, made every other member give up its own candidacy. If the members that could be promoted were all deferring to such a member, nobody was promoted and the cluster stayed without a leader.
Skip such a peer in the comparison, the same way is_failover_possible() already does. A peer on the cluster timeline that is ahead still wins, a peer with an unknown timeline is treated as before, and nothing changes when check_timeline is disabled.
Fixes #3802
inline patch 1: patroni/ha.py, tests/test_ha.py (+33/-1 lines) download inline-1.patch
diff --git a/patroni/ha.py b/patroni/ha.py
index a853b7ce1..86fc0944b 100644
--- a/patroni/ha.py
+++ b/patroni/ha.py
@@ -1401,7 +1401,8 @@ def _is_healthiest_node(self, members: Collection[Member],
logger.info('My wal position exceeds maximum replication lag')
return False # Too far behind last reported wal position on primary
- if not self.is_standby_cluster() and self.check_timeline():
+ check_timeline = not self.is_standby_cluster() and self.check_timeline()
+ if check_timeline:
if my_timeline is None:
logger.info('Can not figure out my timeline')
return False
@@ -1444,6 +1445,13 @@ def _is_healthiest_node(self, members: Collection[Member],
logger.warning('Primary (%s) is still alive', st.member.name)
return False
if my_wal_position < st.wal_position:
+ # With check_timeline enabled a node with the timeline behind the cluster timeline disqualifies
+ # itself, therefore it will never promote and it must not make us give up our own candidacy.
+ # Its WAL position is higher only because of the WAL it generated on the abandoned timeline.
+ if check_timeline and st.timeline and st.timeline < cluster_timeline:
+ logger.info('Ignoring %s being ahead of us: its timeline %s is behind last known '
+ 'cluster timeline %s', st.member.name, st.timeline, cluster_timeline)
+ continue
nodes_ahead += 1
logger.info('Wal position of %s is ahead of my wal position', st.member.name)
# In synchronous mode the former leader might be still accessible and even be ahead of us.
diff --git a/tests/test_ha.py b/tests/test_ha.py
index 47ded458d..93725c94c 100644
--- a/tests/test_ha.py
+++ b/tests/test_ha.py
@@ -1308,6 +1308,30 @@ def test__is_healthiest_node(self):
self.ha.cluster = get_cluster_initialized_without_leader(failover=Failover(0, 'leader', None, None, 'dc1'))
self.assertTrue(self.ha._is_healthiest_node(self.ha.old_cluster.members))
+ def test__is_healthiest_node_peer_on_older_timeline(self):
+ self.p.is_primary = false
+ self.ha.cluster = get_cluster_initialized_without_leader()
+ global_config.update(self.ha.cluster)
+ members = self.ha.old_cluster.members
+ # the peer is ahead on LSN, but it is on a timeline behind the cluster timeline
+ with patch.object(Cluster, 'timeline', PropertyMock(return_value=3)), \
+ patch('patroni.postgresql.Postgresql.replica_cached_timeline', return_value=3):
+ # the peer can't be promoted (check_timeline), so it must not make us defer
+ self.ha.fetch_node_status = get_node_status(wal_position=11, timeline=2)
+ with patch('patroni.ha.logger.info') as mock_info:
+ self.assertTrue(self.ha._is_healthiest_node(members))
+ self.assertIn('Ignoring %s being ahead of us', mock_info.call_args_list[0][0][0])
+ # the peer is on the cluster timeline and ahead: defer to it
+ self.ha.fetch_node_status = get_node_status(wal_position=11, timeline=3)
+ self.assertFalse(self.ha._is_healthiest_node(members))
+ # unknown timeline of the peer: keep the old behaviour
+ self.ha.fetch_node_status = get_node_status(wal_position=11, timeline=0)
+ self.assertFalse(self.ha._is_healthiest_node(members))
+ # check_timeline is disabled: timelines are not taken into account
+ self.ha.fetch_node_status = get_node_status(wal_position=11, timeline=2)
+ with patch.object(Ha, 'check_timeline', Mock(return_value=False)):
+ self.assertFalse(self.ha._is_healthiest_node(members))
+
def test_fetch_node_status(self):
member = Member(0, 'test', 1, {'api_url': 'http://127.0.0.1:8011/patroni';})
self.ha.fetch_node_status(member)
^ permalink raw [nested|flat] 15+ messages in thread
* Re: [patroni/patroni] PR #3803: Ignore peers on an older timeline when choosing the healthiest node
@ 2026-11-18 06:41 "sgdanawe (@sgdanawe)" <noreply+sgdanawe@github.com>
13 siblings, 0 replies; 15+ messages in thread
From: sgdanawe (@sgdanawe) @ 2026-11-18 06:41 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
@sgdanawe requested a review.
^ permalink raw [nested|flat] 15+ messages in thread
* Re: [patroni/patroni] PR #3803: Ignore peers on an older timeline when choosing the healthiest node
@ 2026-11-18 06:41 "sgdanawe (@sgdanawe)" <noreply+sgdanawe@github.com>
13 siblings, 0 replies; 15+ messages in thread
From: sgdanawe (@sgdanawe) @ 2026-11-18 06:41 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
@sgdanawe requested a review.
^ permalink raw [nested|flat] 15+ messages in thread
* Re: [patroni/patroni] PR #3803: Ignore peers on an older timeline when choosing the healthiest node
@ 2026-11-18 08:54 "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
13 siblings, 0 replies; 15+ messages in thread
From: CyberDem0n (@CyberDem0n) @ 2026-11-18 08:54 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
Thanks for the analysis and the patch! The direction is right. Putting the check inside the "ahead" branch, after the "Primary is still alive" check, is the correct place: an old-timeline voter that isn't ahead of us still counts toward the quorum.
Two gaps remain:
1. **Equal LSN still deadlocks.** An old-timeline peer at the same LSN still lands in `eligible_members`. The `failover_priority` check and the multi-site `current_site`/`failover.site` checks then make us defer to a node that will never promote. It needs an exactly equal LSN, so it's less likely, but the outcome is the same. For example, a TL1 peer at my LSN with `failover_priority: 2` makes `_is_healthiest_node()` return `False` on every node.
2. **An unknown timeline should mean "can't promote".** A node that can't determine its own timeline disqualifies itself ("Can not figure out my timeline"). The REST API reports `timeline: null` exactly when that same `IDENTIFY_SYSTEM` call fails, so deferring to such a peer is the same deadlock. `is_failover_possible()` already treats it as not promotable (`not st.timeline or st.timeline < cluster_timeline`).
Suggested shape:
```python
timeline_ok = not check_timeline or bool(st.timeline) and st.timeline >= cluster_timeline
if my_wal_position < st.wal_position:
if not timeline_ok:
logger.info('Ignoring %s being ahead of us: its timeline %s is behind '
'last known cluster timeline %s', st.member.name, st.timeline, cluster_timeline)
continue
...
elif st.wal_position > 0:
...
if my_wal_position == st.wal_position and timeline_ok:
eligible_members.append(st)
```
Please also add tests for:
- equal LSN with a higher `failover_priority`;
- the multi-site case;
- a running primary on an old timeline still blocking;
- in quorum mode, an old-timeline voter that is behind still giving its vote. This guards against the check being moved before the LSN comparison.
^ permalink raw [nested|flat] 15+ messages in thread
* Re: [patroni/patroni] PR #3803: Ignore peers on an older timeline when choosing the healthiest node
@ 2026-11-18 10:11 "sgdanawe (@sgdanawe)" <noreply+sgdanawe@github.com>
13 siblings, 0 replies; 15+ messages in thread
From: sgdanawe (@sgdanawe) @ 2026-11-18 10:11 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
Thanks for the review. Pushed an update:
1. Equal LSN: whether a peer can promote is now decided once, before the LSN comparison. An old-timeline peer at our LSN is no longer added to eligible_members, so it can't make us defer through failover_priority or the site checks.
2. Unknown timeline: a peer with no timeline is treated as unable to promote, the same as in is_failover_possible() and the local "Can not figure out my timeline" check.
I used the shape you suggested. The "Primary is still alive" check still runs first, and the timeline is only used to skip a peer that is ahead or to exclude it from eligible_members. A peer that is behind us still gives its quorum vote.
Tests added or updated:
- equal LSN with a higher failover_priority
- the multi-site cases (current site and requested site)
- a running primary on an old timeline still blocking
- an old-timeline voter that is behind still giving its quorum vote
The old assertion that an unknown-timeline peer ahead of us blocks is now flipped, per point 2. tests/test_ha.py passes.
^ permalink raw [nested|flat] 15+ messages in thread
* Re: [patroni/patroni] PR #3803: Ignore peers on an older timeline when choosing the healthiest node
@ 2026-11-18 10:49 "codecov[bot] (bot) (@codecov[bot])" <noreply+codecov[bot]@github.com>
13 siblings, 0 replies; 15+ messages in thread
From: codecov[bot] (bot) (@codecov[bot]) @ 2026-11-18 10:49 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
## [Codecov](https://app.codecov.io/gh/patroni/patroni/pull/3803?dropdown=coverage&src=pr&el=h1&utm_m...) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:loudspeaker: Thoughts on this report? [Let us know!](https://github.com/codecov/feedback/issues/255)
^ permalink raw [nested|flat] 15+ messages in thread
* Re: [patroni/patroni] PR #3803: Ignore peers on an older timeline when choosing the healthiest node
@ 2026-11-18 13:24 "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
13 siblings, 0 replies; 15+ messages in thread
From: CyberDem0n (@CyberDem0n) @ 2026-11-18 13:24 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
(on patroni/ha.py:1423)
```suggestion
check_timeline = not self.is_standby_cluster() and self.check_timeline()
switchpoints = {}
if check_timeline and self.cluster.history.lines:
try:
switchpoints = {int(line[0]): int(line[1]) for line in self.cluster.history.lines}
except Exception:
pass
```
^ permalink raw [nested|flat] 15+ messages in thread
* Re: [patroni/patroni] PR #3803: Ignore peers on an older timeline when choosing the healthiest node
@ 2026-11-18 13:27 "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
13 siblings, 0 replies; 15+ messages in thread
From: CyberDem0n (@CyberDem0n) @ 2026-11-18 13:27 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
(on patroni/ha.py)
On a replica, Postgres reports the timeline it has **replayed,** while Patroni compares the **WAL position it has received**. A replica that has received WAL on the new timeline but not yet replayed past the switch point therefore looks like "old timeline, ahead".
- Before this PR: the other nodes waited until it caught up, then it promoted with no data loss.
- Now: they skip it, so a **node with less WAL may promote**.
I think it should be fixed:
```suggestion
# A member on an older timeline that replayed WAL beyond the point where its timeline ended has
# diverged from the cluster history. Otherwise it might still be catching up and we wait for it.
# If the switchpoint is unknown we compare with our position, which is beyond it, because we
# passed the timeline check.
if not timeline_ok and (not st.timeline
or st.replay_lsn > switchpoints.get(st.timeline, my_wal_position)):
```
That would require adding a new property to `_MemberStatus`:
```python
@property
def replay_lsn(self) -> int:
"""Value of ``replayed_location`` from JSON, ``0`` if unknown."""
wal: Dict[str, Any] = self.data.get('wal') or self.data.get('xlog') or {}
return wal.get('replayed_location') or 0
```
^ permalink raw [nested|flat] 15+ messages in thread
* Re: [patroni/patroni] PR #3803: Ignore peers on an older timeline when choosing the healthiest node
@ 2026-11-18 14:58 "sgdanawe (@sgdanawe)" <noreply+sgdanawe@github.com>
13 siblings, 0 replies; 15+ messages in thread
From: sgdanawe (@sgdanawe) @ 2026-11-18 14:58 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
Thanks, that is a good catch. Pushed an update.
You are right that the REST API reports the timeline a replica has replayed, while we compare the WAL it has received, so a replica that is still catching up looks like "old timeline, ahead". A peer on an older timeline (or with an unknown timeline) is now ignored only if it has replayed beyond the point where its own timeline ended:
- `_MemberStatus.replay_lsn` reads `replayed_location` from the peer's response.
- The switchpoints come from the cluster history, as you suggested. If the switchpoint is unknown we compare with our own position.
- The same condition is used for the equal-LSN case, so an old-timeline peer that is still catching up stays in `eligible_members`, and one that has diverged is left out.
Tests: the existing cases are updated to set `replay_lsn`, and new ones cover a peer that is still catching up (not skipped), a diverged peer (skipped, in the ahead and equal-LSN cases), a switchpoint that is missing or can't be parsed, and the quorum vote from a behind peer. `tests/test_ha.py` passes.
^ permalink raw [nested|flat] 15+ messages in thread
* Re: [patroni/patroni] PR #3803: Ignore peers on an older timeline when choosing the healthiest node
@ 2026-11-19 10:31 "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
13 siblings, 0 replies; 15+ messages in thread
From: hughcapet (@hughcapet) @ 2026-11-19 10:31 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
@hughcapet applied label "bug".
^ permalink raw [nested|flat] 15+ messages in thread
* Re: [patroni/patroni] PR #3803: Ignore peers on an older timeline when choosing the healthiest node
@ 2026-11-19 13:48 "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
13 siblings, 0 replies; 15+ messages in thread
From: CyberDem0n (@CyberDem0n) @ 2026-11-19 13:48 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
(on patroni/ha.py:1489)
```suggestion
# it can't have acknowledged anything past the switchpoint, so its quorum vote is valid
if st.timeline in switchpoints and st.member.name in voting_set:
logger.info('Got quorum vote from %s', st.member.name)
quorum_votes += 1
continue
```
^ permalink raw [nested|flat] 15+ messages in thread
* Re: [patroni/patroni] PR #3803: Ignore peers on an older timeline when choosing the healthiest node
@ 2026-11-19 13:51 "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
13 siblings, 0 replies; 15+ messages in thread
From: CyberDem0n (@CyberDem0n) @ 2026-11-19 13:51 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
(on tests/test_ha.py)
```suggestion
# diverged voter ahead of us still votes, but only if the divergence is proven by a known switchpoint
self.ha.fetch_node_status = get_node_status(wal_position=11, timeline=2, replay_lsn=9)
self.assertTrue(self.ha._is_healthiest_node(members))
self.ha.fetch_node_status = get_node_status(wal_position=11, timeline=2, replay_lsn=11)
with patch.object(Cluster, 'history', PropertyMock(return_value=None)):
self.assertFalse(self.ha._is_healthiest_node(members))
```
^ permalink raw [nested|flat] 15+ messages in thread
* Re: [patroni/patroni] PR #3803 (approved): Ignore peers on an older timeline when choosing the healthiest node
@ 2026-11-19 17:15 "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
13 siblings, 0 replies; 15+ messages in thread
From: hughcapet (@hughcapet) @ 2026-11-19 17:15 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
^ permalink raw [nested|flat] 15+ messages in thread
* Re: [patroni/patroni] PR #3803: Ignore peers on an older timeline when choosing the healthiest node
@ 2026-11-19 17:16 "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
13 siblings, 0 replies; 15+ messages in thread
From: hughcapet (@hughcapet) @ 2026-11-19 17:16 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
@hughcapet closed this.
^ permalink raw [nested|flat] 15+ messages in thread
* Re: [patroni/patroni] PR #3803: Ignore peers on an older timeline when choosing the healthiest node
@ 2026-11-19 17:16 "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
13 siblings, 0 replies; 15+ messages in thread
From: hughcapet (@hughcapet) @ 2026-11-19 17:16 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
@hughcapet merged this (commit 522f9914806f).
^ permalink raw [nested|flat] 15+ messages in thread
end of thread, other threads:[~2026-11-19 17:16 UTC | newest]
Thread overview: 15+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-11-18 06:41 [patroni/patroni] PR #3803: Ignore peers on an older timeline when choosing the healthiest node "sgdanawe (@sgdanawe)" <noreply+sgdanawe@github.com>
2026-11-18 06:41 ` "sgdanawe (@sgdanawe)" <noreply+sgdanawe@github.com>
2026-11-18 06:41 ` "sgdanawe (@sgdanawe)" <noreply+sgdanawe@github.com>
2026-11-18 08:54 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-18 10:11 ` "sgdanawe (@sgdanawe)" <noreply+sgdanawe@github.com>
2026-11-18 10:49 ` "codecov[bot] (bot) (@codecov[bot])" <noreply+codecov[bot]@github.com>
2026-11-18 13:24 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-18 13:27 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-18 14:58 ` "sgdanawe (@sgdanawe)" <noreply+sgdanawe@github.com>
2026-11-19 10:31 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
2026-11-19 13:48 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-19 13:51 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-19 17:15 ` Re: [patroni/patroni] PR #3803 (approved): Ignore peers on an older timeline when choosing the healthiest node "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
2026-11-19 17:16 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
2026-11-19 17:16 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox