pg.ddx.io patroni/patroni GitHub issues and pull requests
help / color / mirror / Atom feed[patroni/patroni] PR #3779: Disable the watchdog after a failed pre_promote
8+ messages / 3 participants
[nested] [flat]
* [patroni/patroni] PR #3779: Disable the watchdog after a failed pre_promote
@ 2026-11-13 14:45 "egor-voynov-aiven (@egor-voynov-aiven)" <noreply+egor-voynov-aiven@github.com>
0 siblings, 0 replies; 8+ messages in thread
From: egor-voynov-aiven (@egor-voynov-aiven) @ 2026-11-13 14:45 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
When a replica wins the leader race, `enforce_primary_role()` activates the watchdog before the promotion starts, and `pre_promote` runs as the first step of the promotion. If the script exits non-zero, the node releases the leader key and skips the leader race for ttl seconds, but the watchdog stays armed. Nothing feeds it anymore, because keepalive() is only called after the leader key is renewed. That is earlier than the end of the backoff, so the retry the code prepares for can never happen.
This PR disables the watchdog in that branch when Postgres is not running as primary. The node stays up as a replica, honours the backoff, and activates the watchdog again if it wins the race later. A node that is already running as primary keeps its watchdog as before.
inline patch 1: patroni/ha.py, tests/test_ha.py (+21/-3 lines) download inline-1.patch
diff --git a/patroni/ha.py b/patroni/ha.py
index a853b7ce1..b56d61804 100644
--- a/patroni/ha.py
+++ b/patroni/ha.py
@@ -1218,6 +1218,10 @@ def enforce_primary_role(self, message: str, promote_message: str) -> str:
logger.warning("Releasing the leader key voluntarily because the pre-promote script failed")
self._released_leader_key_timestamp = time.monotonic()
self.release_leader_key_voluntarily()
+ if not self.state_handler.is_primary():
+ # the leader key is gone and nobody will feed the watchdog anymore,
+ # disarm it to avoid a reset while the node is in the backoff as a replica
+ self.watchdog.disable()
# discard the result of the failed pre-promote script to be able to re-try promote
self._async_response.reset()
return 'Promotion cancelled because the pre-promote script failed'
diff --git a/tests/test_ha.py b/tests/test_ha.py
index 47ded458d..fce96f504 100644
--- a/tests/test_ha.py
+++ b/tests/test_ha.py
@@ -425,9 +425,23 @@ def test_promotion_cancelled_after_pre_promote_failed(self):
self.p.is_primary = false
self.p._pre_promote = false
self.ha._is_healthiest_node = true
- self.assertEqual(self.ha.run_cycle(), 'promoted self to leader by acquiring session lock')
- self.assertEqual(self.ha.run_cycle(), 'Promotion cancelled because the pre-promote script failed')
- self.assertEqual(self.ha.run_cycle(), 'following a different leader because i am not the healthiest node')
+ with patch.object(Watchdog, 'disable') as mock_disable:
+ self.assertEqual(self.ha.run_cycle(), 'promoted self to leader by acquiring session lock')
+ mock_disable.assert_not_called()
+ self.assertEqual(self.ha.run_cycle(), 'Promotion cancelled because the pre-promote script failed')
+ # the leader key is released, nobody feeds the watchdog anymore
+ mock_disable.assert_called_once()
+ self.assertEqual(self.ha.run_cycle(), 'following a different leader because i am not the healthiest node')
+
+ def test_pre_promote_failed_keeps_watchdog_on_primary(self):
+ self.p.is_primary = false
+ self.p._pre_promote = false
+ self.ha._is_healthiest_node = true
+ with patch.object(Watchdog, 'disable') as mock_disable:
+ self.assertEqual(self.ha.run_cycle(), 'promoted self to leader by acquiring session lock')
+ self.p.is_primary = true
+ self.assertEqual(self.ha.run_cycle(), 'Promotion cancelled because the pre-promote script failed')
+ mock_disable.assert_not_called()
def test_lost_leader_lock_during_promote(self):
with patch('patroni.async_executor.AsyncExecutor.busy', PropertyMock(return_value=True)):
^ permalink raw [nested|flat] 8+ messages in thread
* Re: [patroni/patroni] PR #3779: Disable the watchdog after a failed pre_promote
@ 2026-11-13 14:45 ` "egor-voynov-aiven (@egor-voynov-aiven)" <noreply+egor-voynov-aiven@github.com>
6 siblings, 0 replies; 8+ messages in thread
From: egor-voynov-aiven (@egor-voynov-aiven) @ 2026-11-13 14:45 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
@egor-voynov-aiven requested a review.
^ permalink raw [nested|flat] 8+ messages in thread
* Re: [patroni/patroni] PR #3779: Disable the watchdog after a failed pre_promote
@ 2026-11-13 14:45 ` "egor-voynov-aiven (@egor-voynov-aiven)" <noreply+egor-voynov-aiven@github.com>
6 siblings, 0 replies; 8+ messages in thread
From: egor-voynov-aiven (@egor-voynov-aiven) @ 2026-11-13 14:45 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
@egor-voynov-aiven requested a review.
^ permalink raw [nested|flat] 8+ messages in thread
* Re: [patroni/patroni] PR #3779: Disable the watchdog after a failed pre_promote
@ 2026-11-19 20:03 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
6 siblings, 0 replies; 8+ messages in thread
From: hughcapet (@hughcapet) @ 2026-11-19 20:03 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
@hughcapet applied label "bug".
^ permalink raw [nested|flat] 8+ messages in thread
* Re: [patroni/patroni] PR #3779: Disable the watchdog after a failed pre_promote
@ 2026-11-19 20:27 ` "codecov[bot] (bot) (@codecov[bot])" <noreply+codecov[bot]@github.com>
6 siblings, 0 replies; 8+ messages in thread
From: codecov[bot] (bot) (@codecov[bot]) @ 2026-11-19 20:27 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
## [Codecov](https://app.codecov.io/gh/patroni/patroni/pull/3779?dropdown=coverage&src=pr&el=h1&utm_m...) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:loudspeaker: Thoughts on this report? [Let us know!](https://github.com/codecov/feedback/issues/255)
^ permalink raw [nested|flat] 8+ messages in thread
* Re: [patroni/patroni] PR #3779 (approved): Disable the watchdog after a failed pre_promote
@ 2026-11-20 07:56 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
6 siblings, 0 replies; 8+ messages in thread
From: hughcapet (@hughcapet) @ 2026-11-20 07:56 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
^ permalink raw [nested|flat] 8+ messages in thread
* Re: [patroni/patroni] PR #3779: Disable the watchdog after a failed pre_promote
@ 2026-11-20 07:56 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
6 siblings, 0 replies; 8+ messages in thread
From: hughcapet (@hughcapet) @ 2026-11-20 07:56 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
@hughcapet merged this (commit 4fe0275e34fb).
^ permalink raw [nested|flat] 8+ messages in thread
* Re: [patroni/patroni] PR #3779: Disable the watchdog after a failed pre_promote
@ 2026-11-20 07:56 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
6 siblings, 0 replies; 8+ messages in thread
From: hughcapet (@hughcapet) @ 2026-11-20 07:56 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
@hughcapet closed this.
^ permalink raw [nested|flat] 8+ messages in thread
end of thread, other threads:[~2026-11-20 07:56 UTC | newest]
Thread overview: 8+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-11-13 14:45 [patroni/patroni] PR #3779: Disable the watchdog after a failed pre_promote "egor-voynov-aiven (@egor-voynov-aiven)" <noreply+egor-voynov-aiven@github.com>
2026-11-13 14:45 ` "egor-voynov-aiven (@egor-voynov-aiven)" <noreply+egor-voynov-aiven@github.com>
2026-11-13 14:45 ` "egor-voynov-aiven (@egor-voynov-aiven)" <noreply+egor-voynov-aiven@github.com>
2026-11-19 20:03 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
2026-11-19 20:27 ` "codecov[bot] (bot) (@codecov[bot])" <noreply+codecov[bot]@github.com>
2026-11-20 07:56 ` Re: [patroni/patroni] PR #3779 (approved): Disable the watchdog after a failed pre_promote "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
2026-11-20 07:56 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
2026-11-20 07:56 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox