pg.ddx.io  patroni/patroni GitHub issues and pull requests  
help / color / mirror / Atom feed
[patroni/patroni] PR #3794: Retry watchdog activation after configuration changes
7+ messages / 3 participants
[nested] [flat]

* [patroni/patroni] PR #3794: Retry watchdog activation after configuration changes
@ 2026-11-17 18:31  "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
  0 siblings, 0 replies; 7+ messages in thread

From: CyberDem0n (@CyberDem0n) @ 2026-11-17 18:31 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

When activation falls back to `NullWatchdog`, the configured device must be acquired again once a configuration change makes it usable. Only `keepalive()` did that, and only after `mode` was changed from `off`. But `keepalive()` is only called by the leader, and `reload_config()` of an inactive watchdog reacted only to `driver` changes. Therefore:

- If the watchdog was activated while `ttl` was less than 2x `loop_wait`, the device was never reopened after the timing was fixed. Instead, every keepalive logged `Error while sending keepalive: Setting timeout is not supported on NullWatchdog`.
- Enabling the watchdog with a reload on a replica or on a paused primary had no effect until restart. In `required` mode the node reported `watchdog_failed` and couldn't become the leader. A primary that was paused while `mode` was changed from `off` to `required` was demoted on resume and stayed a replica. In `automatic` mode the node was promoted later without a watchdog and without any warning.
- In `required` mode the HA loop gives up the leader key after a failed activation, but the watchdog stayed active. A reload that fixed the configuration was therefore deferred to a keepalive that never came, and the node stayed unhealthy until restart. E.g. after resuming a cluster in which `ttl` was set below 2x `loop_wait` during pause.

Remember that the device has to be acquired again after the timing check replaced it with `NullWatchdog`. Recreate the implementation in `activate()`, `keepalive()` and `reload_config()` of an inactive watchdog if it isn't running and that is pending, or if `mode` was changed. If activation fails, keep the watchdog active only while it is running.

Extracted from #3786.

### Reproduction

Single-node cluster on Raft with `ttl: 30`, `loop_wait: 10`, `watchdog.driver: testing`, and a FIFO reader on the device that emulates the softdog semantics (armed on open, disarmed on magic close, logs when the host would be reset).

| Scenario | master | this PR |
|---|---|---|
| `automatic`, `safety_margin: 12`, started with `ttl: 20`, then `edit-config -s ttl=30` | the device is never opened, `Error while sending keepalive` on every loop | `activated with 18 second timeout` 10 seconds later |
| pause, `mode: off` -> `automatic` and reload, resume | primary without the device ever being opened | `activated with 25 second timeout` right after resume |
| pause, `mode: off` -> `required` and reload, resume | demoted, replica with `watchdog_failed` until restart | stays primary, `activated with 25 second timeout` |
| `required`, `safety_margin: 12`: pause, `edit-config -s ttl=20`, resume, then `edit-config -s ttl=30` | demoted, replica with `watchdog_failed` until restart | demoted, then `activated with 18 second timeout` and promoted 10 seconds after `ttl=30` |

New tests: `test_reacquire_after_unsafe_timing`, `test_mode_off_after_unsafe_timing` and `test_reacquire_after_mode_change`.


inline patch 1: patroni/watchdog/base.py, tests/test_watchdog.py (+79/-4 lines) download inline-1.patchdiff --git a/patroni/watchdog/base.py b/patroni/watchdog/base.py index be731b98d3..5599226dbf 100644 --- a/patroni/watchdog/base.py +++ b/patroni/watchdog/base.py @@ -92,6 +92,7 @@ def __init__(self, config: Config) -> None: self.active_config: WatchdogConfig = self.config self.lock = RLock() self.active = False + self._impl_pending = False if self.config.mode == MODE_OFF: self.impl = NullWatchdog() @@ -110,12 +111,15 @@ def reload_config(self, config: Config) -> None: self._disable() self.active_config = self.config self.impl = NullWatchdog() + self._impl_pending = False # If watchdog is not active we can apply config immediately to show any warnings early. Otherwise we need to # delay until next time a keepalive is sent so timeout matches up with leader key update. if not self.active: - if self.config.driver != self.active_config.driver or \ - self.config.driver_config != self.active_config.driver_config: + if self._impl_pending or self.config.mode != self.active_config.mode or \ + self.config.driver != self.active_config.driver or \ + self.config.driver_config != self.active_config.driver_config: self.impl = self.config.get_impl() + self._impl_pending = False self.active_config = self.config @synchronized @@ -126,7 +130,16 @@ def activate(self) -> bool: :returns False if a safe watchdog could not be configured, but is required. """ self.active = True - return self._activate() + # A configuration change might have been deferred to a keepalive that was never sent + if not self.impl.is_running and (self._impl_pending or self.config.mode != self.active_config.mode): + self.impl = self.config.get_impl() + self._impl_pending = False + if self._activate(): + return True + # The HA loop gives up the leader key, so keepalive() won't apply configuration changes anymore. Unless a + # running watchdog has to be disabled on demotion, reload_config() must apply them, or the node stays unhealthy. + self.active = self.impl.is_running + return False def _activate(self) -> bool: self.active_config = self.config @@ -135,6 +148,8 @@ def _activate(self) -> bool: logger.warning('Watchdog not supported because leader TTL %s is less than 2x loop_wait %s', self.config.ttl, self.config.loop_wait) self.impl = NullWatchdog() + # Reacquire the configured device once a configuration change fixes the timing. + self._impl_pending = True try: self.impl.open() @@ -211,8 +226,9 @@ def keepalive(self) -> None: self.impl.keepalive() # In case there are any pending configuration changes apply them now. if self.active and self.config != self.active_config: - if self.config.mode != MODE_OFF and self.active_config.mode == MODE_OFF: + if not self.impl.is_running and (self._impl_pending or self.config.mode != self.active_config.mode): self.impl = self.config.get_impl() + self._impl_pending = False self._activate() if self.config.driver != self.active_config.driver \ or self.config.driver_config != self.active_config.driver_config: diff --git a/tests/test_watchdog.py b/tests/test_watchdog.py index 62dc6ccbf3..0eb2c717ad 100644 --- a/tests/test_watchdog.py +++ b/tests/test_watchdog.py @@ -164,9 +164,12 @@ def test_config_reload(self): self.assertFalse(watchdog.is_healthy) self.assertFalse(watchdog.activate()) + # the HA loop gives up the leader key, so the next activation picks up the reloaded configuration watchdog.reload_config({'ttl': 30, 'loop_wait': 15, 'watchdog': {'mode': 'required'}}) self.assertFalse(watchdog.is_running) watchdog.keepalive() + self.assertFalse(watchdog.is_running) + self.assertTrue(watchdog.activate()) self.assertTrue(watchdog.is_running) watchdog.reload_config({'ttl': 60, 'loop_wait': 15, 'watchdog': {'mode': 'required'}}) @@ -179,6 +182,62 @@ def test_config_reload(self): self.assertTrue(watchdog.is_running) self.assertEqual(watchdog.config.timeout, 60 // 2) + @patch('platform.system', Mock(return_value='Linux')) + @patch.object(LinuxWatchdogDevice, 'is_healthy', PropertyMock(return_value=True)) + def test_reacquire_after_unsafe_timing(self): + config = {'ttl': 20, 'loop_wait': 10, 'watchdog': {'safety_margin': 12}} + for mode in ('automatic', 'required'): + with self.subTest(mode=mode): + config['watchdog']['mode'] = mode + watchdog = Watchdog(config) + self.assertEqual(watchdog.activate(), mode == 'automatic') + self.assertFalse(watchdog.is_running) + self.assertEqual(watchdog.is_healthy, mode == 'automatic') + # automatic mode keeps the leader key, required mode gives it up + watchdog.reload_config(dict(config, ttl=30)) + if mode == 'automatic': + watchdog.keepalive() + else: + self.assertTrue(watchdog.is_healthy) + self.assertTrue(watchdog.activate()) + self.assertTrue(watchdog.is_running) + watchdog.disable() + + @patch('platform.system', Mock(return_value='Linux')) + def test_mode_off_after_unsafe_timing(self): + config = {'ttl': 20, 'loop_wait': 10, 'watchdog': {'mode': 'automatic', 'safety_margin': 12}} + for leader in (True, False): + with self.subTest(leader=leader): + watchdog = Watchdog(config) + self.assertTrue(watchdog.activate()) + if not leader: + watchdog.disable() + # the device must not be acquired anymore once the watchdog is turned off + watchdog.reload_config({'ttl': 30, 'loop_wait': 10, 'watchdog': {'mode': 'off'}}) + self.assertTrue(watchdog.activate()) + self.assertFalse(watchdog.is_running) + + @patch('platform.system', Mock(return_value='Linux')) + @patch.object(LinuxWatchdogDevice, 'is_healthy', PropertyMock(return_value=True)) + def test_reacquire_after_mode_change(self): + watchdog = Watchdog({'ttl': 30, 'loop_wait': 10, 'watchdog': {'mode': 'off'}}) + self.assertTrue(watchdog.activate()) + watchdog.disable() + watchdog.reload_config({'ttl': 30, 'loop_wait': 10, 'watchdog': {'mode': 'required'}}) + self.assertTrue(watchdog.is_healthy) + self.assertTrue(watchdog.activate()) + self.assertTrue(watchdog.is_running) + watchdog.disable() + + # the change was deferred to a keepalive, but the watchdog was disabled first + watchdog.reload_config({'ttl': 30, 'loop_wait': 10, 'watchdog': {'mode': 'off'}}) + self.assertTrue(watchdog.activate()) + watchdog.reload_config({'ttl': 30, 'loop_wait': 10, 'watchdog': {'mode': 'required'}}) + watchdog.disable() + self.assertTrue(watchdog.activate()) + self.assertTrue(watchdog.is_running) + watchdog.disable() + class TestNullWatchdog(unittest.TestCase):
^ permalink raw [nested|flat] 7+ messages in thread

* Re: [patroni/patroni] PR #3794: Retry watchdog activation after configuration changes
@ 2026-11-17 18:31  "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
  5 siblings, 0 replies; 7+ messages in thread

From: CyberDem0n (@CyberDem0n) @ 2026-11-17 18:31 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

@CyberDem0n requested a review.

^ permalink  raw  [nested|flat] 7+ messages in thread

* Re: [patroni/patroni] PR #3794: Retry watchdog activation after configuration changes
@ 2026-11-17 18:59  "codecov[bot] (bot) (@codecov[bot])" <noreply+codecov[bot]@github.com>
  5 siblings, 0 replies; 7+ messages in thread

From: codecov[bot] (bot) (@codecov[bot]) @ 2026-11-17 18:59 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

## [Codecov](https://app.codecov.io/gh/patroni/patroni/pull/3794?dropdown=coverage&src=pr&el=h1&utm_m...) Report
:white_check_mark: All modified and coverable lines are covered by tests.

:loudspeaker: Thoughts on this report? [Let us know!](https://github.com/codecov/feedback/issues/255)

^ permalink  raw  [nested|flat] 7+ messages in thread

* Re: [patroni/patroni] PR #3794: Retry watchdog activation after configuration changes
@ 2026-11-18 19:30  "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
  5 siblings, 0 replies; 7+ messages in thread

From: hughcapet (@hughcapet) @ 2026-11-18 19:30 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

@hughcapet applied label "bug".

^ permalink  raw  [nested|flat] 7+ messages in thread

* Re: [patroni/patroni] PR #3794 (approved): Retry watchdog activation after configuration changes
@ 2026-11-19 10:15  "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
  5 siblings, 0 replies; 7+ messages in thread

From: hughcapet (@hughcapet) @ 2026-11-19 10:15 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>



^ permalink  raw  [nested|flat] 7+ messages in thread

* Re: [patroni/patroni] PR #3794: Retry watchdog activation after configuration changes
@ 2026-11-19 10:16  "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
  5 siblings, 0 replies; 7+ messages in thread

From: hughcapet (@hughcapet) @ 2026-11-19 10:16 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

@hughcapet merged this (commit 43c891ad05e1).

^ permalink  raw  [nested|flat] 7+ messages in thread

* Re: [patroni/patroni] PR #3794: Retry watchdog activation after configuration changes
@ 2026-11-19 10:16  "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
  5 siblings, 0 replies; 7+ messages in thread

From: hughcapet (@hughcapet) @ 2026-11-19 10:16 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

@hughcapet closed this.

^ permalink  raw  [nested|flat] 7+ messages in thread


end of thread, other threads:[~2026-11-19 10:16 UTC | newest]

Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-11-17 18:31 [patroni/patroni] PR #3794: Retry watchdog activation after configuration changes "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-17 18:31 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-17 18:59 ` "codecov[bot] (bot) (@codecov[bot])" <noreply+codecov[bot]@github.com>
2026-11-18 19:30 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
2026-11-19 10:15 ` Re: [patroni/patroni] PR #3794 (approved): Retry watchdog activation after configuration changes "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
2026-11-19 10:16 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
2026-11-19 10:16 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox