From: CyberDem0n (@CyberDem0n) <noreply+CyberDem0n@github.com>
To: patroni/patroni <noreply+patroni-patroni@github.com>
Subject: [patroni/patroni] PR #3811: Harden watchdog device replacement and refactor configuration handling
Date: Thu, 19 Nov 2026 21:07:55 +0000
Message-ID: <gh-patroni-patroni-3811@github.com> (raw)
Split out of #3786, which now only adds the thread fallback on top of this.
- Add `WatchdogBase.disable()`, implemented for Linux devices by writing the magic character followed by `WDIOS_DISABLECARD`. Nothing is written after the timer stopped, because legacy drivers restart the timer on every write. If disabling fails, the magic character is revoked.
- Confirm the old implementation stopped before replacing it on mode/driver changes and activation errors. A device that can't be stopped (e.g. `nowayout`) is retained and kept fed instead of being closed and left to reboot the host.
- Add `Watchdog.is_activated` next to `is_running`: a retained device has to be fed, but isn't necessarily safe to lead with. The HA loop checks `is_activated` before promoting, while running as a primary, on the busy path during a promotion, and on the DCS failsafe path. So a `required` mode node gives up leadership when a configuration change applied on keepalive (e.g. a device replacement) doesn't result in a safe watchdog: a promotion is cancelled and the leader key released (if it is too late to cancel, the primary is demoted once the promotion completes), a failsafe primary is demoted offline. A replica also replaces a device that was left open before it promotes.
- A watchdog whose actual timeout leaves less than `retry_timeout` after `loop_wait`, e.g. a fixed-timeout device or the timeout armed for the previous configuration after `loop_wait` was raised, is no longer disabled. Disabling it left a primary unprotected or a `nowayout` timer unfed. Instead it stays armed and the HA loop runs every actual timeout minus `retry_timeout`, but not more often than every half of the timeout, while the watchdog is active (`Watchdog.keepalive_interval`, via `Ha.effective_loop_wait`). The HA loop doesn't wait longer within a cycle either: a scheduled switchover/restart further away than that is deferred to the next cycle, the quorum commit loop is bounded by it, and in priority mode the 2 s wait for a new synchronous standby is skipped, leaving its confirmation to the next cycle. If reading the timeout fails while the descriptor stays open (e.g. a transient `EIO`), the last known timeout is used. That is the timeout the driver returned from `WDIOC_SETTIMEOUT` (`set_timeout()` now returns it), not the requested one. **Behavior changes:** master disabled a timer shorter than `loop_wait` on activation, so in `required` mode the node wasn't eligible. Configurations with `ttl - safety_margin - loop_wait < retry_timeout` now run the primary's HA loop more often and log a warning on activation, e.g. `safety_margin: -1` with defaults every 7.5 s instead of 10 s.
- Extract `_apply_config_changes()` from `keepalive()`: when a reload results in a negative timing slack (no timeout is safe), keep the armed timer fed until it is confirmed as disabled (in `required` mode until the demotion safepoint), and reacquire the device once the timing is fixed. In `automatic` mode the timer is disabled explicitly first, a magic close is only used if the driver doesn't support disabling (`ENOTTY`/`EOPNOTSUPP`/`ENOSYS`). A refused stop (e.g. `EBUSY` because of `nowayout`) retains the device. Only set the timeout when it changed.
- Release a device that disappeared (`ENODEV`) so it can be replaced, and apply configuration changes even if the keepalive failed. Query the device capabilities again on every open, another device might have replaced a removed one at the same path.
- A timeout of zero, e.g. from a driver counting in whole minutes (PC87413 rounds 25 s down), isn't considered safe in `required` mode and doesn't make the HA loop spin.
- Don't lose a driver/device change that was deferred to a keepalive which was never sent: `disable()` now applies pending configuration (a device that failed to close is kept until the next activation confirms it is disabled), and `activate()` also replaces the implementation on a driver/device change, e.g. on a node that lost the leader key with a null watchdog, which the HA loop doesn't disable. `is_healthy` now also checks the implementation the next activation would use: such a node switched to `mode: required` checked the old null watchdog, so it didn't take part in the leader race (`Watchdog device is not usable`) and the REST API reported `watchdog_failed`.
Docs: that the timeout the device actually sets is used (zero isn't safe), the faster HA loop for short watchdog timeouts, the reduced tolerance for slow leader key updates when half of the timeout is shorter than `retry_timeout`, and that changing the mode to `off` is applied immediately and closes the device, so a `nowayout` timer isn't fed anymore and resets the system.
Known limitations:
- Some legacy drivers (e.g. `advantechwdt`) stop the timer on `WDIOS_DISABLECARD` even with `nowayout`, but ignore the magic character and restart the timer on close. This can't be detected.
- Other long actions holding the leader key (`restart`, `starting primary after failure`) aren't interrupted; `enforce_primary_role()` demotes once they complete.
- Waits within one HA cycle (synchronous standby confirmation, scheduled restart) are each bounded by the interval, not their sum. The same already applies to `loop_wait`.