pg.ddx.io  patroni/patroni GitHub issues and pull requests  
help / color / mirror / Atom feed
[patroni/patroni] PR #3813: Fix watchdog cleanup gaps in the HA loop
8+ messages / 3 participants
[nested] [flat]

* [patroni/patroni] PR #3813: Fix watchdog cleanup gaps in the HA loop
@ 2026-11-19 21:17 "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
  0 siblings, 0 replies; 8+ messages in thread

From: CyberDem0n (@CyberDem0n) @ 2026-11-19 21:17 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

`enforce_primary_role()` arms the watchdog before the promotion, and only the leader key holder feeds it in `update_lock()`. Several paths left the watchdog armed although the node didn't hold the leader key and PostgreSQL wasn't a primary. Nobody fed the watchdog anymore, and it reset a healthy replica after `ttl - safety_margin` seconds:

- **`pre_promote` fails.** `enforce_primary_role()` releases the leader key and returns `Promotion cancelled because the pre-promote script failed`. #3779 by @egor-voynov-aiven, already merged, disables the watchdog right there, unless PostgreSQL is running as a primary.
- **The leader key is lost while the promotion is pending.** `handle_long_action_in_progress()` cancels the promote task and returns `lost leader before promote`. If `CriticalTask.cancel()` succeeded, `Postgresql.promote()` returns before `pg_ctl promote`, so PostgreSQL stays a replica. The watchdog is now disabled right after the cancellation, unless PostgreSQL is running as a primary anyway, e.g. it was promoted manually.
- **The leader key is lost otherwise before the promotion**, e.g. it expired or another node took it over after the promotion was postponed because the synchronous replication state was updated by somebody else. `_run_cycle()` now disables the watchdog of a node without the leader key before calling `process_unhealthy_cluster()`/`process_healthy_cluster()`. It also covers the remaining corner cases, e.g. `pre_promote` failing right when the leader key was lost, so the promote task couldn't be cancelled anymore.
- **Updating the leader key failed.** `process_healthy_cluster()` demotes a primary, which disables the watchdog, but returned `not promoting because failed to update leader lock in DCS` for a replica without any cleanup.
- **DCS is not accessible.** `_handle_dcs_error()` returns before reaching the check in `_run_cycle()`. A pending promotion, e.g. a running `pre_promote` script, also kept the watchdog armed without anybody feeding it, so the host was reset if the promotion took longer than the watchdog timeout. Otherwise PostgreSQL was promoted without a leader key that could be renewed, and demoted in the next cycle if DCS was still not accessible. Unless failsafe mode is enabled, `_handle_dcs_error()` now cancels the promotion like `handle_long_action_in_progress()` does after losing the leader key (the code moved to `_cancel_promotion()`): the `pre_promote` script is killed and runs again when the node holds the leader key in a later HA cycle. In failsafe mode the promotion goes on, because the new primary might keep running, and the watchdog isn't fed until it completes.
- **A long action runs without the leader key.** `_run_cycle()` returns the result of `handle_long_action_in_progress()` before reaching the check either. A task started before the HA loop noticed that the leader key was lost kept the watchdog armed for its whole duration, e.g. a long basebackup. For example, a forced reinitialize cancels the pending promotion with `AsyncExecutor.cancel()`, so `handle_long_action_in_progress()` never sees `promote`. With the check in `_run_cycle()` the window is narrow, because the cancelled promotion wakes up the HA loop, but the REST API thread can still win the race. `handle_long_action_in_progress()` now also disables the watchdog of a node without the leader key while any other task is running.

The last four use the new `_disable_watchdog_unless_primary()`. It keeps the watchdog armed while `promote` is scheduled, the role is `primary`, `promoted` or `demoted`, or PostgreSQL is running as a primary: a primary without the leader key still has to be fenced until the existing code demotes it. `demoted` counts as a primary because `demote()` sets it even if `pg_ctl stop` failed, while `is_primary()` falls back to the cached role when it can't connect.

Three related changes:

- **A primary re-acquires the leader key.** A primary running without the leader key (see the comment in `is_healthiest_node()`) still has the watchdog armed, but `acquire_lock()` doesn't feed it since #452. The next keepalive happened in the next HA cycle, up to `2 * loop_wait` after the previous one. With e.g. `ttl: 20`, `loop_wait: 10`, `retry_timeout: 5` (a valid config, 15 second watchdog timeout with 5 seconds of timing slack) this resets the host. With etcd3, the change of the leader key wakes up the HA loop and the next cycle feeds the watchdog immediately, which hides the problem. `acquire_lock()` now sends a keepalive if the watchdog is running. Otherwise `keepalive()` could apply a deferred configuration change, e.g. to `mode: required`, and leave a device with an unsafe timeout armed, ignoring the failed activation. Then `enforce_primary_role()` would skip the activation and promote. With `primary_race_backoff` (#3453), `process_unhealthy_cluster()` also postponed the leader race of the primary like the one of the replicas: in the first HA cycle without the leader key it returned `wal position moved since last heart beat loop` without trying to acquire it, so even the keepalive in `acquire_lock()` came one HA cycle later. The backoff is documented for standbys and doesn't apply to the primary anymore.
- **Bootstrap of a standby cluster.** `post_bootstrap()` activated the watchdog also for a standby leader, so it stayed armed although the node isn't a primary, and in `required` mode the bootstrap was cancelled if the activation failed. A standby leader elected later (`enforce_follow_remote_member()`) never activates the watchdog, and the bootstrap now does the same.
- **A `pre_promote` script killed by a cancellation.** A forced reinitialize cancels the promotion with `AsyncExecutor.cancel()`, which kills the running `pre_promote` script without cancelling the `CriticalTask`. `Postgresql.promote()` stored the exit code of the killed script as the result, and the reinitialize doesn't reset it. The next time the node won the leader race, `enforce_primary_role()` released the leader key with `Promotion cancelled because the pre-promote script failed` and skipped the leader race for `ttl` seconds. `promote()` now doesn't store the result if the script was killed by a cancellation, and logs `PostgreSQL promote cancelled.`

In the DCS outage case the watchdog can stay armed for one more HA cycle after a failed promotion, which can't be cancelled anymore: `AsyncExecutor.run()` resets the scheduled action under the executor lock, which is held by the running HA cycle. The next cycle, woken up by the executor, disables it.

The gaps don't depend on the watchdog driver, so they would also trigger the thread fallback from #3786. Disabling the watchdog disarms that fallback too. Merging #3786, #3805, #3806 or #3656 into this branch is clean and passes `test_ha.py`, `test_watchdog.py` and `test_patroni.py`.

### Reproduction

etcd 3.4 and a two-node cluster with `ttl: 30`, `loop_wait: 10`, `retry_timeout: 10` unless noted. postgresql0 is the primary with the watchdog off. postgresql1 has `watchdog.mode: automatic`, `watchdog.driver: testing` with a FIFO reader on the device that emulates the softdog semantics (armed on open, magic close, timeout set by Patroni), and a `pre_promote` script. postgresql0 is stopped gracefully, postgresql1 takes the leader key, activates the watchdog with a 25 second timeout and runs `pre_promote`:

| `pre_promote` | master | this PR |
|---|---|---|
| exits with 1: the leader key is released | fires 25 seconds after the release | disarmed 0.1 seconds after the release, in the same HA cycle; the same after the retry when the backoff expired |
| overwrites the leader key and sleeps: `lost leader before promote` | fires 25 seconds after the last keepalive | disarmed in the cycle that cancels the promotion, a second after the activation |
| kills etcd and exits with 1: `DCS is not accessible` | fires 25 seconds after the activation | disarmed in the second DCS error cycle, 17.8 seconds after the activation |
| sleeps; the leader key is overwritten, then `POST /reinitialize` with `{"force": true}` cancels the promotion (etcd v2 API, see below) | fires 25 seconds after the activation, while the replica is being reinitialized | disarmed 0.01 seconds after the cancellation, in the HA cycle woken up by the cancelled promotion, before the reinitialize starts |

The `master` column was measured before #3779 was merged; #3779 fixes the first scenario. Without it, the check in `_run_cycle()` disarmed the watchdog in that scenario in the next HA cycle, 10 seconds after the release.

In the reinitialize scenario, the HA loop doesn't watch the leader key while it holds it with the etcd v2 API, so the change isn't noticed before the REST API call. The HA cycle woken up by the cancelled promotion then usually runs before the REST API thread schedules the reinitialize. To hit the race, the reaction of the HA loop to the change of the leader key was delayed by 3 seconds (etcd3, a wrapper around `Etcd3.watch()`, `ha.py` unchanged). Without the `_disable_watchdog_unless_primary()` call in `handle_long_action_in_progress()`, the following cycles return `reinitialize in progress` and the watchdog fires 25 seconds after the last keepalive. With it, the watchdog is disarmed in the first of these cycles.

Primary re-acquiring the leader key: etcd v2 API, `ttl: 20`, `retry_timeout: 5` (15 second watchdog timeout). postgresql1 became the primary, then its leader key was deleted 6 seconds after a keepalive.

- master: postgresql1 acquires the leader key 10 seconds after the last keepalive without feeding the watchdog, and it fires 5 seconds later. The next keepalive comes 20 seconds after the previous one.
- this PR: the keepalive is sent when the leader key is acquired.

With etcd3 master doesn't fire in this scenario, because the next keepalive is sent 0.1 seconds after acquiring the leader key.

### Tests

- `test_lost_leader_lock_during_promote` (extended)
- `test_disable_watchdog_without_lock` (new)
- `test_disable_watchdog_when_not_promoting_because_update_lock_failed` (new)
- `test_no_dcs_connection_replica_disables_watchdog` (new, also covers the cancelled promotion)
- `test_disable_watchdog_during_long_action_without_lock` (new)
- `test_acquire_lock_as_primary` (extended)
- `test_leader_race_with_deferred_watchdog_config` (new)
- `test_bootstrap_as_standby_leader` (extended)
- `test_promote` (extended)

Each of them fails without the corresponding change, and all but `test_leader_race_with_deferred_watchdog_config`, which guards the `is_running` check in `acquire_lock()`, also fail on master. Removing any of the conditions in `_disable_watchdog_unless_primary()` also breaks a test.






^ permalink  raw  [nested|flat] 8+ messages in thread

* Re: [patroni/patroni] PR #3813: Fix watchdog cleanup gaps in the HA loop
@ 2026-11-19 21:17 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
  6 siblings, 0 replies; 8+ messages in thread

From: CyberDem0n (@CyberDem0n) @ 2026-11-19 21:17 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

@CyberDem0n requested a review.

^ permalink  raw  [nested|flat] 8+ messages in thread

* Re: [patroni/patroni] PR #3813: Fix watchdog cleanup gaps in the HA loop
@ 2026-11-20 05:03 ` "codecov[bot] (bot) (@codecov[bot])" <noreply+codecov[bot]@github.com>
  6 siblings, 0 replies; 8+ messages in thread

From: codecov[bot] (bot) (@codecov[bot]) @ 2026-11-20 05:03 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

## [Codecov](https://app.codecov.io/gh/patroni/patroni/pull/3813?dropdown=coverage&src=pr&el=h1&utm_m...) Report
:white_check_mark: All modified and coverable lines are covered by tests.

:loudspeaker: Thoughts on this report? [Let us know!](https://github.com/codecov/feedback/issues/255)

^ permalink  raw  [nested|flat] 8+ messages in thread

* Re: [patroni/patroni] PR #3813: Fix watchdog cleanup gaps in the HA loop
@ 2026-11-20 07:58 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
  6 siblings, 0 replies; 8+ messages in thread

From: hughcapet (@hughcapet) @ 2026-11-20 07:58 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

@hughcapet applied label "bug".

^ permalink  raw  [nested|flat] 8+ messages in thread

* Re: [patroni/patroni] PR #3813 (dismissed): Fix watchdog cleanup gaps in the HA loop
@ 2026-11-20 08:50 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
  6 siblings, 0 replies; 8+ messages in thread

From: hughcapet (@hughcapet) @ 2026-11-20 08:50 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>



^ permalink  raw  [nested|flat] 8+ messages in thread

* Re: [patroni/patroni] PR #3813 (approved): Fix watchdog cleanup gaps in the HA loop
@ 2026-11-20 14:51 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
  6 siblings, 0 replies; 8+ messages in thread

From: hughcapet (@hughcapet) @ 2026-11-20 14:51 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>



^ permalink  raw  [nested|flat] 8+ messages in thread

* Re: [patroni/patroni] PR #3813: Fix watchdog cleanup gaps in the HA loop
@ 2026-11-20 14:53 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
  6 siblings, 0 replies; 8+ messages in thread

From: CyberDem0n (@CyberDem0n) @ 2026-11-20 14:53 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

@CyberDem0n merged this (commit acd27d25e08c).

^ permalink  raw  [nested|flat] 8+ messages in thread

* Re: [patroni/patroni] PR #3813: Fix watchdog cleanup gaps in the HA loop
@ 2026-11-20 14:53 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
  6 siblings, 0 replies; 8+ messages in thread

From: CyberDem0n (@CyberDem0n) @ 2026-11-20 14:53 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

@CyberDem0n closed this.

^ permalink  raw  [nested|flat] 8+ messages in thread


end of thread, other threads:[~2026-11-20 14:53 UTC | newest]

Thread overview: 8+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-11-19 21:17 [patroni/patroni] PR #3813: Fix watchdog cleanup gaps in the HA loop "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-19 21:17 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-20 05:03 ` "codecov[bot] (bot) (@codecov[bot])" <noreply+codecov[bot]@github.com>
2026-11-20 07:58 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
2026-11-20 08:50 ` Re: [patroni/patroni] PR #3813 (dismissed): Fix watchdog cleanup gaps in the HA loop "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
2026-11-20 14:51 ` Re: [patroni/patroni] PR #3813 (approved): Fix watchdog cleanup gaps in the HA loop "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
2026-11-20 14:53 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-20 14:53 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox