From: CyberDem0n (@CyberDem0n) <noreply+CyberDem0n@github.com>
To: patroni/patroni <noreply+patroni-patroni@github.com>
Subject: [patroni/patroni] PR #3790: Fix QuorumStateResolver errors when changing synchronous_node_count while adding nodes
Date: Tue, 17 Nov 2026 16:07:48 +0000
Message-ID: <gh-patroni-patroni-3790@github.com> (raw)
`QuorumStateResolver` raised `QuorumError` in three situations where a safe transition exists. Transitions are generated before any of them is applied, therefore the exception is raised on every HA loop (`Unexpected exception raised, please report it as a BUG`), nothing is applied and synchronous replication management is stuck in the current state.
1. Adding nodes together with increasing `numsync` was interrupted after `synchronous_standby_names` was updated, but before `/sync` (e.g. the compare-and-set failed), and then one of the voters disappeared. Case 2 (`synchronous_standby_names` is a superset of `/sync`) tried to remove the dead voter from `synchronous_standby_names` while it was still in `/sync`. `synchronous_standby_names` kept the dead node and commits were blocked until it came back.
Now Case 2 removes only dead nodes that aren't voters, dead voters are removed afterwards by `__remove_gone_nodes()`.
2. `synchronous_node_count` was increased by more than the number of nodes being added at the same time (both changes seen in the same HA loop). `__add_new_nodes()` calculated `quorum` from the target `numsync`, although `synchronous_standby_names` was updated only by the number of added nodes. The new nodes were never added and the new `synchronous_node_count` was never applied.
Now `quorum` is calculated from the `numsync` that was actually set, and the rest of the increase is handled by `__handle_replication_factor_change()`.
3. `synchronous_node_count` was reduced by more than half while some sync nodes weren't yet confirmed to reach the "safe" LSN (e.g. right after a node was added), therefore `quorum` was already increased for them. `__handle_replication_factor_change()` updated `quorum` only if `quorum - sync_increase < len(voters)`, which could be false although `quorum` was still lower than required for the reduced `numsync`. The new `synchronous_node_count` wasn't applied until enough sync nodes were confirmed.
Now `quorum` is also updated if it is lower than required for the reduced `numsync`.
The test for 2 also needs the fix for 1, because `check_state_transitions()` also checks interrupted transitions with nodes gone away.
Reproduced on PostgreSQL 16 with 4 nodes in quorum mode and `nosync` tag on `node4`:
| Scenario | master | this PR |
|---|---|---|
| 1. `synchronous_node_count: 3`; `nosync` removed from `node4` while `/sync` writes fail; voter `node2` killed | `QuorumError` on every loop, `ANY 3 (node2,node3,node4)` kept, commits blocked | `ANY 2 (node3,node4)` right after `/sync` writes succeed again |
| 2. `synchronous_node_count` 1 → 3 and `nosync` removed from `node4` while the primary's Patroni was paused | `QuorumError` on every loop, `ANY 1 (node2,node3)` kept | `ANY 3 (node2,node3,node4)` with `quorum: 0` in two HA loops |
| 3. `synchronous_node_count: 3` and `synchronous_commit: remote_apply`; `nosync` removed from `node4` while it replays with `recovery_min_apply_delay: 1h`; then `synchronous_node_count` 3 → 1 | `QuorumError` on every loop, `ANY 3 (node2,node3,node4)` kept until `node4` catches up | `ANY 1 (node2,node3,node4)` with `quorum: 2` while `node4` is still behind |
1 and 2 are extracted from #3467.