pg.ddx.io patroni/patroni GitHub issues and pull requestshelp / color / mirror / Atom feed
[patroni/patroni] PR #3790: Fix QuorumStateResolver errors when changing synchronous_node_count while adding nodes 8+ messages / 3 participants [nested] [flat]
* [patroni/patroni] PR #3790: Fix QuorumStateResolver errors when changing synchronous_node_count while adding nodes @ 2026-11-17 16:07 "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com> 0 siblings, 0 replies; 8+ messages in thread From: CyberDem0n (@CyberDem0n) @ 2026-11-17 16:07 UTC (permalink / raw) To: patroni/patroni <noreply+patroni-patroni@github.com> `QuorumStateResolver` raised `QuorumError` in three situations where a safe transition exists. Transitions are generated before any of them is applied, therefore the exception is raised on every HA loop (`Unexpected exception raised, please report it as a BUG`), nothing is applied and synchronous replication management is stuck in the current state. 1. Adding nodes together with increasing `numsync` was interrupted after `synchronous_standby_names` was updated, but before `/sync` (e.g. the compare-and-set failed), and then one of the voters disappeared. Case 2 (`synchronous_standby_names` is a superset of `/sync`) tried to remove the dead voter from `synchronous_standby_names` while it was still in `/sync`. `synchronous_standby_names` kept the dead node and commits were blocked until it came back. Now Case 2 removes only dead nodes that aren't voters, dead voters are removed afterwards by `__remove_gone_nodes()`. 2. `synchronous_node_count` was increased by more than the number of nodes being added at the same time (both changes seen in the same HA loop). `__add_new_nodes()` calculated `quorum` from the target `numsync`, although `synchronous_standby_names` was updated only by the number of added nodes. The new nodes were never added and the new `synchronous_node_count` was never applied. Now `quorum` is calculated from the `numsync` that was actually set, and the rest of the increase is handled by `__handle_replication_factor_change()`. 3. `synchronous_node_count` was reduced by more than half while some sync nodes weren't yet confirmed to reach the "safe" LSN (e.g. right after a node was added), therefore `quorum` was already increased for them. `__handle_replication_factor_change()` updated `quorum` only if `quorum - sync_increase < len(voters)`, which could be false although `quorum` was still lower than required for the reduced `numsync`. The new `synchronous_node_count` wasn't applied until enough sync nodes were confirmed. Now `quorum` is also updated if it is lower than required for the reduced `numsync`. The test for 2 also needs the fix for 1, because `check_state_transitions()` also checks interrupted transitions with nodes gone away. Reproduced on PostgreSQL 16 with 4 nodes in quorum mode and `nosync` tag on `node4`: | Scenario | master | this PR | |---|---|---| | 1. `synchronous_node_count: 3`; `nosync` removed from `node4` while `/sync` writes fail; voter `node2` killed | `QuorumError` on every loop, `ANY 3 (node2,node3,node4)` kept, commits blocked | `ANY 2 (node3,node4)` right after `/sync` writes succeed again | | 2. `synchronous_node_count` 1 → 3 and `nosync` removed from `node4` while the primary's Patroni was paused | `QuorumError` on every loop, `ANY 1 (node2,node3)` kept | `ANY 3 (node2,node3,node4)` with `quorum: 0` in two HA loops | | 3. `synchronous_node_count: 3` and `synchronous_commit: remote_apply`; `nosync` removed from `node4` while it replays with `recovery_min_apply_delay: 1h`; then `synchronous_node_count` 3 → 1 | `QuorumError` on every loop, `ANY 3 (node2,node3,node4)` kept until `node4` catches up | `ANY 1 (node2,node3,node4)` with `quorum: 2` while `node4` is still behind | 1 and 2 are extracted from #3467. ^ permalink raw [nested|flat] 8+ messages in thread
* Re: [patroni/patroni] PR #3790: Fix QuorumStateResolver errors in the middle of adding nodes @ 2026-11-17 16:07 "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com> 6 siblings, 0 replies; 8+ messages in thread From: CyberDem0n (@CyberDem0n) @ 2026-11-17 16:07 UTC (permalink / raw) To: patroni/patroni <noreply+patroni-patroni@github.com> @CyberDem0n requested a review. ^ permalink raw [nested|flat] 8+ messages in thread
* Re: [patroni/patroni] PR #3790: Fix QuorumStateResolver errors in the middle of adding nodes @ 2026-11-17 16:32 "codecov[bot] (bot) (@codecov[bot])" <noreply+codecov[bot]@github.com> 6 siblings, 0 replies; 8+ messages in thread From: codecov[bot] (bot) (@codecov[bot]) @ 2026-11-17 16:32 UTC (permalink / raw) To: patroni/patroni <noreply+patroni-patroni@github.com> ## [Codecov](https://app.codecov.io/gh/patroni/patroni/pull/3790?dropdown=coverage&src=pr&el=h1&utm_m...) Report :white_check_mark: All modified and coverable lines are covered by tests. :loudspeaker: Thoughts on this report? [Let us know!](https://github.com/codecov/feedback/issues/255) ^ permalink raw [nested|flat] 8+ messages in thread
* Re: [patroni/patroni] PR #3790: Fix QuorumStateResolver errors in the middle of adding nodes @ 2026-11-18 17:25 "hughcapet (@hughcapet)" <noreply+hughcapet@github.com> 6 siblings, 0 replies; 8+ messages in thread From: hughcapet (@hughcapet) @ 2026-11-18 17:25 UTC (permalink / raw) To: patroni/patroni <noreply+patroni-patroni@github.com> @hughcapet applied label "bug". ^ permalink raw [nested|flat] 8+ messages in thread
* Re: [patroni/patroni] PR #3790: Fix QuorumStateResolver errors when changing synchronous_node_count while adding nodes @ 2026-11-18 19:30 "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com> 6 siblings, 0 replies; 8+ messages in thread From: CyberDem0n (@CyberDem0n) @ 2026-11-18 19:30 UTC (permalink / raw) To: patroni/patroni <noreply+patroni-patroni@github.com> @CyberDem0n renamed this. ^ permalink raw [nested|flat] 8+ messages in thread
* Re: [patroni/patroni] PR #3790 (approved): Fix QuorumStateResolver errors when changing synchronous_node_count while adding nodes @ 2026-11-19 17:50 "hughcapet (@hughcapet)" <noreply+hughcapet@github.com> 6 siblings, 0 replies; 8+ messages in thread From: hughcapet (@hughcapet) @ 2026-11-19 17:50 UTC (permalink / raw) To: patroni/patroni <noreply+patroni-patroni@github.com> ^ permalink raw [nested|flat] 8+ messages in thread
* Re: [patroni/patroni] PR #3790: Fix QuorumStateResolver errors when changing synchronous_node_count while adding nodes @ 2026-11-19 17:51 "hughcapet (@hughcapet)" <noreply+hughcapet@github.com> 6 siblings, 0 replies; 8+ messages in thread From: hughcapet (@hughcapet) @ 2026-11-19 17:51 UTC (permalink / raw) To: patroni/patroni <noreply+patroni-patroni@github.com> @hughcapet merged this (commit 5f192dd5cd55). ^ permalink raw [nested|flat] 8+ messages in thread
* Re: [patroni/patroni] PR #3790: Fix QuorumStateResolver errors when changing synchronous_node_count while adding nodes @ 2026-11-19 17:51 "hughcapet (@hughcapet)" <noreply+hughcapet@github.com> 6 siblings, 0 replies; 8+ messages in thread From: hughcapet (@hughcapet) @ 2026-11-19 17:51 UTC (permalink / raw) To: patroni/patroni <noreply+patroni-patroni@github.com> @hughcapet closed this. ^ permalink raw [nested|flat] 8+ messages in thread
end of thread, other threads:[~2026-11-19 17:51 UTC | newest] Thread overview: 8+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2026-11-17 16:07 [patroni/patroni] PR #3790: Fix QuorumStateResolver errors when changing synchronous_node_count while adding nodes "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com> 2026-11-17 16:07 ` Re: [patroni/patroni] PR #3790: Fix QuorumStateResolver errors in the middle of adding nodes "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com> 2026-11-17 16:32 ` Re: [patroni/patroni] PR #3790: Fix QuorumStateResolver errors in the middle of adding nodes "codecov[bot] (bot) (@codecov[bot])" <noreply+codecov[bot]@github.com> 2026-11-18 17:25 ` Re: [patroni/patroni] PR #3790: Fix QuorumStateResolver errors in the middle of adding nodes "hughcapet (@hughcapet)" <noreply+hughcapet@github.com> 2026-11-18 19:30 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com> 2026-11-19 17:50 ` Re: [patroni/patroni] PR #3790 (approved): Fix QuorumStateResolver errors when changing synchronous_node_count while adding nodes "hughcapet (@hughcapet)" <noreply+hughcapet@github.com> 2026-11-19 17:51 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com> 2026-11-19 17:51 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox