pg.ddx.io  patroni/patroni GitHub issues and pull requests  
help / color / mirror / Atom feed
[patroni/patroni] PR #3790: Fix QuorumStateResolver errors when changing synchronous_node_count while adding nodes
8+ messages / 3 participants
[nested] [flat]

* [patroni/patroni] PR #3790: Fix QuorumStateResolver errors when changing synchronous_node_count while adding nodes
@ 2026-11-17 16:07 "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
  0 siblings, 0 replies; 8+ messages in thread

From: CyberDem0n (@CyberDem0n) @ 2026-11-17 16:07 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

`QuorumStateResolver` raised `QuorumError` in three situations where a safe transition exists. Transitions are generated before any of them is applied, therefore the exception is raised on every HA loop (`Unexpected exception raised, please report it as a BUG`), nothing is applied and synchronous replication management is stuck in the current state.

1. Adding nodes together with increasing `numsync` was interrupted after `synchronous_standby_names` was updated, but before `/sync` (e.g. the compare-and-set failed), and then one of the voters disappeared. Case 2 (`synchronous_standby_names` is a superset of `/sync`) tried to remove the dead voter from `synchronous_standby_names` while it was still in `/sync`. `synchronous_standby_names` kept the dead node and commits were blocked until it came back.
   Now Case 2 removes only dead nodes that aren't voters, dead voters are removed afterwards by `__remove_gone_nodes()`.
2. `synchronous_node_count` was increased by more than the number of nodes being added at the same time (both changes seen in the same HA loop). `__add_new_nodes()` calculated `quorum` from the target `numsync`, although `synchronous_standby_names` was updated only by the number of added nodes. The new nodes were never added and the new `synchronous_node_count` was never applied.
   Now `quorum` is calculated from the `numsync` that was actually set, and the rest of the increase is handled by `__handle_replication_factor_change()`.
3. `synchronous_node_count` was reduced by more than half while some sync nodes weren't yet confirmed to reach the "safe" LSN (e.g. right after a node was added), therefore `quorum` was already increased for them. `__handle_replication_factor_change()` updated `quorum` only if `quorum - sync_increase < len(voters)`, which could be false although `quorum` was still lower than required for the reduced `numsync`. The new `synchronous_node_count` wasn't applied until enough sync nodes were confirmed.
   Now `quorum` is also updated if it is lower than required for the reduced `numsync`.

The test for 2 also needs the fix for 1, because `check_state_transitions()` also checks interrupted transitions with nodes gone away.

Reproduced on PostgreSQL 16 with 4 nodes in quorum mode and `nosync` tag on `node4`:

| Scenario | master | this PR |
|---|---|---|
| 1. `synchronous_node_count: 3`; `nosync` removed from `node4` while `/sync` writes fail; voter `node2` killed | `QuorumError` on every loop, `ANY 3 (node2,node3,node4)` kept, commits blocked | `ANY 2 (node3,node4)` right after `/sync` writes succeed again |
| 2. `synchronous_node_count` 1 → 3 and `nosync` removed from `node4` while the primary's Patroni was paused | `QuorumError` on every loop, `ANY 1 (node2,node3)` kept | `ANY 3 (node2,node3,node4)` with `quorum: 0` in two HA loops |
| 3. `synchronous_node_count: 3` and `synchronous_commit: remote_apply`; `nosync` removed from `node4` while it replays with `recovery_min_apply_delay: 1h`; then `synchronous_node_count` 3 → 1 | `QuorumError` on every loop, `ANY 3 (node2,node3,node4)` kept until `node4` catches up | `ANY 1 (node2,node3,node4)` with `quorum: 2` while `node4` is still behind |

1 and 2 are extracted from #3467.


^ permalink  raw  [nested|flat] 8+ messages in thread

* Re: [patroni/patroni] PR #3790: Fix QuorumStateResolver errors in the middle of adding nodes
@ 2026-11-17 16:07 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
  6 siblings, 0 replies; 8+ messages in thread

From: CyberDem0n (@CyberDem0n) @ 2026-11-17 16:07 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

@CyberDem0n requested a review.

^ permalink  raw  [nested|flat] 8+ messages in thread

* Re: [patroni/patroni] PR #3790: Fix QuorumStateResolver errors in the middle of adding nodes
@ 2026-11-17 16:32 ` "codecov[bot] (bot) (@codecov[bot])" <noreply+codecov[bot]@github.com>
  6 siblings, 0 replies; 8+ messages in thread

From: codecov[bot] (bot) (@codecov[bot]) @ 2026-11-17 16:32 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

## [Codecov](https://app.codecov.io/gh/patroni/patroni/pull/3790?dropdown=coverage&src=pr&el=h1&utm_m...) Report
:white_check_mark: All modified and coverable lines are covered by tests.

:loudspeaker: Thoughts on this report? [Let us know!](https://github.com/codecov/feedback/issues/255)

^ permalink  raw  [nested|flat] 8+ messages in thread

* Re: [patroni/patroni] PR #3790: Fix QuorumStateResolver errors in the middle of adding nodes
@ 2026-11-18 17:25 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
  6 siblings, 0 replies; 8+ messages in thread

From: hughcapet (@hughcapet) @ 2026-11-18 17:25 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

@hughcapet applied label "bug".

^ permalink  raw  [nested|flat] 8+ messages in thread

* Re: [patroni/patroni] PR #3790: Fix QuorumStateResolver errors when changing synchronous_node_count while adding nodes
@ 2026-11-18 19:30 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
  6 siblings, 0 replies; 8+ messages in thread

From: CyberDem0n (@CyberDem0n) @ 2026-11-18 19:30 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

@CyberDem0n renamed this.

^ permalink  raw  [nested|flat] 8+ messages in thread

* Re: [patroni/patroni] PR #3790 (approved): Fix QuorumStateResolver errors when changing synchronous_node_count while adding nodes
@ 2026-11-19 17:50 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
  6 siblings, 0 replies; 8+ messages in thread

From: hughcapet (@hughcapet) @ 2026-11-19 17:50 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>



^ permalink  raw  [nested|flat] 8+ messages in thread

* Re: [patroni/patroni] PR #3790: Fix QuorumStateResolver errors when changing synchronous_node_count while adding nodes
@ 2026-11-19 17:51 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
  6 siblings, 0 replies; 8+ messages in thread

From: hughcapet (@hughcapet) @ 2026-11-19 17:51 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

@hughcapet merged this (commit 5f192dd5cd55).

^ permalink  raw  [nested|flat] 8+ messages in thread

* Re: [patroni/patroni] PR #3790: Fix QuorumStateResolver errors when changing synchronous_node_count while adding nodes
@ 2026-11-19 17:51 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
  6 siblings, 0 replies; 8+ messages in thread

From: hughcapet (@hughcapet) @ 2026-11-19 17:51 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

@hughcapet closed this.

^ permalink  raw  [nested|flat] 8+ messages in thread


end of thread, other threads:[~2026-11-19 17:51 UTC | newest]

Thread overview: 8+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-11-17 16:07 [patroni/patroni] PR #3790: Fix QuorumStateResolver errors when changing synchronous_node_count while adding nodes "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-17 16:07 ` Re: [patroni/patroni] PR #3790: Fix QuorumStateResolver errors in the middle of adding nodes "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-17 16:32 ` Re: [patroni/patroni] PR #3790: Fix QuorumStateResolver errors in the middle of adding nodes "codecov[bot] (bot) (@codecov[bot])" <noreply+codecov[bot]@github.com>
2026-11-18 17:25 ` Re: [patroni/patroni] PR #3790: Fix QuorumStateResolver errors in the middle of adding nodes "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
2026-11-18 19:30 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-19 17:50 ` Re: [patroni/patroni] PR #3790 (approved): Fix QuorumStateResolver errors when changing synchronous_node_count while adding nodes "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
2026-11-19 17:51 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
2026-11-19 17:51 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox