pg.ddx.io patroni/patroni GitHub issues and pull requests
help / color / mirror / Atom feed[patroni/patroni] PR #3817: Make remaining quorum members catch up when quorum becomes stricter
4+ messages / 2 participants
[nested] [flat]
* [patroni/patroni] PR #3817: Make remaining quorum members catch up when quorum becomes stricter
@ 2026-11-19 21:59 "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
0 siblings, 0 replies; 4+ messages in thread
From: CyberDem0n (@CyberDem0n) @ 2026-11-19 21:59 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
In quorum commit mode, when nodes are removed from `synchronous_standby_names` or `num` is increased, a transaction acknowledged before the change may exist only on the removed nodes or on fewer than `num` nodes. The same applies when the previous value is unknown, i.e. after Patroni restart or promote.
Remaining members are now counted as synchronous only after they reach the primary flush LSN recorded after the change, and until then voters aren't removed and the quorum isn't reduced in `/sync`. We wait for:
- only members that are streaming. Others can't acknowledge commits and will use the new value after reconnect;
- `min(num, synchronous_node_count)` members, or all eligible ones if there are not enough of them. Lowering `synchronous_node_count` doesn't wait for more members than needed.
While catching up, `QuorumStateResolver` only grows membership without weakening the election requirement: new healthy standbys are first added to voters with the quorum increased accordingly, and only after that to `synchronous_standby_names`.
After a failover the new primary immediately replaces the former one as the leader in `/sync`, and the former primary takes its place among the voters. The set of nodes that could vote and the quorum don't change, but the former leader loses the privilege to be promoted without votes while `synchronous_standby_names` is being verified.
The `QuorumStateResolver` bugs that became easier to hit because of deferred `/sync` updates were already fixed in #3790.
This PR doesn't depend on the checkpointer verification below it. It is stacked on top of it only because stacks are linear. Extracted from #3467.
---
Stack replacing #3467, bottom to top:
1. #3814 Reset synchronous replication state on demote (independent, can be merged on its own)
2. #3815 Verify that walsenders apply synchronous_standby_names before updating /sync
3. #3816 Make sure checkpointer enforces synchronous_standby_names
4. #3817 Make remaining quorum members catch up when quorum becomes stricter ← **this PR**
inline patch 1: docs/replication_modes.rst, patroni/ha.py, patroni/postgresql/sync.py and 4 more (+352/-27 lines) download inline-1.patch
diff --git a/docs/replication_modes.rst b/docs/replication_modes.rst
index 5039f91240..9ce912434f 100644
--- a/docs/replication_modes.rst
+++ b/docs/replication_modes.rst
@@ -184,6 +184,8 @@ In this mode, Patroni maintains synchronization state in the DCS, containing the
On each iteration of HA loop, Patroni re-evaluates synchronous standby choices and quorum, based on node availability and requested cluster configuration. In PostgreSQL versions above 9.6 all eligible nodes are added as synchronous standbys as soon as their replication catches up to leader.
+When the quorum requirement in ``synchronous_standby_names`` becomes stricter (some nodes are removed from it or ``ANY N`` is increased), or when its previous value is unknown (after Patroni restart or promote), recent commits might have been confirmed only by nodes that are no longer listed, or by fewer nodes than the new value requires. Until enough of the listed and streaming standbys (``min(N, synchronous_node_count)``) have caught up with the primary's WAL position, Patroni doesn't remove voters and doesn't reduce the quorum in the ``/sync`` key. A change of only ``N`` isn't visible in ``pg_stat_replication`` (``sync_state`` is ``quorum`` either way), therefore Patroni can't confirm that walsender processes already use the new value. They process the configuration reload before handling the next replies from standbys, which makes the window when a commit could still be confirmed using the old ``N`` very short. After a failover the new primary still replaces the former one as the leader in the ``/sync`` key, and the former primary takes its place among the voters. New healthy standbys can still be added during this time: Patroni first adds them as voters and increases the quorum accordingly, and only after that adds them to ``synchronous_standby_names``.
+
Quorum commit helps to reduce worst case latencies, even during normal operation, as a higher latency of replicating to one standby can be compensated by other standbys.
The quorum-based synchronous mode could be enabled by setting ``synchronous_mode`` to ``quorum`` using ``patronictl edit-config`` command or via Patroni REST interface. See :ref:`dynamic configuration <dynamic_configuration>` for instructions.
diff --git a/patroni/ha.py b/patroni/ha.py
index 8f5c24eb7b..6e542b650e 100644
--- a/patroni/ha.py
+++ b/patroni/ha.py
@@ -963,6 +963,21 @@ def _check_timeout(offset: float = 0) -> bool:
while True:
sync_state = self.state_handler.sync_handler.current_state(self.cluster)
if sync_state is None:
+ if leader != self.state_handler.name: # failover
+ # Verification of synchronous_standby_names might take long after promote, while commits
+ # could already be acknowledged by standbys. The former leader must immediately lose the
+ # privilege to be promoted without votes. It takes the place of the new leader in voters,
+ # therefore the set of nodes that could vote and the quorum don't change.
+ voters = CaseInsensitiveSet((CaseInsensitiveSet(sync.voters)
+ - CaseInsensitiveSet([self.state_handler.name]))
+ | CaseInsensitiveSet([leader]))
+ logger.info("Setting leader to %s, quorum to %d of (%s)",
+ self.state_handler.name, sync.quorum, ", ".join(sorted(voters)))
+ sync = self.dcs.write_sync_state(self.state_handler.name, voters, sync.quorum,
+ version=sync.version)
+ if not sync:
+ return logger.info('Synchronous replication key updated by someone else.')
+ leader = self.state_handler.name
transition = 'restart'
transitions = []
else:
@@ -979,7 +994,8 @@ def _check_timeout(offset: float = 0) -> bool:
numsync_confirmed=len(sync_state.sync_confirmed),
active=sync_state.active,
sync_wanted=global_config.synchronous_node_count,
- leader_wanted=self.state_handler.name))
+ leader_wanted=self.state_handler.name,
+ quorum_catchup=sync_state.quorum_catchup))
for i, (transition, leader, num, nodes) in enumerate(transitions):
if _check_timeout():
@@ -993,7 +1009,7 @@ def _check_timeout(offset: float = 0) -> bool:
elif transition == 'sync':
if self.state_handler.sync_handler.set_synchronous_standby_names(nodes, num) and \
i < len(transitions) - 1:
- # If synchronous_standby_names change is not yet effective
+ # If synchronous_standby_names change is not yet effective (or remaining nodes must catch up)
# and this is not the last transition we need to wait and restart the state machine.
transition = 'restart'
break
@@ -1007,12 +1023,13 @@ def _check_timeout(offset: float = 0) -> bool:
if transition == 'break' and sync_state:
self.state_handler.slots_handler.handle_manage_sync_slots_toggle(sync_state.active)
return
- # Here we consider two cases:
+ # Here we consider three cases:
# 1. synchronous_standby_names was transitioned from empty to non-empty and it may take
# some time for checkpointer to enforce it and for nodes to become synchronous.
# 2. some nodes were removed from synchronous_standby_names and we need to wait until
# this change becomes effective after reload.
- # In both cases we need to wait and restart state machine hoping that we can update
+ # 3. quorum became stricter and remaining nodes must catch up with the primary.
+ # In all cases we need to wait and restart state machine hoping that we can update
# /sync key earlier than in loop_wait seconds.
time.sleep(1)
self.state_handler.reset_cluster_info_state(None)
diff --git a/patroni/postgresql/sync.py b/patroni/postgresql/sync.py
index 1ed21ad625..afaf279aaa 100644
--- a/patroni/postgresql/sync.py
+++ b/patroni/postgresql/sync.py
@@ -184,12 +184,14 @@ class _SyncState(NamedTuple):
confirmed to be synchronous according to the ``pg_stat_replication`` view.
Only nodes that caught up with the :attr:`SyncHandler._primary_flush_lsn` are counted.
:ivar active: collection of node names that are streaming and have no restrictions to become synchronous.
+ :ivar quorum_catchup: whether only conservative quorum membership additions are allowed until replicas catch up.
"""
sync_type: str
numsync: int
sync: CaseInsensitiveSet
sync_confirmed: CaseInsensitiveSet
active: CaseInsensitiveSet
+ quorum_catchup: bool = False
class _Replica(NamedTuple):
@@ -324,6 +326,8 @@ def reset_state(self) -> None:
self._checkpoint_retry_at = 0.0
# value that checkpointer was verified to enforce by wait_for_checkpointer() right before promote
self._checkpointer_verified = ''
+ # quorum became stricter and remaining members must catch up before they could be counted as synchronous
+ self._quorum_catchup = False
@property
def checkpointer_pending(self) -> bool:
@@ -386,8 +390,16 @@ def _handle_synchronous_standby_names_change(self) -> Optional[bool]:
# It takes some time before synchronous_standby_names names change propagates via checkpointer.
# Only previously verified members can remain active while the new value is pending.
- self._set_active_synchronous_standby_names_members(
- CaseInsensitiveSet(self._active_synchronous_standby_names_members & ssn_data.members))
+ active = CaseInsensitiveSet(self._active_synchronous_standby_names_members & ssn_data.members)
+ if not global_config.is_quorum_commit_mode or not ssn_data.members or ssn_data.has_star:
+ self._quorum_catchup = False
+ elif not self._ssn_data.members or not self._ssn_data.has_star \
+ and (self._ssn_data.members - ssn_data.members or ssn_data.num > self._ssn_data.num):
+ # After restart the previous value is unknown. Transactions might exist only on removed nodes
+ # or on fewer than num nodes, so verify catch-up before relaxing the election quorum.
+ self._quorum_catchup = True
+ active = CaseInsensitiveSet()
+ self._set_active_synchronous_standby_names_members(active)
self._ssn_data = ssn_data
self._synchronous_standby_names_pending = bool(ssn_data.members)
@@ -403,9 +415,16 @@ def _handle_synchronous_standby_names_change(self) -> Optional[bool]:
if self._ssn_data.sync_type == 'priority' and row['sync_state'] in ('potential', 'sync')
or self._ssn_data.sync_type == 'quorum' and row['sync_state'] == 'quorum'])
+ members = self._ssn_data.members
+ if self._quorum_catchup:
+ # Members that are not streaming can't acknowledge commits and will use the new value after (re)connect.
+ # Don't wait for them, otherwise remaining nodes would never be checked against _primary_flush_lsn.
+ streaming = CaseInsensitiveSet([row['application_name'] for row in pg_stat_replication])
+ members = CaseInsensitiveSet(members & streaming)
+
# Nodes listed in synchronous_standby_names should have corresponding sync_state in pg_stat_replication.
# Only from this moment we count synchronous_standby_names effectively applied.
- if replication != self._ssn_data.members:
+ if replication != members:
# If some (not ALL) nodes were removed from synchronous_standby_names we need to notify caller
# that it has to wait until synchronous_standby_names change becomes active after reload.
return bool(self._ssn_data.members and not self._ssn_data.has_star
@@ -607,7 +626,8 @@ def current_state(self, cluster: Cluster) -> Optional[_SyncState]:
:param cluster: current cluster topology from DCS
:returns: current synchronous replication state as a :class:`_SyncState` object or `None` in case if we detect
- that some nodes were removed from synchronous_standby_names and change is not yet active after reload.
+ that some nodes were removed from synchronous_standby_names and change is not yet active after reload,
+ or while application of a stricter quorum setting is still being verified.
"""
if self._handle_synchronous_standby_names_change():
return None
@@ -639,11 +659,24 @@ def current_state(self, cluster: Cluster) -> Optional[_SyncState]:
if len(active) >= sync_node_count:
break
+ # Preserve the election quorum while catching up, but expose active candidates so that the HA loop can
+ # admit replacements without relaxing the existing election guarantees.
+ quorum_catchup = False
+ if not global_config.is_quorum_commit_mode:
+ self._quorum_catchup = False
+ elif self._quorum_catchup:
+ if self._synchronous_standby_names_pending:
+ return None
+ if len(sync_confirmed) >= min(self._ssn_data.num, sync_node_count):
+ self._quorum_catchup = False
+ elif (active & self._ssn_data.members) - sync_confirmed:
+ quorum_catchup = True
+
# We need to handle old case of synchronous_standby_names='*' and when Patroni is restarted while PostgreSQL
# is running and replace it with __patroni_strict_sync_replica_placeholder__. It requires having non-empty
# value of _SyncState.sync which doesn't match any exiting cluster members.
sync = CaseInsensitiveSet([SYNC_STRICT_PLACEHOLDER]) if self._ssn_data.has_star else self._ssn_data.members
- return _SyncState(self._ssn_data.sync_type, self._ssn_data.num, sync, sync_confirmed, active)
+ return _SyncState(self._ssn_data.sync_type, self._ssn_data.num, sync, sync_confirmed, active, quorum_catchup)
def set_synchronous_standby_names(self, sync: Collection[str], num: Optional[int] = None) -> Optional[bool]:
"""Constructs and sets ``synchronous_standby_names`` GUC value.
@@ -655,7 +688,8 @@ def set_synchronous_standby_names(self, sync: Collection[str], num: Optional[int
:param num: specifies number of nodes to sync to. The *num* is set only in case if quorum commit is enabled
:returns: `True` if caller needs to wait until synchronous_standby_names change becomes active after reload
- (including enforcement by checkpointer).
+ (including enforcement by checkpointer), or until remaining nodes caught up after quorum became
+ stricter.
"""
# Update synchronized_standby_slots in _server_parameters first (no reload yet) so
# that the subsequent set_synchronous_standby_names() reload picks up both changes.
@@ -709,4 +743,4 @@ def set_synchronous_standby_names(self, sync: Collection[str], num: Optional[int
if self._postgresql.get_primary_timeline() > 0:
# Reload could take longer than 100ms, in this case the new value is still not visible.
return self._handle_synchronous_standby_names_change() or self._checkpoint_required\
- or self._synchronous_standby_names != (sync_param or '')
+ or self._synchronous_standby_names != (sync_param or '') or self._quorum_catchup
diff --git a/patroni/quorum.py b/patroni/quorum.py
index 4677d45fc1..41682a154d 100644
--- a/patroni/quorum.py
+++ b/patroni/quorum.py
@@ -106,11 +106,12 @@ class QuorumStateResolver:
and are eligible to be listed in ``synchronous_standby_names``.
:ivar sync_wanted: desired number of synchronous nodes (``synchronous_node_count`` from the global configuration).
:ivar leader_wanted: the desired leader (could be different from the :attr:`leader` right after a failover).
+ :ivar quorum_catchup: whether only conservative membership additions are allowed until replicas catch up.
"""
def __init__(self, leader: str, quorum: int, voters: Collection[str],
numsync: int, sync: Collection[str], numsync_confirmed: int,
- active: Collection[str], sync_wanted: int, leader_wanted: str) -> None:
+ active: Collection[str], sync_wanted: int, leader_wanted: str, quorum_catchup: bool = False) -> None:
"""Instantiate :class:``QuorumStateResolver`` based on input parameters.
:param leader: name of the leader, according to the ``/sync`` key.
@@ -127,17 +128,20 @@ def __init__(self, leader: str, quorum: int, voters: Collection[str],
:param sync_wanted: desired number of synchronous nodes
(``synchronous_node_count`` from the global configuration).
:param leader_wanted: the desired leader (could be different from the *leader* right after a failover).
+ :param quorum_catchup: whether historical durability is still being verified after restart or a stricter SSN.
"""
- self.leader = leader
+ self.leader: str = leader
self.quorum = quorum
self.voters = CaseInsensitiveSet(voters)
- self.numsync = min(numsync, len(sync)) # numsync can't be bigger than number of listed synchronous nodes.
+ # While catch-up is pending, preserve the actual setting even if there are too few listed nodes.
+ self.numsync = numsync if quorum_catchup else min(numsync, len(sync))
self.sync = CaseInsensitiveSet(sync)
self.numsync_confirmed = numsync_confirmed
self.active = CaseInsensitiveSet(active)
self.sync_wanted = sync_wanted
self.leader_wanted = leader_wanted
+ self.quorum_catchup = quorum_catchup
def check_invariants(self) -> None:
"""Checks invariant of ``synchronous_standby_names`` and ``/sync`` key in DCS.
@@ -397,15 +401,44 @@ def __handle_replication_factor_change(self) -> Iterator[Transition]:
adjust_quorum=self.sync_wanted > self.numsync_confirmed)
yield from self.sync_update(self.numsync + sync_increase, self.sync)
+ def __handle_quorum_catchup(self) -> Iterator[Transition]:
+ """Grow membership without weakening the existing election requirement.
+
+ Bypass readiness adjustments and invariant repair while historical durability is unverified.
+ A quorum that temporarily prevents all elections must stay in place until catch-up completes.
+ """
+ if self.leader_wanted != self.leader: # failover
+ # The former leader takes the place of the new one in voters, therefore the set of nodes that could
+ # vote and the quorum don't change. But the former leader must lose the privilege to be promoted without
+ # votes, because it might not have commits acknowledged after the failover.
+ self.voters = CaseInsensitiveSet((self.voters - CaseInsensitiveSet([self.leader_wanted]))
+ | CaseInsensitiveSet([self.leader]))
+ self.leader = self.leader_wanted
+ yield Transition('quorum', self.leader, self.quorum, self.voters)
+ added = CaseInsensitiveSet(self.active - self.sync)
+ added_voters = CaseInsensitiveSet(added - self.voters)
+ added_voters.discard(self.leader) # The DCS leader is already an implicit member.
+ if added_voters:
+ self.quorum += len(added_voters)
+ self.voters = CaseInsensitiveSet(self.voters | added_voters)
+ yield Transition('quorum', self.leader, self.quorum, self.voters)
+ if added:
+ self.sync = CaseInsensitiveSet(self.sync | added)
+ yield Transition('sync', self.leader, self.numsync, self.sync)
+ yield Transition('restart', self.leader, self.quorum, self.voters)
+
def _generate_transitions(self) -> Iterator[Transition]:
"""Produce a set of changes to safely transition from the current state to the desired.
:yields: transitions as :class:`Transition` objects.
"""
logger.debug("Quorum state: leader %s quorum %s, voters %s, numsync %s, sync %s, "
- "numsync_confirmed %s, active %s, sync_wanted %s leader_wanted %s",
+ "numsync_confirmed %s, active %s, sync_wanted %s leader_wanted %s quorum_catchup %s",
self.leader, self.quorum, self.voters, self.numsync, self.sync,
- self.numsync_confirmed, self.active, self.sync_wanted, self.leader_wanted)
+ self.numsync_confirmed, self.active, self.sync_wanted, self.leader_wanted, self.quorum_catchup)
+ if self.quorum_catchup:
+ yield from self.__handle_quorum_catchup()
+ return
try:
if self.leader_wanted != self.leader: # failover
voters = (self.voters - CaseInsensitiveSet([self.leader_wanted])) | CaseInsensitiveSet([self.leader])
diff --git a/tests/test_ha.py b/tests/test_ha.py
index 11e8cae3a0..9b7958aa66 100644
--- a/tests/test_ha.py
+++ b/tests/test_ha.py
@@ -2467,6 +2467,92 @@ def test__process_quorum_replication(self):
self.assertEqual(mock_set_sync.call_count, 1)
self.assertEqual(mock_set_sync.call_args_list[0][0], ('ANY 1 (a)',))
+ def test_quorum_catchup_adds_replacement(self):
+ self.p.name = 'leader'
+ self.ha.cluster = get_cluster_initialized_with_leader(sync=('leader', 'a,b', 1))
+ pending = _SyncState('quorum', 1, CaseInsensitiveSet('a'), CaseInsensitiveSet(),
+ CaseInsensitiveSet('ac'), True)
+ ready = _SyncState('quorum', 1, CaseInsensitiveSet('ac'), CaseInsensitiveSet('c'),
+ CaseInsensitiveSet('ac'))
+ self.p.sync_handler.current_state = Mock(side_effect=[pending, ready])
+ write_sync = self.ha.dcs.write_sync_state = Mock(
+ side_effect=[SyncState(1, 'leader', 'a,b,c', 2), SyncState(2, 'leader', 'a,c', 1)])
+
+ def set_sync(nodes, num):
+ self.assertEqual(write_sync.call_count, 1)
+ self.assertEqual(write_sync.call_args[0], ('leader', CaseInsensitiveSet('abc'), 2))
+ self.assertEqual((nodes, num), (CaseInsensitiveSet('ac'), 1))
+ return True
+
+ set_sync_mock = self.p.sync_handler.set_synchronous_standby_names = Mock(side_effect=set_sync)
+ self.ha._process_quorum_replication()
+ self.assertEqual(write_sync.call_count, 2)
+ self.assertEqual(write_sync.call_args[0], ('leader', CaseInsensitiveSet('ac'), 1))
+ self.assertEqual(write_sync.call_args[1], {'version': 1})
+ set_sync_mock.assert_called_once()
+
+ # A failed DCS update must not let the newcomer acknowledge commits.
+ self.p.sync_handler.current_state = Mock(return_value=pending)
+ self.ha.dcs.write_sync_state = Mock(return_value=None)
+ set_sync_mock.reset_mock()
+ self.ha._process_quorum_replication()
+ set_sync_mock.assert_not_called()
+
+ def test_quorum_catchup_changes_leader(self):
+ self.p.name = 'newleader'
+ self.ha.cluster = get_cluster_initialized_with_leader(sync=('leader', 'a,newleader', 1))
+ self.p.sync_handler.current_state = Mock(return_value=_SyncState(
+ 'quorum', 1, CaseInsensitiveSet('a'), CaseInsensitiveSet(), CaseInsensitiveSet(['a', 'leader']), True))
+ write_sync = self.ha.dcs.write_sync_state = Mock()
+ set_sync = self.p.sync_handler.set_synchronous_standby_names = Mock(return_value=True)
+ with patch('time.monotonic', side_effect=[0, 0, 0, 100]):
+ self.ha._process_quorum_replication()
+ # the former leader must not keep the privilege to be promoted without votes
+ write_sync.assert_called_once()
+ self.assertEqual(write_sync.call_args[0], ('newleader', CaseInsensitiveSet(['a', 'leader']), 1))
+ set_sync.assert_called_once_with(CaseInsensitiveSet(['a', 'leader']), 1)
+
+ def test_quorum_unverified_changes_leader(self):
+ self.p.name = 'newleader'
+ self.ha.cluster = get_cluster_initialized_with_leader(sync=('leader', 'a,newleader', 1))
+ self.p.sync_handler.current_state = Mock(return_value=None)
+ write_sync = self.ha.dcs.write_sync_state = Mock(return_value=SyncState(1, 'newleader', 'a,leader', 1))
+ with patch('time.monotonic', side_effect=[0, 0, 100]):
+ self.ha._process_quorum_replication()
+ # the former leader must lose the privilege to be promoted without votes before verification completes
+ write_sync.assert_called_once_with('newleader', CaseInsensitiveSet(['a', 'leader']), 1, version=0)
+
+ write_sync = self.ha.dcs.write_sync_state = Mock(return_value=None)
+ with patch('time.monotonic', side_effect=[0, 100]):
+ self.ha._process_quorum_replication()
+ write_sync.assert_called_once()
+
+ @patch.object(global_config.__class__, 'is_quorum_commit_mode', PropertyMock(return_value=True))
+ def test_quorum_catchup_preserves_voters_after_restart(self):
+ self.p._major_version = 150000
+ self.p.name = 'leader'
+ self.ha.cluster = get_cluster_initialized_with_leader(sync=('leader', 'a,b,c', 2))
+ self.ha.cluster.members.extend(Member(0, name, 1, {'state': 'running'}) for name in 'ab')
+ replication = [{'pid': i + 100, 'application_name': name, 'sync_state': 'quorum', 'flush_lsn': 100}
+ for i, name in enumerate('ab')]
+ write_sync = self.ha.dcs.write_sync_state = Mock(return_value=SyncState(1, 'leader', 'a,b', 1))
+ set_sync = self.p.sync_handler.set_synchronous_standby_names = Mock()
+ with patch.object(Postgresql, 'synchronous_standby_names', return_value='ANY 1 (a,b)'), \
+ patch.object(Postgresql, 'synchronous_commit', return_value='on'), \
+ patch.object(Postgresql, 'pg_stat_replication', return_value=replication), \
+ patch.object(Postgresql, 'last_operation', return_value=200), \
+ patch.object(Postgresql, 'query', return_value=[(True, True)]), \
+ patch('time.monotonic', side_effect=range(100)):
+ self.ha._process_quorum_replication()
+ write_sync.assert_not_called()
+ replication[0]['flush_lsn'] = 200
+ state = self.p.sync_handler.current_state(self.ha.cluster)
+ self.assertFalse(state.quorum_catchup)
+ self.assertEqual(state.sync_confirmed, CaseInsensitiveSet('a'))
+ self.ha._process_quorum_replication()
+ write_sync.assert_called_once_with('leader', CaseInsensitiveSet('ab'), 1, version=0)
+ set_sync.assert_not_called()
+
def test_is_failover_possible(self):
self.p._major_version = 140000 # supports_multiple_sync
self.p.name = 'leader'
diff --git a/tests/test_quorum.py b/tests/test_quorum.py
index 00f6578347..45379fa92f 100644
--- a/tests/test_quorum.py
+++ b/tests/test_quorum.py
@@ -17,12 +17,13 @@ def tearDown(self):
def check_state_transitions(self, leader: str, quorum: int, voters: Set[str], numsync: int, sync: Set[str],
numsync_confirmed: int, active: Set[str], sync_wanted: int, leader_wanted: str,
- expected: List[Tuple[str, str, int, Set[str]]]) -> None:
+ expected: List[Tuple[str, str, int, Set[str]]], quorum_catchup: bool = False) -> None:
self.nesting += 1
kwargs = {
'leader': leader, 'quorum': quorum, 'voters': voters,
'numsync': numsync, 'sync': sync, 'numsync_confirmed': numsync_confirmed,
- 'active': active, 'sync_wanted': sync_wanted, 'leader_wanted': leader_wanted
+ 'active': active, 'sync_wanted': sync_wanted, 'leader_wanted': leader_wanted,
+ 'quorum_catchup': quorum_catchup
}
state = {k: v for k, v in kwargs.items()}
result = list(QuorumStateResolver(**kwargs))
@@ -580,3 +581,52 @@ def test_reduce_replication_factor_before_new_node_confirmed(self):
('quorum', leader, 2, set('bcd')), # increase quorum before reducing replication factor
('sync', leader, 1, set('bcd')), # reduce replication factor
])
+
+ def test_quorum_catchup_preserves_election_guarantees(self):
+ self.check_state_transitions(leader='a', quorum=2, voters=set('bcd'),
+ numsync=1, sync=set('bc'), numsync_confirmed=0, active=set('bc'),
+ sync_wanted=1, leader_wanted='a', quorum_catchup=True, expected=[
+ ('restart', 'a', 2, set('bcd')),
+ ])
+ # A mismatched state must not be repaired using unverified durability.
+ self.check_state_transitions(leader='a', quorum=1, voters=set('bc'),
+ numsync=1, sync=set('bd'), numsync_confirmed=0, active=set('bd'),
+ sync_wanted=1, leader_wanted='a', quorum_catchup=True, expected=[
+ ('restart', 'a', 1, set('bc')),
+ ])
+
+ def test_quorum_catchup_adds_replacements(self):
+ for numsync in (1, 2):
+ with self.subTest(numsync=numsync):
+ self.check_state_transitions(leader='a', quorum=1, voters=set('bc'),
+ numsync=numsync, sync=set('b'), numsync_confirmed=0, active=set('bd'),
+ sync_wanted=2, leader_wanted='a', quorum_catchup=True, expected=[
+ ('quorum', 'a', 2, set('bcd')),
+ ('sync', 'a', numsync, set('bd')),
+ ('restart', 'a', 2, set('bcd')),
+ ])
+ self.check_state_transitions(leader='a', quorum=2, voters=set('bcd'),
+ numsync=1, sync=set('b'), numsync_confirmed=0, active=set('bd'),
+ sync_wanted=1, leader_wanted='a', quorum_catchup=True, expected=[
+ ('sync', 'a', 1, set('bd')),
+ ('restart', 'a', 2, set('bcd')),
+ ])
+
+ def test_quorum_catchup_with_empty_voters(self):
+ self.check_state_transitions(leader='a', quorum=0, voters=set(),
+ numsync=1, sync=set('b'), numsync_confirmed=0, active=set('bd'),
+ sync_wanted=1, leader_wanted='a', quorum_catchup=True, expected=[
+ ('quorum', 'a', 1, set('d')),
+ ('sync', 'a', 1, set('bd')),
+ ('restart', 'a', 1, set('d')),
+ ])
+
+ def test_quorum_catchup_changes_leader(self):
+ # the former leader replaces the new one in voters, quorum is preserved
+ self.check_state_transitions(leader='a', quorum=1, voters=set('bc'),
+ numsync=1, sync=set('b'), numsync_confirmed=0, active=set('ab'),
+ sync_wanted=1, leader_wanted='c', quorum_catchup=True, expected=[
+ ('quorum', 'c', 1, set('ab')),
+ ('sync', 'c', 1, set('ab')),
+ ('restart', 'c', 1, set('ab')),
+ ])
diff --git a/tests/test_sync.py b/tests/test_sync.py
index 222d550502..6671585d13 100644
--- a/tests/test_sync.py
+++ b/tests/test_sync.py
@@ -49,7 +49,7 @@ def test_pick_sync_standby(self):
self.assertEqual(self.s.current_state(self.cluster), ('priority', 1,
CaseInsensitiveSet([self.leadermem.name]),
CaseInsensitiveSet([self.leadermem.name]),
- CaseInsensitiveSet([self.leadermem.name])))
+ CaseInsensitiveSet([self.leadermem.name]), False))
# prefer node with sync_state='potential', even if it is slightly behind of async
pg_stat_replication[0]['sync_state'] = 'potential'
@@ -57,7 +57,7 @@ def test_pick_sync_standby(self):
r['write_lsn'] = r.pop('flush_lsn')
with patch.object(Postgresql, "_cluster_info_state_get", side_effect=['', 'remote_write', pg_stat_replication]):
self.assertEqual(self.s.current_state(self.cluster), ('off', 0, CaseInsensitiveSet(), CaseInsensitiveSet(),
- CaseInsensitiveSet([self.leadermem.name])))
+ CaseInsensitiveSet([self.leadermem.name]), False))
# when there are no sync or potential candidates we pick async with the minimal replication lag
for i, r in enumerate(pg_stat_replication):
@@ -65,20 +65,20 @@ def test_pick_sync_standby(self):
missing = pg_stat_replication.pop(0)
with patch.object(Postgresql, "_cluster_info_state_get", side_effect=['', 'remote_apply', pg_stat_replication]):
self.assertEqual(self.s.current_state(self.cluster), ('off', 0, CaseInsensitiveSet(), CaseInsensitiveSet(),
- CaseInsensitiveSet([self.me.name])))
+ CaseInsensitiveSet([self.me.name]), False))
# unknown sync node is ignored
missing.update(application_name='missing', sync_state='sync')
pg_stat_replication.insert(0, missing)
with patch.object(Postgresql, "_cluster_info_state_get", side_effect=['', 'remote_apply', pg_stat_replication]):
self.assertEqual(self.s.current_state(self.cluster), ('off', 0, CaseInsensitiveSet(), CaseInsensitiveSet(),
- CaseInsensitiveSet([self.me.name])))
+ CaseInsensitiveSet([self.me.name]), False))
# invalid synchronous_standby_names and empty pg_stat_replication
with patch.object(Postgresql, "_cluster_info_state_get", side_effect=['a b', 'remote_apply', None]):
self.p._major_version = 90400
self.assertEqual(self.s.current_state(self.cluster), ('off', 0, CaseInsensitiveSet(), CaseInsensitiveSet(),
- CaseInsensitiveSet()))
+ CaseInsensitiveSet(), False))
# synchronous_standby_names contains '*'
with patch.object(Postgresql, "_cluster_info_state_get", side_effect=['*', None, 'remote_apply', None]):
@@ -86,7 +86,7 @@ def test_pick_sync_standby(self):
self.assertEqual(self.s.current_state(self.cluster),
('priority', 1,
CaseInsensitiveSet(['__patroni_strict_sync_replica_placeholder__']),
- CaseInsensitiveSet(), CaseInsensitiveSet()))
+ CaseInsensitiveSet(), CaseInsensitiveSet(), False))
@patch.object(Postgresql, 'last_operation', Mock(return_value=1))
def test_current_state_quorum(self):
@@ -104,7 +104,7 @@ def test_current_state_quorum(self):
self.assertEqual(self.s.current_state(self.cluster),
('quorum', 1, CaseInsensitiveSet([self.me.name, self.leadermem.name]),
CaseInsensitiveSet([self.me.name, self.leadermem.name]),
- CaseInsensitiveSet([self.leadermem.name, self.me.name])))
+ CaseInsensitiveSet([self.leadermem.name, self.me.name]), False))
@patch('time.sleep', Mock())
def test_current_state_after_repromote(self):
@@ -142,7 +142,7 @@ def test_current_state_cascading(self):
pg_stat_replication, 'on', pg_stat_replication]):
self.assertEqual(self.s.current_state(self.cluster),
('priority', 2, CaseInsensitiveSet([self.other.name, self.leadermem.name]),
- CaseInsensitiveSet(), CaseInsensitiveSet([self.me.name])))
+ CaseInsensitiveSet(), CaseInsensitiveSet([self.me.name]), False))
def test_current_state_waiting(self):
pg_stat_replication = [
@@ -482,6 +482,108 @@ def test_set_sync_standby_slow_reload(self):
self.assertFalse(self.s.set_synchronous_standby_names([a, b]))
self.assertIsNone(self.s.set_synchronous_standby_names([a, b]))
+ @patch('time.sleep', Mock())
+ @patch.object(Postgresql, 'is_primary', Mock(return_value=True))
+ @patch.object(Postgresql, 'get_primary_timeline', Mock(return_value=1))
+ @patch.object(Postgresql, 'state', PropertyMock(return_value=PostgresqlState.RUNNING))
+ def test_current_state_quorum_catchup(self):
+ self.cluster.config.data['synchronous_mode'] = 'quorum'
+ global_config.update(self.cluster)
+ self.other.data['tags'] = {}
+ a, b, c = self.leadermem.name, self.me.name, self.other.name
+ replication = [{'pid': 100 + i, 'application_name': name, 'sync_state': 'quorum', 'flush_lsn': 100}
+ for i, name in enumerate((a, b, c))]
+
+ with patch.object(Postgresql, 'synchronous_standby_names',
+ return_value='ANY 1 ({0},{1},"{2}")'.format(a, b, c)) as mock_ssn, \
+ patch.object(Postgresql, 'pg_stat_replication', return_value=replication), \
+ patch.object(Postgresql, 'last_operation', return_value=100) as mock_lsn, \
+ patch.object(ConfigHandler, 'set_synchronous_standby_names', Mock(return_value=True)):
+ self.assertEqual(self.s.current_state(self.cluster).sync_confirmed, CaseInsensitiveSet([a, b, c]))
+
+ # c was removed, transactions acknowledged only by c must reach one of remaining nodes
+ mock_ssn.return_value = 'ANY 1 ({0},{1})'.format(a, b)
+ mock_lsn.return_value = 200
+ replication[2]['sync_state'] = 'async'
+ self.assertTrue(self.s.set_synchronous_standby_names([a, b], 1))
+ self.assertTrue(self.s.current_state(self.cluster).quorum_catchup)
+ replication[1]['flush_lsn'] = 200
+ self.assertEqual(self.s.current_state(self.cluster).sync_confirmed, CaseInsensitiveSet([b]))
+
+ # num was increased, transactions acknowledged by one node must reach two nodes
+ self.cluster.config.data['synchronous_node_count'] = 2
+ global_config.update(self.cluster)
+ mock_ssn.return_value = 'ANY 2 ({0},{1})'.format(a, b)
+ mock_lsn.return_value = 300
+ replication[0]['flush_lsn'] = 300
+ self.assertTrue(self.s.current_state(self.cluster).quorum_catchup)
+
+ # don't wait when there are not enough eligible nodes and all of them caught up
+ self.cluster.config.data['maximum_lag_on_syncnode'] = 50
+ global_config.update(self.cluster)
+ self.assertEqual(self.s.current_state(self.cluster).sync_confirmed, CaseInsensitiveSet([a]))
+ self.cluster.config.data['maximum_lag_on_syncnode'] = -1
+ global_config.update(self.cluster)
+ self.assertTrue(self.s.current_state(self.cluster).quorum_catchup)
+
+ # synchronous_node_count was reduced, don't wait for more nodes than needed
+ self.cluster.config.data['synchronous_node_count'] = 1
+ global_config.update(self.cluster)
+ self.assertEqual(self.s.current_state(self.cluster).sync_confirmed, CaseInsensitiveSet([a]))
+ replication[1]['flush_lsn'] = 300
+ self.assertEqual(self.s.current_state(self.cluster).sync_confirmed, CaseInsensitiveSet([a, b]))
+
+ def test_current_state_quorum_catchup_degraded(self):
+ self.cluster.config.data.update(synchronous_mode='quorum', synchronous_node_count=2)
+ global_config.update(self.cluster)
+ self.other.data['tags'] = {}
+ a, b, c = self.leadermem.name, self.me.name, self.other.name
+ replication = [{'pid': 100 + i, 'application_name': name, 'sync_state': 'quorum', 'flush_lsn': 100}
+ for i, name in enumerate((a, b, c))]
+
+ with patch.object(Postgresql, 'synchronous_standby_names',
+ return_value='ANY 2 ({0},{1},"{2}")'.format(a, b, c)) as mock_ssn, \
+ patch.object(Postgresql, 'pg_stat_replication', return_value=replication), \
+ patch.object(Postgresql, 'last_operation', return_value=100) as mock_lsn:
+ self.assertEqual(self.s.current_state(self.cluster).sync_confirmed, CaseInsensitiveSet([a, b, c]))
+
+ # c was removed and b disconnected, transactions acknowledged by b and c must reach a
+ mock_ssn.return_value = 'ANY 2 ({0},{1})'.format(a, b)
+ mock_lsn.return_value = 200
+ replication[2].update(sync_state='async', flush_lsn=200)
+ del replication[1]
+ self.assertTrue(self.s.current_state(self.cluster).quorum_catchup)
+ replication[0]['flush_lsn'] = 200
+ self.assertEqual(self.s.current_state(self.cluster).sync_confirmed, CaseInsensitiveSet([a]))
+
+ # b was replaced with c, but c is not yet reporting the new sync_state
+ mock_ssn.return_value = 'ANY 1 ({0},"{1}")'.format(a, c)
+ mock_lsn.return_value = 300
+ self.assertIsNone(self.s.current_state(self.cluster))
+ replication[1]['sync_state'] = 'quorum'
+ self.assertTrue(self.s.current_state(self.cluster).quorum_catchup)
+ replication[0]['flush_lsn'] = 300
+ self.assertEqual(self.s.current_state(self.cluster).sync_confirmed, CaseInsensitiveSet([a]))
+
+ def test_quorum_catchup_after_reset(self):
+ self.cluster.config.data['synchronous_mode'] = 'quorum'
+ global_config.update(self.cluster)
+ a = self.leadermem.name
+ replication = [{'pid': 100, 'application_name': a, 'sync_state': 'quorum', 'flush_lsn': 100}]
+ with patch.object(Postgresql, 'synchronous_standby_names', return_value='ANY 1 ({0})'.format(a)), \
+ patch.object(Postgresql, 'pg_stat_replication', return_value=replication), \
+ patch.object(Postgresql, 'last_operation', return_value=200) as mock_lsn:
+ for target in (200, 300):
+ mock_lsn.return_value = target
+ self.s.reset_state()
+ state = self.s.current_state(self.cluster)
+ self.assertTrue(state.quorum_catchup)
+ self.assertFalse(state.sync_confirmed)
+ replication[0]['flush_lsn'] = target
+ state = self.s.current_state(self.cluster)
+ self.assertFalse(state.quorum_catchup)
+ self.assertEqual(state.sync_confirmed, CaseInsensitiveSet([a]))
+
@patch('time.sleep', Mock())
def test_set_sync_standby(self):
def value_in_conf():
@@ -654,4 +756,5 @@ def test_do_not_prick_yourself(self):
with patch.object(Postgresql, "_cluster_info_state_get", side_effect=[self.leadermem.name, pg_stat_replication,
'on', pg_stat_replication]):
self.assertEqual(self.s.current_state(cluster), ('priority', 1, CaseInsensitiveSet([self.leadermem.name]),
- CaseInsensitiveSet(), CaseInsensitiveSet([self.me.name])))
+ CaseInsensitiveSet(), CaseInsensitiveSet([self.me.name]),
+ False))
^ permalink raw [nested|flat] 4+ messages in thread
* Re: [patroni/patroni] PR #3817: Make remaining quorum members catch up when quorum becomes stricter
@ 2026-11-19 21:59 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2 siblings, 0 replies; 4+ messages in thread
From: CyberDem0n (@CyberDem0n) @ 2026-11-19 21:59 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
@CyberDem0n applied label "bug".
^ permalink raw [nested|flat] 4+ messages in thread
* Re: [patroni/patroni] PR #3817: Make remaining quorum members catch up when quorum becomes stricter
@ 2026-11-19 21:59 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2 siblings, 0 replies; 4+ messages in thread
From: CyberDem0n (@CyberDem0n) @ 2026-11-19 21:59 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
@CyberDem0n requested a review.
^ permalink raw [nested|flat] 4+ messages in thread
* Re: [patroni/patroni] PR #3817: Make remaining quorum members catch up when quorum becomes stricter
@ 2026-11-19 22:21 ` "codecov[bot] (bot) (@codecov[bot])" <noreply+codecov[bot]@github.com>
2 siblings, 0 replies; 4+ messages in thread
From: codecov[bot] (bot) (@codecov[bot]) @ 2026-11-19 22:21 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
## [Codecov](https://app.codecov.io/gh/patroni/patroni/pull/3817?dropdown=coverage&src=pr&el=h1&utm_m...) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:loudspeaker: Thoughts on this report? [Let us know!](https://github.com/codecov/feedback/issues/255)
^ permalink raw [nested|flat] 4+ messages in thread
end of thread, other threads:[~2026-11-19 22:21 UTC | newest]
Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-11-19 21:59 [patroni/patroni] PR #3817: Make remaining quorum members catch up when quorum becomes stricter "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-19 21:59 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-19 21:59 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-19 22:21 ` "codecov[bot] (bot) (@codecov[bot])" <noreply+codecov[bot]@github.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox