pg.ddx.io  patroni/patroni GitHub issues and pull requests  
help / color / mirror / Atom feed
[patroni/patroni] PR #3815: Verify that walsenders apply synchronous_standby_names before updating /sync
4+ messages / 2 participants
[nested] [flat]

* [patroni/patroni] PR #3815: Verify that walsenders apply synchronous_standby_names before updating /sync
@ 2026-11-19 21:59 "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
  0 siblings, 0 replies; 4+ messages in thread

From: CyberDem0n (@CyberDem0n) @ 2026-11-19 21:59 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

Previously, after updating `synchronous_standby_names` and reloading, we:
1. waited until `SHOW synchronous_standby_names` reported the new value;
2. memorized the value returned by `pg_current_wal_insert_lsn()`;
3. waited until standbys from `synchronous_standby_names` confirmed write/flush/apply LSN from [2].

This is not enough, because `SHOW synchronous_standby_names` in our connection doesn't mean that PostgreSQL already enforces the new value. Updating `/sync` too early may allow a failover to a node that doesn't have all acknowledged transactions.

This PR makes Patroni consider the change applied only when **`pg_stat_replication` matches**: the set of nodes reported as `sync`/`potential`/`quorum` must match the members of `synchronous_standby_names`. Walsenders report the new `sync_state` as soon as they process the reload. Only after that is the primary flush LSN that new members must reach recorded.
- Nodes removed from `synchronous_standby_names` are tracked separately, so removals can't bypass this verification: `current_state()` returns `None`, and callers wait (the quorum state machine restarts) instead of updating `/sync`.
- Only previously verified members stay active across successive changes, so replicas are never confirmed against a stale WAL position.
- If backends were already waiting for synchronous replication, some WAL is generated right after the change, so that walsenders process feedback from standbys and release them without waiting for `wal_receiver_status_interval`.
- A change of only `N` in `ANY N (...)` isn't visible in `pg_stat_replication`. Walsenders process the reload before handling the next replies from standbys, therefore the window when the old `N` is used is very short.

**A slow reload is noticed.** If the reload took longer than 100ms, `set_synchronous_standby_names()` used to return without asking the caller to wait, and subsequent calls returned early because `postgresql.conf` was already up to date. Now the caller waits until the new value is visible.

Also fixes typos in comments. Extracted from #3467.

---
Stack replacing #3467, bottom to top:
1. #3814 Reset synchronous replication state on demote (independent, can be merged on its own)
2. #3815 Verify that walsenders apply synchronous_standby_names before updating /sync ← **this PR**
3. #3816 Make sure checkpointer enforces synchronous_standby_names
4. #3817 Make remaining quorum members catch up when quorum becomes stricter


inline patch 1: docs/replication_modes.rst, patroni/ha.py, patroni/postgresql/sync.py and 2 more (+308/-64 lines) download inline-1.patchdiff --git a/docs/replication_modes.rst b/docs/replication_modes.rst index 7a803ec142..51cafd1495 100644 --- a/docs/replication_modes.rst +++ b/docs/replication_modes.rst @@ -136,6 +136,8 @@ When in synchronous mode Patroni maintains synchronization state in the DCS (``/ Patroni will only assign one or more synchronous standby nodes based on ``synchronous_node_count`` parameter to ``synchronous_standby_names``. +A standby is published as synchronous in the ``/sync`` key only after Patroni has verified that the new value of ``synchronous_standby_names`` is effectively applied: ``pg_stat_replication`` reports the expected ``sync_state`` and the standby has caught up with the primary's WAL position at the time of the change. + On each HA loop iteration Patroni re-evaluates synchronous standby nodes choice. If the current list of synchronous standby nodes are connected and has not requested its synchronous status to be removed it remains picked. Otherwise the cluster members available for sync that are furthest ahead in replication are picked. Example: diff --git a/patroni/ha.py b/patroni/ha.py index ed7c7f5200..b6013c353f 100644 --- a/patroni/ha.py +++ b/patroni/ha.py @@ -961,22 +961,27 @@ def _check_timeout(offset: float = 0) -> bool: return time.monotonic() - start_time + offset >= self.dcs.loop_wait while True: - transition = 'break' # we need define transition value if `QuorumStateResolver` produced no changes sync_state = self.state_handler.sync_handler.current_state(self.cluster) + if sync_state is None: + transition = 'restart' + transitions = [] + else: + if leader == self.state_handler.name and \ + self._handle_synchronous_strict_mode(sync, sync_state): + return - if leader == self.state_handler.name and \ - self._handle_synchronous_strict_mode(sync, sync_state): - return - - for transition, leader, num, nodes in QuorumStateResolver(leader=leader, - quorum=sync.quorum, - voters=sync.voters, - numsync=sync_state.numsync, - sync=sync_state.sync, - numsync_confirmed=len(sync_state.sync_confirmed), - active=sync_state.active, - sync_wanted=global_config.synchronous_node_count, - leader_wanted=self.state_handler.name): + transition = 'break' # we need to define transition value if `QuorumStateResolver` produced no changes + transitions = list(QuorumStateResolver(leader=leader, + quorum=sync.quorum, + voters=sync.voters, + numsync=sync_state.numsync, + sync=sync_state.sync, + numsync_confirmed=len(sync_state.sync_confirmed), + active=sync_state.active, + sync_wanted=global_config.synchronous_node_count, + leader_wanted=self.state_handler.name)) + + for i, (transition, leader, num, nodes) in enumerate(transitions): if _check_timeout(): return @@ -986,20 +991,29 @@ def _check_timeout(offset: float = 0) -> bool: if not sync: return logger.info('Synchronous replication key updated by someone else.') elif transition == 'sync': - self.state_handler.sync_handler.set_synchronous_standby_names(nodes, num) + if self.state_handler.sync_handler.set_synchronous_standby_names(nodes, num) and \ + i < len(transitions) - 1: + # If synchronous_standby_names change is not yet effective + # and this is not the last transition we need to wait and restart the state machine. + transition = 'restart' + break - if transition == 'break' and sync_state.sync_type != 'quorum': + if transition == 'break' and sync_state and sync_state.sync_type != 'quorum': # FIRST -> ANY self.state_handler.sync_handler.set_synchronous_standby_names(sync_state.sync, sync_state.numsync) if transition != 'restart' or _check_timeout(1): # Check if manage_synchronized_standby_slots feature state changed even when no sync changes - if transition == 'break': + if transition == 'break' and sync_state: self.state_handler.slots_handler.handle_manage_sync_slots_toggle(sync_state.active) return - # synchronous_standby_names was transitioned from empty to non-empty and it may take - # some time for nodes to become synchronous. In this case we want to restart state machine - # hoping that we can update /sync key earlier than in loop_wait seconds. + # Here we consider two cases: + # 1. synchronous_standby_names was transitioned from empty to non-empty and it may take + # some time for nodes to become synchronous. + # 2. some nodes were removed from synchronous_standby_names and we need to wait until + # this change becomes effective after reload. + # In both cases we need to wait and restart state machine hoping that we can update + # /sync key earlier than in loop_wait seconds. time.sleep(1) self.state_handler.reset_cluster_info_state(None) @@ -1017,6 +1031,9 @@ def _process_multisync_replication(self) -> None: return current_state = self.state_handler.sync_handler.current_state(self.cluster) + if current_state is None: + return + picked = current_state.active allow_promote = current_state.sync_confirmed voters = CaseInsensitiveSet(sync.voters) @@ -1063,7 +1080,10 @@ def _process_multisync_replication(self) -> None: if picked and allow_promote != picked: # Wait for PostgreSQL to enable synchronous mode and see if we can immediately set sync_standby time.sleep(2) - allow_promote = self.state_handler.sync_handler.current_state(self.cluster).sync_confirmed + current_state = self.state_handler.sync_handler.current_state(self.cluster) + if current_state is None: + return + allow_promote = current_state.sync_confirmed if allow_promote and allow_promote != sync_common: if self.dcs.write_sync_state(self.state_handler.name, allow_promote, 0, version=sync.version): diff --git a/patroni/postgresql/sync.py b/patroni/postgresql/sync.py index 76aa1851df..8e3a380075 100644 --- a/patroni/postgresql/sync.py +++ b/patroni/postgresql/sync.py @@ -306,42 +306,94 @@ def reset_state(self) -> None: without reaching the WAL position of the new primary. """ self._synchronous_standby_names = '' # last known value of synchronous_standby_names + self._synchronous_standby_names_pending = False + # names from synchronous_standby_names that are counted as synchronous according to pg_stat_replication + self._active_synchronous_standby_names_members = CaseInsensitiveSet() self._ssn_data = deepcopy(_EMPTY_SSN) self._primary_flush_lsn = 0 # "sync" replication connections, that were verified to reach self._primary_flush_lsn at some point self._ready_replicas = CaseInsensitiveDict({}) # keys: member names, values: connection pids - def _handle_synchronous_standby_names_change(self) -> None: - """Handles changes of "synchronous_standby_names" GUC. + def _set_active_synchronous_standby_names_members(self, members: CaseInsensitiveSet) -> None: + """Record nodes from synchronous_standby_names that are counted as synchronous according to pg_stat_replication. - If "synchronous_standby_names" was changed, we need to check that newly added replicas have - reached `self._primary_flush_lsn`. Only after that they could be counted as synchronous. + :param members: new value for :attr:`_active_synchronous_standby_names_members` """ - synchronous_standby_names = self._postgresql.synchronous_standby_names() - if synchronous_standby_names == self._synchronous_standby_names: - return - - self._synchronous_standby_names = synchronous_standby_names - try: - self._ssn_data = parse_sync_standby_names(synchronous_standby_names) - except ValueError as e: - logger.warning('%s', e) - self._ssn_data = deepcopy(_EMPTY_SSN) + self._active_synchronous_standby_names_members = members # Invalidate cache of "sync" connections for app_name in list(self._ready_replicas.keys()): - if app_name not in self._ssn_data.members: + if app_name not in self._active_synchronous_standby_names_members: del self._ready_replicas[app_name] - # Newly connected replicas will be counted as sync only when reached self._primary_flush_lsn - self._primary_flush_lsn = self._postgresql.last_operation() - # Ensure some WAL traffic to move replication + def _generate_wal(self) -> None: + """Ensure some WAL traffic to move replication, so that standbys send feedback.""" self._postgresql.query("""DO $$ BEGIN SET local synchronous_commit = 'off'; PERFORM * FROM pg_catalog.txid_current(); END;$$""") + + def _handle_synchronous_standby_names_change(self) -> Optional[bool]: + """Handles changes of "synchronous_standby_names" GUC. + + If "synchronous_standby_names" was changed, we need to check that newly added replicas have + reached `self._primary_flush_lsn`. Only after that they could be counted as synchronous. + + :returns: `True` if we detect that some nodes were removed from synchronous_standby_names and caller needs + to wait until changes becomes active after reload. + """ + synchronous_standby_names = self._postgresql.synchronous_standby_names() + + if synchronous_standby_names != self._synchronous_standby_names: + # Backends that already wait for synchronous replication are released only when walsenders process + # feedback from standbys. Without WAL traffic it might take wal_receiver_status_interval. + release_waiters = bool(self._synchronous_standby_names) + self._synchronous_standby_names = synchronous_standby_names + + try: + ssn_data = parse_sync_standby_names(synchronous_standby_names) + except ValueError as e: + logger.warning('%s', e) + ssn_data = deepcopy(_EMPTY_SSN) + + # It takes some time before synchronous_standby_names names change propagates via checkpointer. + # Only previously verified members can remain active while the new value is pending. + self._set_active_synchronous_standby_names_members( + CaseInsensitiveSet(self._active_synchronous_standby_names_members & ssn_data.members)) + + self._ssn_data = ssn_data + self._synchronous_standby_names_pending = bool(ssn_data.members) + if release_waiters and ssn_data.members: + self._generate_wal() + + if not self._synchronous_standby_names_pending: + return + + pg_stat_replication = self._postgresql.pg_stat_replication() + replication = CaseInsensitiveSet( + [row['application_name'] for row in pg_stat_replication + if self._ssn_data.sync_type == 'priority' and row['sync_state'] in ('potential', 'sync') + or self._ssn_data.sync_type == 'quorum' and row['sync_state'] == 'quorum']) + + # Nodes listed in synchronous_standby_names should have corresponding sync_state in pg_stat_replication. + # Only from this moment we count synchronous_standby_names effectively applied. + if replication != self._ssn_data.members: + # If some (not ALL) nodes were removed from synchronous_standby_names we need to notify caller + # that it has to wait until synchronous_standby_names change becomes active after reload. + return bool(self._ssn_data.members and not self._ssn_data.has_star + and len(replication - self._ssn_data.members)) + + self._set_active_synchronous_standby_names_members(self._ssn_data.members) + + # Newly connected replicas will be counted as sync only when reached self._primary_flush_lsn. + # NB: a change of only N in "ANY N (...)" isn't visible in pg_stat_replication. A walsender that didn't process + # the reload yet could still confirm a commit after self._primary_flush_lsn using the old N, but walsenders + # process the reload before handling the next replies from standbys, therefore the window is very short. + self._primary_flush_lsn = self._postgresql.last_operation() + self._generate_wal() self._postgresql.reset_cluster_info_state(None) # Reset internal cache to query fresh values + self._synchronous_standby_names_pending = False def _process_replica_readiness(self, cluster: Cluster, replica_list: _ReplicaList) -> None: """Flags replicas as truly "synchronous" when they have caught up with ``_primary_flush_lsn``. @@ -353,7 +405,7 @@ def _process_replica_readiness(self, cluster: Cluster, replica_list: _ReplicaLis # if standby name is listed in the /sync key we can count it as synchronous, otherwise # it becomes really synchronous when sync_state = 'sync' and it is known that it managed to catch up if replica.application_name not in self._ready_replicas\ - and replica.application_name in self._ssn_data.members: + and replica.application_name in self._active_synchronous_standby_names_members: if global_config.is_quorum_commit_mode: # When quorum commit is enabled we can't check against cluster.sync because nodes # are written there when at least one of them caught up with _primary_flush_lsn. @@ -368,7 +420,7 @@ def _process_replica_readiness(self, cluster: Cluster, replica_list: _ReplicaLis # "really" synchronous when sync_state = 'sync' and we known that it managed to catch up self._ready_replicas[replica.application_name] = replica.pid - def current_state(self, cluster: Cluster) -> _SyncState: + def current_state(self, cluster: Cluster) -> Optional[_SyncState]: """Find the best candidates to be the synchronous standbys. Current synchronous standby is always preferred, unless it has disconnected or does not want to be a @@ -385,9 +437,11 @@ def current_state(self, cluster: Cluster) -> _SyncState: :param cluster: current cluster topology from DCS - :returns: current synchronous replication state as a :class:`_SyncState` object + :returns: current synchronous replication state as a :class:`_SyncState` object or `None` in case if we detect + that some nodes were removed from synchronous_standby_names and change is not yet active after reload. """ - self._handle_synchronous_standby_names_change() + if self._handle_synchronous_standby_names_change(): + return None replica_list = _ReplicaList(self._postgresql, cluster) self._process_replica_readiness(cluster, replica_list) @@ -422,7 +476,7 @@ def current_state(self, cluster: Cluster) -> _SyncState: sync = CaseInsensitiveSet([SYNC_STRICT_PLACEHOLDER]) if self._ssn_data.has_star else self._ssn_data.members return _SyncState(self._ssn_data.sync_type, self._ssn_data.num, sync, sync_confirmed, active) - def set_synchronous_standby_names(self, sync: Collection[str], num: Optional[int] = None) -> None: + def set_synchronous_standby_names(self, sync: Collection[str], num: Optional[int] = None) -> Optional[bool]: """Constructs and sets ``synchronous_standby_names`` GUC value. .. note:: @@ -430,6 +484,8 @@ def set_synchronous_standby_names(self, sync: Collection[str], num: Optional[int :param sync: set of nodes to sync to :param num: specifies number of nodes to sync to. The *num* is set only in case if quorum commit is enabled + + :returns: `True` if caller needs to wait until synchronous_standby_names change becomes active after reload. """ # Update synchronized_standby_slots in _server_parameters first (no reload yet) so # that the subsequent set_synchronous_standby_names() reload picks up both changes. @@ -466,16 +522,19 @@ def set_synchronous_standby_names(self, sync: Collection[str], num: Optional[int self._postgresql.config.write_postgresql_conf() self._postgresql.reload() - if not (ssn_changed - and self._postgresql.state == PostgresqlState.RUNNING - and self._postgresql.is_primary()) or sync_strict: + if sync_strict or self._postgresql.state != PostgresqlState.RUNNING or not self._postgresql.is_primary(): return - time.sleep(0.1) # Usually it takes 1ms to reload postgresql.conf, but we will give it 100ms + if ssn_changed: + time.sleep(0.1) # Usually it takes 1ms to reload postgresql.conf, but we will give it 100ms - # Reset internal cache to query fresh values - self._postgresql.reset_cluster_info_state(None) + # Reset internal cache to query fresh values + self._postgresql.reset_cluster_info_state(None) + elif self._synchronous_standby_names == (sync_param or ''): + return # the value was already written and observed earlier # timeline == 0 -- indicates that this is the replica if self._postgresql.get_primary_timeline() > 0: - self._handle_synchronous_standby_names_change() + # Reload could take longer than 100ms, in this case the new value is still not visible. + return self._handle_synchronous_standby_names_change()\ + or self._synchronous_standby_names != (sync_param or '') diff --git a/tests/test_ha.py b/tests/test_ha.py index d6adbbd761..c1247acc7c 100644 --- a/tests/test_ha.py +++ b/tests/test_ha.py @@ -635,7 +635,6 @@ def test_call_failsafe_member(self): ('Request failed to %s: POST %s (%s)', 'test', 'http://localhost:8011/failsafe';, e)) self.assertFalse(ret.accepted) - @patch('time.sleep', Mock()) def test_bootstrap_from_another_member(self): self.ha.cluster = get_cluster_initialized_with_leader() self.assertEqual(self.ha.bootstrap(), 'trying to bootstrap from replica \'other\'') @@ -778,7 +777,6 @@ def test_reinitialize(self): self.ha.state_handler.name = self.ha.cluster.leader.name self.assertIsNotNone(self.ha.reinitialize()) - @patch('time.sleep', Mock()) def test_restart(self): self.assertEqual(self.ha.restart({}), (True, 'restarted successfully')) self.p.restart = Mock(return_value=None) @@ -791,7 +789,6 @@ def test_restart(self): with patch.object(self.ha, "restart_matches", return_value=False): self.assertEqual(self.ha.restart({'foo': 'bar'}), (False, "restart conditions are not satisfied")) - @patch('time.sleep', Mock()) @patch.object(ConfigHandler, 'replace_pg_hba', Mock()) @patch.object(ConfigHandler, 'replace_pg_ident', Mock()) @patch.object(PostmasterProcess, 'start', Mock(return_value=MockPostmaster())) @@ -1869,6 +1866,28 @@ def test__process_multisync_replication(self): mock_set_sync.assert_not_called() mock_cfg_set_sync.assert_called_once_with("SOME_SSN") + def test__process_multisync_replication_wait(self): + self.ha.has_lock = true + self.ha.is_synchronous_mode = true + mock_set_sync = self.p.sync_handler.set_synchronous_standby_names = Mock() + self.ha.dcs.write_sync_state = Mock(return_value=SyncState.empty()) + self.p.name = 'leader' + self.ha.cluster = get_cluster_initialized_with_leader(sync=('leader', 'a,b')) + + # Test _process_multisync_replication early exit if waiting for synchronous_standby_names change apply + self.p.sync_handler.current_state = Mock(return_value=None) + self.ha.run_cycle() + mock_set_sync.assert_not_called() + + # Test wait until synchronous_standby_names change is applied after reload when some nodes are removed + self.p.sync_handler.current_state = Mock(side_effect=[_SyncState('priority', 1, CaseInsensitiveSet('ab'), + CaseInsensitiveSet(), + CaseInsensitiveSet(['a'])), + None]) + self.ha.run_cycle() + self.assertEqual(mock_set_sync.call_count, 1) + self.assertEqual(mock_set_sync.call_args_list[0][0][0], CaseInsensitiveSet('a')) + def test_sync_replication_become_primary(self): self.ha.is_synchronous_mode = true @@ -2055,7 +2074,6 @@ def test_shutdown_citus_worker(self): self.assertEqual(self.ha.patroni.request.call_args[0][2], 'citus') self.assertEqual(self.ha.patroni.request.call_args[0][3]['type'], 'before_demote') - @patch('time.sleep', Mock()) def test_leader_with_not_accessible_data_directory(self): self.ha.cluster = get_cluster_initialized_with_leader() self.ha.has_lock = true @@ -2312,8 +2330,8 @@ def test__process_quorum_replication(self): self.ha.cluster = get_cluster_initialized_with_leader(sync=('leader', 'foo')) # Test the sync node is removed from voters, added to ssn with patch.object(Postgresql, 'synchronous_standby_names', Mock(return_value='other')), \ - patch('time.sleep', Mock()): - self.ha.run_cycle() + patch.object(Postgresql, 'pg_stat_replication', Mock(return_value=[])): + self.assertEqual(self.ha.run_cycle(), 'no action. I am (leader), the leader with the lock') self.assertEqual(mock_write_sync.call_count, 1) self.assertEqual(mock_write_sync.call_args_list[0][0], (self.p.name, CaseInsensitiveSet(), 0)) self.assertEqual(mock_write_sync.call_args_list[0][1], {'version': 0}) @@ -2328,7 +2346,8 @@ def test__process_quorum_replication(self): mock_write_sync.reset_mock() mock_set_sync.reset_mock() with patch.object(global_config.__class__, 'is_synchronous_mode_strict', PropertyMock(return_value=True)), \ - patch.object(Postgresql, 'synchronous_standby_names', Mock(return_value='ANY 1 (foo)')): + patch.object(Postgresql, 'synchronous_standby_names', Mock(return_value='ANY 1 (foo)')), \ + patch.object(Postgresql, 'pg_stat_replication', Mock(return_value=[])): self.ha.run_cycle() mock_write_sync.assert_not_called() self.assertEqual(mock_set_sync.call_count, 1) @@ -2346,6 +2365,7 @@ def test__process_quorum_replication(self): # Test that _process_quorum_replication doesn't take longer than loop_wait with patch.object(Postgresql, 'synchronous_standby_names', Mock(return_value='ANY 1 (foo)')), \ + patch.object(Postgresql, 'pg_stat_replication', Mock(return_value=[])), \ patch('time.monotonic', Mock(side_effect=[30, 60, 90, 120, 150])): self.ha.process_sync_replication() @@ -2357,12 +2377,38 @@ def test__process_quorum_replication(self): CaseInsensitiveSet(['foo']))) mock_write_sync.reset_mock() mock_set_sync.reset_mock() - with patch.object(Postgresql, 'synchronous_standby_names', Mock(return_value='ANY 1 (foo)')): + with patch.object(Postgresql, 'synchronous_standby_names', Mock(return_value='ANY 1 (foo)')), \ + patch.object(Postgresql, 'pg_stat_replication', Mock(return_value=[])): self.ha.run_cycle() mock_write_sync.assert_not_called() self.assertEqual(mock_set_sync.call_count, 1) self.assertEqual(mock_set_sync.call_args_list[0][0], ('ANY 1 (foo)',)) + # Test wait until synchronous_standby_names change is applied after reload when some nodes are removed + self.p.sync_handler.current_state = Mock(side_effect=[_SyncState('quorum', 1, CaseInsensitiveSet('ab'), + CaseInsensitiveSet(), + CaseInsensitiveSet(['a'])), + None, + _SyncState('quorum', 1, CaseInsensitiveSet('a'), + CaseInsensitiveSet('a'), + CaseInsensitiveSet('a'))]) + mock_write_sync = self.ha.dcs.write_sync_state = Mock(return_value=SyncState(1, 'leader', 'a,b', 2)) + self.ha.cluster = get_cluster_initialized_with_leader(sync=('leader', 'a,b')) + mock_set_sync.reset_mock() + with patch.object(Postgresql, 'synchronous_standby_names', Mock(return_value='ANY 1 (a)')), \ + patch('time.time', Mock(return_value=100)), \ + patch.object(Postgresql, 'pg_stat_replication', + Mock(return_value=[{'application_name': 'a', 'sync_state': 'quorum'}, + {'application_name': 'b', 'sync_state': 'quorum'}])): + self.ha.run_cycle() + self.assertEqual(mock_write_sync.call_count, 2) + self.assertEqual(mock_write_sync.call_args_list[0][0], (self.p.name, CaseInsensitiveSet('ab'), 2)) + self.assertEqual(mock_write_sync.call_args_list[0][1], {'version': 0}) + self.assertEqual(mock_write_sync.call_args_list[1][0], (self.p.name, CaseInsensitiveSet('a'), 0)) + self.assertEqual(mock_write_sync.call_args_list[1][1], {'version': 1}) + self.assertEqual(mock_set_sync.call_count, 1) + self.assertEqual(mock_set_sync.call_args_list[0][0], ('ANY 1 (a)',)) + def test_is_failover_possible(self): self.p._major_version = 140000 # supports_multiple_sync self.p.name = 'leader' diff --git a/tests/test_sync.py b/tests/test_sync.py index 8919372dfe..fbd1f6b239 100644 --- a/tests/test_sync.py +++ b/tests/test_sync.py @@ -42,7 +42,7 @@ def test_pick_sync_standby(self): {'pid': 102, 'application_name': self.other.name, 'sync_state': 'async', 'flush_lsn': 2}] # sync node is a bit behind of async, but we prefer it anyway - with patch.object(Postgresql, "_cluster_info_state_get", side_effect=[self.leadermem.name, + with patch.object(Postgresql, "_cluster_info_state_get", side_effect=[self.leadermem.name, pg_stat_replication, 'on', pg_stat_replication]): self.assertEqual(self.s.current_state(self.cluster), ('priority', 1, CaseInsensitiveSet([self.leadermem.name]), @@ -79,7 +79,7 @@ def test_pick_sync_standby(self): CaseInsensitiveSet())) # synchronous_standby_names contains '*' - with patch.object(Postgresql, "_cluster_info_state_get", side_effect=['*', 'remote_apply', None]): + with patch.object(Postgresql, "_cluster_info_state_get", side_effect=['*', None, 'remote_apply', None]): self.p._major_version = 90400 self.assertEqual(self.s.current_state(self.cluster), ('priority', 1, @@ -98,7 +98,7 @@ def test_current_state_quorum(self): # sync node is a bit behind of async, but we prefer it anyway with patch.object(Postgresql, "_cluster_info_state_get", side_effect=['ANY 1 ({0},"{1}")'.format(self.leadermem.name, self.me.name), - 'on', pg_stat_replication]): + pg_stat_replication, 'on', pg_stat_replication]): self.assertEqual(self.s.current_state(self.cluster), ('quorum', 1, CaseInsensitiveSet([self.me.name, self.leadermem.name]), CaseInsensitiveSet([self.me.name, self.leadermem.name]), @@ -137,11 +137,128 @@ def test_current_state_cascading(self): self.leadermem.data['tags'] = {'replicatefrom': self.me.name} with patch.object(Postgresql, "_cluster_info_state_get", side_effect=['2 ({0},"{1}")'.format(self.leadermem.name, self.other.name), - 'on', pg_stat_replication]): + pg_stat_replication, 'on', pg_stat_replication]): self.assertEqual(self.s.current_state(self.cluster), ('priority', 2, CaseInsensitiveSet([self.other.name, self.leadermem.name]), CaseInsensitiveSet(), CaseInsensitiveSet([self.me.name]))) + def test_current_state_waiting(self): + pg_stat_replication = [ + {'pid': 100, 'application_name': self.me.name, 'sync_state': 'sync', 'flush_lsn': 1}, + {'pid': 101, 'application_name': self.other.name, 'sync_state': 'sync', 'flush_lsn': 2}] + + # some nodes were removed from synchronous_standby_names + with patch.object(Postgresql, "_cluster_info_state_get", + side_effect=['1 ("{0}")'.format(self.other.name), pg_stat_replication]): + self.assertIsNone(self.s.current_state(self.cluster)) + + @patch.object(Postgresql, 'last_operation', Mock(return_value=100)) + def test_current_state_waiting_after_removal(self): + self.cluster.config.data['synchronous_mode'] = 'quorum' + global_config.update(self.cluster) + a, b = self.leadermem.name, self.me.name + replication = [ + {'pid': 100, 'application_name': a, 'sync_state': 'quorum', 'flush_lsn': 100}, + {'pid': 101, 'application_name': b, 'sync_state': 'quorum', 'flush_lsn': 100}] + + with patch.object(Postgresql, 'synchronous_standby_names', + return_value='ANY 1 ({0},{1})'.format(a, b)) as mock_ssn, \ + patch.object(Postgresql, 'pg_stat_replication', return_value=replication): + self.assertEqual(self.s.current_state(self.cluster).sync_confirmed, CaseInsensitiveSet([a, b])) + + # SHOW changed, but b is still reported as quorum. + mock_ssn.return_value = 'ANY 1 ({0})'.format(a) + for _ in range(2): + self.assertIsNone(self.s.current_state(self.cluster)) + + replication[1]['sync_state'] = 'async' + self.assertEqual(self.s.current_state(self.cluster).sync_confirmed, CaseInsensitiveSet([a])) + + def test_current_state_overlapping_changes(self): + self.cluster.config.data['synchronous_node_count'] = 3 + global_config.update(self.cluster) + self.other.data['tags'] = {} + a, b, c = self.leadermem.name, self.me.name, self.other.name + replication = [ + {'pid': 100, 'application_name': a, 'sync_state': 'sync', 'flush_lsn': 100}, + {'pid': 101, 'application_name': b, 'sync_state': 'async', 'flush_lsn': 100}] + + with patch.object(Postgresql, 'synchronous_standby_names', return_value=a) as mock_ssn, \ + patch.object(Postgresql, 'pg_stat_replication', return_value=replication), \ + patch.object(Postgresql, 'last_operation', return_value=100) as mock_lsn: + self.assertEqual(self.s.current_state(self.cluster).sync_confirmed, CaseInsensitiveSet([a])) + + mock_ssn.return_value = '2 ({0},{1})'.format(a, b) + mock_lsn.return_value = 200 + replication[0]['flush_lsn'] = 200 + self.assertEqual(self.s.current_state(self.cluster).sync_confirmed, CaseInsensitiveSet([a])) + + # Add c before the previous configuration was verified. + mock_ssn.return_value = '3 ({0},{1},"{2}")'.format(a, b, c) + mock_lsn.return_value = 300 + replication[0]['flush_lsn'] = 300 + replication[1]['sync_state'] = 'sync' + replication.append({'pid': 102, 'application_name': c, 'sync_state': 'async', 'flush_lsn': 100}) + self.assertEqual(self.s.current_state(self.cluster).sync_confirmed, CaseInsensitiveSet([a])) + + # All names now match, but b and c have not reached the new WAL boundary. + mock_lsn.return_value = 400 + replication[0]['flush_lsn'] = 400 + replication[2]['sync_state'] = 'sync' + self.assertEqual(self.s.current_state(self.cluster).sync_confirmed, CaseInsensitiveSet([a])) + replication[1]['flush_lsn'] = 300 + self.assertEqual(self.s.current_state(self.cluster).sync_confirmed, CaseInsensitiveSet([a])) + + # Catching up to that boundary is sufficient even if the primary advances. + mock_lsn.return_value = 500 + replication[0]['flush_lsn'] = 500 + replication[1]['flush_lsn'] = replication[2]['flush_lsn'] = 400 + self.assertEqual(self.s.current_state(self.cluster).sync_confirmed, CaseInsensitiveSet([a, b, c])) + + @patch.object(Postgresql, 'last_operation', Mock(return_value=1)) + def test_current_state_releases_waiters_on_change(self): + a, b = self.leadermem.name, '"{0}"'.format(self.other.name) + + def wal_generated(): + return any('txid_current' in c[0][0] for c in mock_query.call_args_list) + + # walsenders didn't process the reload yet, therefore the new value remains pending + with patch.object(Postgresql, 'synchronous_standby_names', return_value=a) as mock_ssn, \ + patch.object(Postgresql, 'pg_stat_replication', return_value=[]), \ + patch.object(Postgresql, 'query') as mock_query: + # nothing waited for synchronous replication while synchronous_standby_names was empty + self.assertEqual(self.s.current_state(self.cluster).sync_confirmed, CaseInsensitiveSet()) + self.assertFalse(wal_generated()) + # backends that were waiting for the old value must be released without waiting for verification + mock_ssn.return_value = b + self.assertEqual(self.s.current_state(self.cluster).sync_confirmed, CaseInsensitiveSet()) + self.assertTrue(wal_generated()) + + @patch('time.sleep', Mock()) + @patch.object(Postgresql, 'is_primary', Mock(return_value=True)) + @patch.object(Postgresql, 'get_primary_timeline', Mock(return_value=1)) + @patch.object(Postgresql, 'last_operation', Mock(return_value=1)) + @patch.object(Postgresql, 'state', PropertyMock(return_value=PostgresqlState.RUNNING)) + def test_set_sync_standby_slow_reload(self): + a, b = self.leadermem.name, self.me.name + replication = [{'pid': 100, 'application_name': a, 'sync_state': 'sync', 'flush_lsn': 1}, + {'pid': 101, 'application_name': b, 'sync_state': 'async', 'flush_lsn': 1}] + + with patch.object(Postgresql, 'synchronous_standby_names', return_value=a) as mock_ssn, \ + patch.object(Postgresql, 'pg_stat_replication', return_value=replication), \ + patch.object(ConfigHandler, 'set_synchronous_standby_names', Mock(return_value=True)) as mock_set: + self.assertFalse(self.s.set_synchronous_standby_names([a])) + + # postgresql.conf was updated, but the new value is not yet visible + self.assertTrue(self.s.set_synchronous_standby_names([a, b])) + mock_set.return_value = None # postgresql.conf already has the new value + self.assertTrue(self.s.set_synchronous_standby_names([a, b])) + + mock_ssn.return_value = '2 ({0},{1})'.format(a, b) + replication[1]['sync_state'] = 'sync' + self.assertFalse(self.s.set_synchronous_standby_names([a, b])) + self.assertIsNone(self.s.set_synchronous_standby_names([a, b])) + @patch('time.sleep', Mock()) def test_set_sync_standby(self): def value_in_conf(): @@ -311,7 +428,7 @@ def test_do_not_prick_yourself(self): # Faulty case, application_name of the current primary is in synchronous_standby_names and in # the pg_stat_replication. We need to check that primary is not selected as the synchronous node. - with patch.object(Postgresql, "_cluster_info_state_get", side_effect=[self.leadermem.name, + with patch.object(Postgresql, "_cluster_info_state_get", side_effect=[self.leadermem.name, pg_stat_replication, 'on', pg_stat_replication]): self.assertEqual(self.s.current_state(cluster), ('priority', 1, CaseInsensitiveSet([self.leadermem.name]), CaseInsensitiveSet(), CaseInsensitiveSet([self.me.name])))
^ permalink raw [nested|flat] 4+ messages in thread

* Re: [patroni/patroni] PR #3815: Verify that walsenders apply synchronous_standby_names before updating /sync
@ 2026-11-19 21:59 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
  2 siblings, 0 replies; 4+ messages in thread

From: CyberDem0n (@CyberDem0n) @ 2026-11-19 21:59 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

@CyberDem0n requested a review.

^ permalink  raw  [nested|flat] 4+ messages in thread

* Re: [patroni/patroni] PR #3815: Verify that walsenders apply synchronous_standby_names before updating /sync
@ 2026-11-19 21:59 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
  2 siblings, 0 replies; 4+ messages in thread

From: CyberDem0n (@CyberDem0n) @ 2026-11-19 21:59 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

@CyberDem0n applied label "bug".

^ permalink  raw  [nested|flat] 4+ messages in thread

* Re: [patroni/patroni] PR #3815: Verify that walsenders apply synchronous_standby_names before updating /sync
@ 2026-11-19 22:07 ` "codecov[bot] (bot) (@codecov[bot])" <noreply+codecov[bot]@github.com>
  2 siblings, 0 replies; 4+ messages in thread

From: codecov[bot] (bot) (@codecov[bot]) @ 2026-11-19 22:07 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

## [Codecov](https://app.codecov.io/gh/patroni/patroni/pull/3815?dropdown=coverage&src=pr&el=h1&utm_m...) Report
:white_check_mark: All modified and coverable lines are covered by tests.

:loudspeaker: Thoughts on this report? [Let us know!](https://github.com/codecov/feedback/issues/255)

^ permalink  raw  [nested|flat] 4+ messages in thread


end of thread, other threads:[~2026-11-19 22:07 UTC | newest]

Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-11-19 21:59 [patroni/patroni] PR #3815: Verify that walsenders apply synchronous_standby_names before updating /sync "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-19 21:59 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-19 21:59 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-19 22:07 ` "codecov[bot] (bot) (@codecov[bot])" <noreply+codecov[bot]@github.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox