pg.ddx.io  patroni/patroni GitHub issues and pull requests  
help / color / mirror / Atom feed
From: guriandoro (@guriandoro) <noreply+guriandoro@github.com>
To: patroni/patroni <noreply+patroni-patroni@github.com>
Subject: [patroni/patroni] PR #3697: Report 'promoted' role in patronictl list and /cluster endpoint
Date: Tue, 18 Aug 2026 21:39:13 +0000
Message-ID: <gh-patroni-patroni-3697@github.com> (raw)

## What problem does this solve?

Fixes the visibility half of #3603 (the log warning is sent as a separate PR).

After a failover, a node can win the leader lock and receive the promote
request, yet Postgres never finishes the promotion because
`restore_command` hangs (in my case: a stalled NFS mount behind
pgBackRest's `archive-get`). Postgres keeps running as a standby,
`pg_is_in_recovery()` stays `true`, and the cluster has no writable node.

Meanwhile `patronictl list` and the `/cluster` REST endpoint show a
healthy `Leader` / `role: leader`, because `cluster_as_json()` hardcodes
the role for whichever member holds the lock. The DCS itself tells the
truth: the member key keeps `"role": "promoted"` until promotion
completes. That signal was being dropped at the presentation layer.

## What does this PR do?

In `cluster_as_json()` (`patroni/utils.py`), when the lock holder's
member key still reports `role: promoted`, pass it through instead of
hardcoding `leader`:

- `patronictl list` shows `Promoted` in the Role column while promotion
  has not completed, reverting to `Leader` once it does.
- `GET /cluster` returns `"role": "promoted"` for that member.

Side effect (arguably a second fix): since `replication_state` is only
suppressed for role `leader`, the State column for a stuck member now
shows the actual state (e.g. `in archive recovery`) instead of
`running` — which is exactly what the operator needs to see to find the
hanging `restore_command`.

During a normal failover the `Promoted` role is visible for at most a
cycle or two, same as the `promoted` role in the DCS member key today.

Standby clusters are unaffected (`standby_leader` takes precedence, as
before). All other surfaces (`/patroni`, `/primary`, `/metrics`, the DCS
member key) already report this state correctly and are untouched.

## Repro / verification

Original repro from the issue: 2-node cluster with pgBackRest over NFS,
block NFS return traffic on the replica with iptables, `kill -9` Patroni
on the leader. The replica takes the lock, promote is issued,
`archive-get 00000002.history` hangs forever, `pg_is_in_recovery()`
stays true.

Verified live against this branch (PG 18.4, etcd v3.5, 2-node local
cluster): the hanging `restore_command` was simulated with a script that
sleeps forever on timeline-history requests, then Patroni was `kill -9`'d
on the leader. While node2 was stuck (`pg_is_in_recovery()` = `t`):

```
$ patronictl list
+ Cluster: repro3603 (7675475385382874329) ----------------+----+-------------+-----+------------+-----+
| Member | Host           | Role     | State               | TL | Receive LSN | Lag | Replay LSN | Lag |
+--------+----------------+----------+---------------------+----+-------------+-----+------------+-----+
| node2  | 127.0.0.1:5434 | Promoted | in archive recovery |  1 |             |     |            |     |
+--------+----------------+----------+---------------------+----+-------------+-----+------------+-----+

$ curl -s http://127.0.0.1:8009/cluster
{"members": [{"name": "node2", "role": "promoted", "state": "in archive recovery", ...}], "scope": "repro3603"}
```

Once the restore_command hang was released, promotion completed and both
surfaces reverted on the next loop:

```
$ patronictl list
| node2  | 127.0.0.1:5434 | Leader | running |  2 | ...

$ curl -s http://127.0.0.1:8009/cluster
{"members": [{"name": "node2", "role": "leader", "state": "running", ...}], "scope": "repro3603"}
```

## Tests

Extended `test_output_members` in `tests/test_ctl.py`: a cluster whose
lock holder has `role: promoted` in its member data renders `Promoted`
in the Role column. Full suite passes (706 tests). Docs updated:
`docs/rest_api.rst` role enumeration, `cluster_as_json` and
`output_members` docstrings.

inline patch 1: docs/rest_api.rst, patroni/ctl.py, patroni/utils.py and 1 more (+58/-12 lines) download inline-1.patchdiff --git a/docs/rest_api.rst b/docs/rest_api.rst index ce2f5c04d..0cd68c370 100644 --- a/docs/rest_api.rst +++ b/docs/rest_api.rst @@ -479,6 +479,7 @@ Cluster status endpoints } } + The ``role`` of a member is one of ``leader``, ``promoted``, ``standby_leader``, ``sync_standby``, ``quorum_standby``, or ``replica``. The ``promoted`` role is reported for a member that already holds the leader lock while PostgreSQL promotion has not completed yet, i.e. PostgreSQL is still running in recovery and therefore is read-only. - The ``GET /history`` endpoint provides a view on the history of cluster switchovers/failovers. The format is very similar to the content of history files in the ``pg_wal`` directory. The only difference is the timestamp field showing when the new timeline was created. diff --git a/patroni/ctl.py b/patroni/ctl.py index a214ebb72..732efcfa1 100644 --- a/patroni/ctl.py +++ b/patroni/ctl.py @@ -537,14 +537,16 @@ def get_all_members(cluster: Cluster, group: Optional[int], if is_citus_cluster() and group is None: clusters.update(cluster.workers) if role in (CtlPostgresqlRole.LEADER, CtlPostgresqlRole.PRIMARY, CtlPostgresqlRole.STANDBY_LEADER): - # In the DCS the members' role can be one among: ``primary``, ``master``, ``replica`` or ``standby_leader``. - # ``primary`` and ``master`` are the same thing. + # In the DCS the members' role can be one among: ``primary``, ``master``, ``replica``, ``standby_leader``, + # or ``promoted``. ``primary`` and ``master`` are the same thing. ``promoted`` means the member holds the + # leader lock but Postgres promotion has not completed yet, so it is neither a primary nor a standby leader. for cluster in clusters.values(): if cluster.leader is not None and cluster.leader.name and\ (role == CtlPostgresqlRole.LEADER - or cluster.leader.data.get('role') not in (PostgresqlRole.PRIMARY, PostgresqlRole.MASTER) + or cluster.leader.data.get('role') not in (PostgresqlRole.PRIMARY, PostgresqlRole.MASTER, + PostgresqlRole.PROMOTED) and role == CtlPostgresqlRole.STANDBY_LEADER - or cluster.leader.data.get('role') != PostgresqlRole.STANDBY_LEADER + or cluster.leader.data.get('role') not in (PostgresqlRole.STANDBY_LEADER, PostgresqlRole.PROMOTED) and role == CtlPostgresqlRole.PRIMARY): yield cluster.leader.member return @@ -1593,7 +1595,8 @@ def output_members(cluster: Cluster, name: str, extended: bool = False, * ``Site``: site of the Patroni node, as per ``site`` configuration; * ``Member``: name of the Patroni node, as per ``name`` configuration; * ``Host``: hostname (or IP) and port, as per ``postgresql.listen`` configuration; - * ``Role``: ``Leader``, ``Standby Leader``, ``Sync Standby`` or ``Replica``; + * ``Role``: ``Leader``, ``Promoted`` (holds the leader lock but promotion has not completed yet), + ``Standby Leader``, ``Sync Standby`` or ``Replica``; * ``State``: one of :class:`~patroni.postgresql.misc.PostgresqlState`; * ``TL``: current timeline in Postgres; * ``Receive LSN``: last received LSN (``pg_catalog.pg_last_(xlog|wal)_receive_(location|lsn)()``); @@ -2367,14 +2370,16 @@ def change_cluster_role(cluster_name: str, force: bool, standby_config: Optional """ demote = bool(standby_config) action_name = 'demot' if demote else 'promot' - target_role = PostgresqlRole.STANDBY_LEADER if demote else PostgresqlRole.PRIMARY + # the member key reports role 'promoted' as soon as promotion is initiated with ``pg_ctl promote``, and only + # switches to 'primary' once Postgres has left recovery; don't make the user wait for the latter + target_roles = (PostgresqlRole.STANDBY_LEADER,) if demote else (PostgresqlRole.PRIMARY, PostgresqlRole.PROMOTED) dcs = get_dcs(cluster_name, None) cluster = dcs.get_cluster() leader_name = cluster.leader and cluster.leader.name if not leader_name: raise PatroniCtlException(f'Cluster has no leader, {action_name}ion is not possible') - if cluster.leader and cluster.leader.data.get('role') == target_role: + if cluster.leader and cluster.leader.data.get('role') in target_roles: raise PatroniCtlException('Cluster is already in the required state') click.echo('Current cluster topology') @@ -2404,7 +2409,7 @@ def change_cluster_role(cluster_name: str, force: bool, standby_config: Optional old_leader = cluster.get_member(leader_name, False) old_leader_state = old_leader and old_leader.data.get('state') - if not is_unlocked and leader_role == target_role and leader_state == PostgresqlState.RUNNING: + if not is_unlocked and leader_role in target_roles and leader_state == PostgresqlState.RUNNING: if not demote or old_leader_state == PostgresqlState.RUNNING: click.echo( f'{datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")} cluster is successfully {action_name}ed') diff --git a/patroni/utils.py b/patroni/utils.py index a8f8ad94e..df4dfc709 100644 --- a/patroni/utils.py +++ b/patroni/utils.py @@ -922,7 +922,8 @@ def cluster_as_json(cluster: 'Cluster') -> Dict[str, Any]: * ``members``: list of members in the cluster. Each value is a :class:`dict` that may have the following keys: * ``name``: the name of the host (unique in the cluster). The ``members`` list is sorted by this key; - * ``role``: ``leader``, ``standby_leader``, ``sync_standby``, ``quorum_standby``, or ``replica``; + * ``role``: ``leader``, ``promoted`` (holds the leader lock but promotion has not completed yet), + ``standby_leader``, ``sync_standby``, ``quorum_standby``, or ``replica``; * ``state``: one of :class:`~patroni.postgresql.misc.PostgresqlState`; * ``api_url``: REST API URL based on ``restapi->connect_address`` configuration; * ``host``: PostgreSQL host based on ``postgresql->connect_address``; @@ -948,7 +949,7 @@ def cluster_as_json(cluster: 'Cluster') -> Dict[str, Any]: * ``to``: name of the member to be promoted. """ from . import global_config - from .postgresql.misc import format_lsn + from .postgresql.misc import format_lsn, PostgresqlRole config = global_config.from_cluster(cluster) leader_name = cluster.leader.name if cluster.leader else None @@ -958,13 +959,20 @@ def cluster_as_json(cluster: 'Cluster') -> Dict[str, Any]: sync_role = 'quorum_standby' if config.is_quorum_commit_mode else 'sync_standby' for m in cluster.members: if m.name == leader_name: - role = 'standby_leader' if config.is_standby_cluster else 'leader' + if config.is_standby_cluster: + role = 'standby_leader' + elif m.data.get('role') == PostgresqlRole.PROMOTED: + # the member holds the leader lock, but Postgres promotion has not completed yet (still in recovery) + role = 'promoted' + else: + role = 'leader' elif config.is_synchronous_mode and cluster.sync.matches(m.name): role = sync_role else: role = 'replica' - state = (m.data.get('replication_state', '') if role != 'leader' else '') or m.data.get('state', '') + state = (m.data.get('replication_state', '') if role not in ('leader', 'promoted') else '')\ + or m.data.get('state', '') member = {'name': m.name, 'role': role, 'state': state, 'api_url': m.api_url} conn_kwargs = m.conn_kwargs() if conn_kwargs.get('host'): diff --git a/tests/test_ctl.py b/tests/test_ctl.py index 5079024e7..3572c6656 100644 --- a/tests/test_ctl.py +++ b/tests/test_ctl.py @@ -155,6 +155,14 @@ def test_output_members(self): self.assertIsNone(output_members(cluster, name='abc')) self.assertIn('to: site dc2', mock_echo.call_args_list[1][0][0]) + # the lock holder is shown as 'Promoted' while Postgres promotion has not completed yet + mock_echo.reset_mock() + cluster = get_cluster_initialized_with_leader() + cluster.members[0].data.update(role=PostgresqlRole.PROMOTED, state='running', + replication_state='in archive recovery') + self.assertIsNone(output_members(cluster, name='abc', fmt='tsv')) + self.assertIn('\tPromoted\trunning\t', mock_echo.call_args_list[1][0][0]) + @patch('patroni.dcs.AbstractDCS.set_failover_value', Mock()) def test_switchover(self): # Confirm @@ -533,6 +541,16 @@ def test_get_all_members(self): self.assertEqual(len(r), 1) self.assertEqual(r[0].name, 'leader') + # a lock holder whose promotion has not completed yet is neither a primary nor a standby leader, + # but it is still the leader (it holds the lock) + cluster = get_cluster_initialized_with_leader() + cluster.members[0].data['role'] = PostgresqlRole.PROMOTED + self.assertEqual(list(get_all_members(cluster, None, role=CtlPostgresqlRole.PRIMARY)), []) + self.assertEqual(list(get_all_members(cluster, None, role=CtlPostgresqlRole.STANDBY_LEADER)), []) + r = list(get_all_members(cluster, None, role=CtlPostgresqlRole.LEADER)) + self.assertEqual(len(r), 1) + self.assertEqual(r[0].name, 'leader') + r = list(get_all_members(get_cluster_initialized_with_leader(), None, role=CtlPostgresqlRole.REPLICA)) self.assertEqual(len(r), 1) self.assertEqual(r[0].name, 'other') @@ -885,6 +903,13 @@ def test_cluster_promote(self, mock_patch): with patch('patroni.dcs.AbstractDCS.get_cluster', Mock(return_value=only_leader_cluster)): result = self.runner.invoke(ctl, ['promote-cluster', 'dummy']) assert 'Cluster is already in the required state' in result.output + # a 'promoted' lock holder means promotion is already initiated + m = Member(0, 'leader', 28, {'conn_url': 'postgres://replicator:rep-pass@127.0.0.1:5435/postgres', + 'role': PostgresqlRole.PROMOTED, 'state': 'running'}) + promoted_cluster = get_cluster('12345678901', Leader(0, 0, m), [m], None, SyncState.empty(), None) + with patch('patroni.dcs.AbstractDCS.get_cluster', Mock(return_value=promoted_cluster)): + result = self.runner.invoke(ctl, ['promote-cluster', 'dummy']) + assert 'Cluster is already in the required state' in result.output # PATCH error mock_patch.return_value.status = 500 result = self.runner.invoke(ctl, ['demote-cluster', 'dummy', '--restore-command', 'foo', '--force']) @@ -900,6 +925,13 @@ def test_cluster_promote(self, mock_patch): patch('time.sleep', Mock()): result = self.runner.invoke(ctl, ['promote-cluster', 'dummy', '--force']) assert result.exit_code == 0 + # success as soon as promotion is initiated (role 'promoted'), without waiting for it to complete + with patch('patroni.dcs.AbstractDCS.get_cluster', Mock(side_effect=[standby_cluster, standby_cluster, + promoted_cluster])), \ + patch('time.sleep', Mock()): + result = self.runner.invoke(ctl, ['promote-cluster', 'dummy', '--force']) + assert result.exit_code == 0 + assert 'cluster is successfully promoted' in result.output class TestPatronictlPrettyTable(unittest.TestCase):

view thread (13+ messages)  latest in thread

Message-ID: <gh-patroni-patroni-3697@github.com>
Permalink:  ../gh-patroni-patroni-3697@github.com/

 ·  · 


This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox