From: guriandoro (@guriandoro) <noreply+guriandoro@github.com>
To: patroni/patroni <noreply+patroni-patroni@github.com>
Subject: [patroni/patroni] PR #3697: Report 'promoted' role in patronictl list and /cluster endpoint
Date: Tue, 18 Aug 2026 21:39:13 +0000
Message-ID: <gh-patroni-patroni-3697@github.com> (raw)
## What problem does this solve?
Fixes the visibility half of #3603 (the log warning is sent as a separate PR).
After a failover, a node can win the leader lock and receive the promote
request, yet Postgres never finishes the promotion because
`restore_command` hangs (in my case: a stalled NFS mount behind
pgBackRest's `archive-get`). Postgres keeps running as a standby,
`pg_is_in_recovery()` stays `true`, and the cluster has no writable node.
Meanwhile `patronictl list` and the `/cluster` REST endpoint show a
healthy `Leader` / `role: leader`, because `cluster_as_json()` hardcodes
the role for whichever member holds the lock. The DCS itself tells the
truth: the member key keeps `"role": "promoted"` until promotion
completes. That signal was being dropped at the presentation layer.
## What does this PR do?
In `cluster_as_json()` (`patroni/utils.py`), when the lock holder's
member key still reports `role: promoted`, pass it through instead of
hardcoding `leader`:
- `patronictl list` shows `Promoted` in the Role column while promotion
has not completed, reverting to `Leader` once it does.
- `GET /cluster` returns `"role": "promoted"` for that member.
Side effect (arguably a second fix): since `replication_state` is only
suppressed for role `leader`, the State column for a stuck member now
shows the actual state (e.g. `in archive recovery`) instead of
`running` — which is exactly what the operator needs to see to find the
hanging `restore_command`.
During a normal failover the `Promoted` role is visible for at most a
cycle or two, same as the `promoted` role in the DCS member key today.
Standby clusters are unaffected (`standby_leader` takes precedence, as
before). All other surfaces (`/patroni`, `/primary`, `/metrics`, the DCS
member key) already report this state correctly and are untouched.
## Repro / verification
Original repro from the issue: 2-node cluster with pgBackRest over NFS,
block NFS return traffic on the replica with iptables, `kill -9` Patroni
on the leader. The replica takes the lock, promote is issued,
`archive-get 00000002.history` hangs forever, `pg_is_in_recovery()`
stays true.
Verified live against this branch (PG 18.4, etcd v3.5, 2-node local
cluster): the hanging `restore_command` was simulated with a script that
sleeps forever on timeline-history requests, then Patroni was `kill -9`'d
on the leader. While node2 was stuck (`pg_is_in_recovery()` = `t`):
```
$ patronictl list
+ Cluster: repro3603 (7675475385382874329) ----------------+----+-------------+-----+------------+-----+
| Member | Host | Role | State | TL | Receive LSN | Lag | Replay LSN | Lag |
+--------+----------------+----------+---------------------+----+-------------+-----+------------+-----+
| node2 | 127.0.0.1:5434 | Promoted | in archive recovery | 1 | | | | |
+--------+----------------+----------+---------------------+----+-------------+-----+------------+-----+
$ curl -s http://127.0.0.1:8009/cluster
{"members": [{"name": "node2", "role": "promoted", "state": "in archive recovery", ...}], "scope": "repro3603"}
```
Once the restore_command hang was released, promotion completed and both
surfaces reverted on the next loop:
```
$ patronictl list
| node2 | 127.0.0.1:5434 | Leader | running | 2 | ...
$ curl -s http://127.0.0.1:8009/cluster
{"members": [{"name": "node2", "role": "leader", "state": "running", ...}], "scope": "repro3603"}
```
## Tests
Extended `test_output_members` in `tests/test_ctl.py`: a cluster whose
lock holder has `role: promoted` in its member data renders `Promoted`
in the Role column. Full suite passes (706 tests). Docs updated:
`docs/rest_api.rst` role enumeration, `cluster_as_json` and
`output_members` docstrings.
inline patch 1: docs/rest_api.rst, patroni/ctl.py, patroni/utils.py and 1 more (+58/-12 lines) download inline-1.patchdiff --git a/docs/rest_api.rst b/docs/rest_api.rstindex ce2f5c04d..0cd68c370 100644--- a/docs/rest_api.rst+++ b/docs/rest_api.rst@@ -479,6 +479,7 @@ Cluster status endpoints
}
}
+ The ``role`` of a member is one of ``leader``, ``promoted``, ``standby_leader``, ``sync_standby``, ``quorum_standby``, or ``replica``. The ``promoted`` role is reported for a member that already holds the leader lock while PostgreSQL promotion has not completed yet, i.e. PostgreSQL is still running in recovery and therefore is read-only.
- The ``GET /history`` endpoint provides a view on the history of cluster switchovers/failovers. The format is very similar to the content of history files in the ``pg_wal`` directory. The only difference is the timestamp field showing when the new timeline was created.
diff --git a/patroni/ctl.py b/patroni/ctl.pyindex a214ebb72..732efcfa1 100644--- a/patroni/ctl.py+++ b/patroni/ctl.py@@ -537,14 +537,16 @@ def get_all_members(cluster: Cluster, group: Optional[int],
if is_citus_cluster() and group is None:
clusters.update(cluster.workers)
if role in (CtlPostgresqlRole.LEADER, CtlPostgresqlRole.PRIMARY, CtlPostgresqlRole.STANDBY_LEADER):
- # In the DCS the members' role can be one among: ``primary``, ``master``, ``replica`` or ``standby_leader``.- # ``primary`` and ``master`` are the same thing.+ # In the DCS the members' role can be one among: ``primary``, ``master``, ``replica``, ``standby_leader``,+ # or ``promoted``. ``primary`` and ``master`` are the same thing. ``promoted`` means the member holds the+ # leader lock but Postgres promotion has not completed yet, so it is neither a primary nor a standby leader.
for cluster in clusters.values():
if cluster.leader is not None and cluster.leader.name and\
(role == CtlPostgresqlRole.LEADER
- or cluster.leader.data.get('role') not in (PostgresqlRole.PRIMARY, PostgresqlRole.MASTER)+ or cluster.leader.data.get('role') not in (PostgresqlRole.PRIMARY, PostgresqlRole.MASTER,+ PostgresqlRole.PROMOTED)
and role == CtlPostgresqlRole.STANDBY_LEADER
- or cluster.leader.data.get('role') != PostgresqlRole.STANDBY_LEADER+ or cluster.leader.data.get('role') not in (PostgresqlRole.STANDBY_LEADER, PostgresqlRole.PROMOTED)
and role == CtlPostgresqlRole.PRIMARY):
yield cluster.leader.member
return
@@ -1593,7 +1595,8 @@ def output_members(cluster: Cluster, name: str, extended: bool = False,
* ``Site``: site of the Patroni node, as per ``site`` configuration;
* ``Member``: name of the Patroni node, as per ``name`` configuration;
* ``Host``: hostname (or IP) and port, as per ``postgresql.listen`` configuration;
- * ``Role``: ``Leader``, ``Standby Leader``, ``Sync Standby`` or ``Replica``;+ * ``Role``: ``Leader``, ``Promoted`` (holds the leader lock but promotion has not completed yet),+ ``Standby Leader``, ``Sync Standby`` or ``Replica``;
* ``State``: one of :class:`~patroni.postgresql.misc.PostgresqlState`;
* ``TL``: current timeline in Postgres;
* ``Receive LSN``: last received LSN (``pg_catalog.pg_last_(xlog|wal)_receive_(location|lsn)()``);
@@ -2367,14 +2370,16 @@ def change_cluster_role(cluster_name: str, force: bool, standby_config: Optional
"""
demote = bool(standby_config)
action_name = 'demot' if demote else 'promot'
- target_role = PostgresqlRole.STANDBY_LEADER if demote else PostgresqlRole.PRIMARY+ # the member key reports role 'promoted' as soon as promotion is initiated with ``pg_ctl promote``, and only+ # switches to 'primary' once Postgres has left recovery; don't make the user wait for the latter+ target_roles = (PostgresqlRole.STANDBY_LEADER,) if demote else (PostgresqlRole.PRIMARY, PostgresqlRole.PROMOTED)
dcs = get_dcs(cluster_name, None)
cluster = dcs.get_cluster()
leader_name = cluster.leader and cluster.leader.name
if not leader_name:
raise PatroniCtlException(f'Cluster has no leader, {action_name}ion is not possible')
- if cluster.leader and cluster.leader.data.get('role') == target_role:+ if cluster.leader and cluster.leader.data.get('role') in target_roles:
raise PatroniCtlException('Cluster is already in the required state')
click.echo('Current cluster topology')
@@ -2404,7 +2409,7 @@ def change_cluster_role(cluster_name: str, force: bool, standby_config: Optional
old_leader = cluster.get_member(leader_name, False)
old_leader_state = old_leader and old_leader.data.get('state')
- if not is_unlocked and leader_role == target_role and leader_state == PostgresqlState.RUNNING:+ if not is_unlocked and leader_role in target_roles and leader_state == PostgresqlState.RUNNING:
if not demote or old_leader_state == PostgresqlState.RUNNING:
click.echo(
f'{datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")} cluster is successfully {action_name}ed')
diff --git a/patroni/utils.py b/patroni/utils.pyindex a8f8ad94e..df4dfc709 100644--- a/patroni/utils.py+++ b/patroni/utils.py@@ -922,7 +922,8 @@ def cluster_as_json(cluster: 'Cluster') -> Dict[str, Any]:
* ``members``: list of members in the cluster. Each value is a :class:`dict` that may have the following keys:
* ``name``: the name of the host (unique in the cluster). The ``members`` list is sorted by this key;
- * ``role``: ``leader``, ``standby_leader``, ``sync_standby``, ``quorum_standby``, or ``replica``;+ * ``role``: ``leader``, ``promoted`` (holds the leader lock but promotion has not completed yet),+ ``standby_leader``, ``sync_standby``, ``quorum_standby``, or ``replica``;
* ``state``: one of :class:`~patroni.postgresql.misc.PostgresqlState`;
* ``api_url``: REST API URL based on ``restapi->connect_address`` configuration;
* ``host``: PostgreSQL host based on ``postgresql->connect_address``;
@@ -948,7 +949,7 @@ def cluster_as_json(cluster: 'Cluster') -> Dict[str, Any]:
* ``to``: name of the member to be promoted.
"""
from . import global_config
- from .postgresql.misc import format_lsn+ from .postgresql.misc import format_lsn, PostgresqlRole
config = global_config.from_cluster(cluster)
leader_name = cluster.leader.name if cluster.leader else None
@@ -958,13 +959,20 @@ def cluster_as_json(cluster: 'Cluster') -> Dict[str, Any]:
sync_role = 'quorum_standby' if config.is_quorum_commit_mode else 'sync_standby'
for m in cluster.members:
if m.name == leader_name:
- role = 'standby_leader' if config.is_standby_cluster else 'leader'+ if config.is_standby_cluster:+ role = 'standby_leader'+ elif m.data.get('role') == PostgresqlRole.PROMOTED:+ # the member holds the leader lock, but Postgres promotion has not completed yet (still in recovery)+ role = 'promoted'+ else:+ role = 'leader'
elif config.is_synchronous_mode and cluster.sync.matches(m.name):
role = sync_role
else:
role = 'replica'
- state = (m.data.get('replication_state', '') if role != 'leader' else '') or m.data.get('state', '')+ state = (m.data.get('replication_state', '') if role not in ('leader', 'promoted') else '')\+ or m.data.get('state', '')
member = {'name': m.name, 'role': role, 'state': state, 'api_url': m.api_url}
conn_kwargs = m.conn_kwargs()
if conn_kwargs.get('host'):
diff --git a/tests/test_ctl.py b/tests/test_ctl.pyindex 5079024e7..3572c6656 100644--- a/tests/test_ctl.py+++ b/tests/test_ctl.py@@ -155,6 +155,14 @@ def test_output_members(self):
self.assertIsNone(output_members(cluster, name='abc'))
self.assertIn('to: site dc2', mock_echo.call_args_list[1][0][0])
+ # the lock holder is shown as 'Promoted' while Postgres promotion has not completed yet+ mock_echo.reset_mock()+ cluster = get_cluster_initialized_with_leader()+ cluster.members[0].data.update(role=PostgresqlRole.PROMOTED, state='running',+ replication_state='in archive recovery')+ self.assertIsNone(output_members(cluster, name='abc', fmt='tsv'))+ self.assertIn('\tPromoted\trunning\t', mock_echo.call_args_list[1][0][0])+
@patch('patroni.dcs.AbstractDCS.set_failover_value', Mock())
def test_switchover(self):
# Confirm
@@ -533,6 +541,16 @@ def test_get_all_members(self):
self.assertEqual(len(r), 1)
self.assertEqual(r[0].name, 'leader')
+ # a lock holder whose promotion has not completed yet is neither a primary nor a standby leader,+ # but it is still the leader (it holds the lock)+ cluster = get_cluster_initialized_with_leader()+ cluster.members[0].data['role'] = PostgresqlRole.PROMOTED+ self.assertEqual(list(get_all_members(cluster, None, role=CtlPostgresqlRole.PRIMARY)), [])+ self.assertEqual(list(get_all_members(cluster, None, role=CtlPostgresqlRole.STANDBY_LEADER)), [])+ r = list(get_all_members(cluster, None, role=CtlPostgresqlRole.LEADER))+ self.assertEqual(len(r), 1)+ self.assertEqual(r[0].name, 'leader')+
r = list(get_all_members(get_cluster_initialized_with_leader(), None, role=CtlPostgresqlRole.REPLICA))
self.assertEqual(len(r), 1)
self.assertEqual(r[0].name, 'other')
@@ -885,6 +903,13 @@ def test_cluster_promote(self, mock_patch):
with patch('patroni.dcs.AbstractDCS.get_cluster', Mock(return_value=only_leader_cluster)):
result = self.runner.invoke(ctl, ['promote-cluster', 'dummy'])
assert 'Cluster is already in the required state' in result.output
+ # a 'promoted' lock holder means promotion is already initiated+ m = Member(0, 'leader', 28, {'conn_url': 'postgres://replicator:rep-pass@127.0.0.1:5435/postgres',+ 'role': PostgresqlRole.PROMOTED, 'state': 'running'})+ promoted_cluster = get_cluster('12345678901', Leader(0, 0, m), [m], None, SyncState.empty(), None)+ with patch('patroni.dcs.AbstractDCS.get_cluster', Mock(return_value=promoted_cluster)):+ result = self.runner.invoke(ctl, ['promote-cluster', 'dummy'])+ assert 'Cluster is already in the required state' in result.output
# PATCH error
mock_patch.return_value.status = 500
result = self.runner.invoke(ctl, ['demote-cluster', 'dummy', '--restore-command', 'foo', '--force'])
@@ -900,6 +925,13 @@ def test_cluster_promote(self, mock_patch):
patch('time.sleep', Mock()):
result = self.runner.invoke(ctl, ['promote-cluster', 'dummy', '--force'])
assert result.exit_code == 0
+ # success as soon as promotion is initiated (role 'promoted'), without waiting for it to complete+ with patch('patroni.dcs.AbstractDCS.get_cluster', Mock(side_effect=[standby_cluster, standby_cluster,+ promoted_cluster])), \+ patch('time.sleep', Mock()):+ result = self.runner.invoke(ctl, ['promote-cluster', 'dummy', '--force'])+ assert result.exit_code == 0+ assert 'cluster is successfully promoted' in result.output
class TestPatronictlPrettyTable(unittest.TestCase):