From: CyberDem0n (@CyberDem0n) <noreply+CyberDem0n@github.com>
To: patroni/patroni <noreply+patroni-patroni@github.com>
Subject: [patroni/patroni] PR #3809: Fix race in priority failover test
Date: Thu, 19 Nov 2026 19:25:40 +0000
Message-ID: <gh-patroni-patroni-3809@github.com> (raw)
The `check higher failover priority is respected` scenario in `priority_failover.feature` is flaky. It failed in [behave (ubuntu, 3.14, raft)](https://github.com/patroni/patroni/actions/runs/44583303686/job/135769421754) with:
```
ASSERT FAILED: postgres-3 role didn't change to primary after 50 seconds
```
### Cause
- `postgres-3` tried to stream before the primary had created its replication slot (`replication slot "postgres_3" does not exist`), then waited `wal_retrieve_retry_interval` (5s) before retrying.
- The scenario's existing checks didn't catch this:
- `add_table` calls `pg_switch_wal()`, so the `replication works` table reached `postgres-3` through `restore_command`.
- `have the same wal position` only compares replay LSNs.
- `postgres-0` was shut down during the retry wait, so only `postgres-2` received the shutdown checkpoint. `postgres-3` logged `Wal position of postgres-2 is ahead of my wal position`, and `postgres-2` (priority 1) took the leader lock. Patroni behaved correctly; the test didn't set up the conditions it assumed.
### Fix
Wait until both replicas report `replication_state=streaming` before comparing WAL positions and shutting down `postgres-0`. This is the same check `site_awareness.feature` uses. The primary doesn't exit until streaming replicas have received the shutdown checkpoint, so both replicas end at the same LSN and failover priority decides which one is promoted.
### Testing
Ran locally with `DCS=raft` and the CI timeout multiplier (PostgreSQL 16):
- **Forced race:** a temporary step (not committed) terminated `postgres-3`'s walsender and dropped its slot, recreating the CI condition.
- Original feature: fails with the same assertion.
- Fixed feature: passes. It waits about 5s for streaming to resume, then `postgres-3` is promoted.
- **Normal runs:** the fixed feature passes repeatedly.
inline patch 1: features/priority_failover.feature (+2/-0 lines) download inline-1.patchdiff --git a/features/priority_failover.feature b/features/priority_failover.featureindex 1fbcd46c9..a4e244675 100644--- a/features/priority_failover.feature+++ b/features/priority_failover.feature@@ -16,6 +16,8 @@ Feature: priority replication
And I configure and start postgres-3 with a tag failover_priority 2
Then replication works from postgres-0 to postgres-2 after 20 seconds
And replication works from postgres-0 to postgres-3 after 20 seconds
+ And Response on GET http://127.0.0.1:8010/patroni contains replication_state=streaming after 10 seconds+ And Response on GET http://127.0.0.1:8011/patroni contains replication_state=streaming after 10 seconds
And postgres-2 and postgres-3 have the same wal position after 10 seconds
When I shut down postgres-0
Then postgres-3 role is the primary after 10 seconds