pg.ddx.io  patroni/patroni GitHub issues and pull requests  
help / color / mirror / Atom feed
From: CyberDem0n (@CyberDem0n) <noreply+CyberDem0n@github.com>
To: patroni/patroni <noreply+patroni-patroni@github.com>
Subject: [patroni/patroni] PR #3809: Fix race in priority failover test
Date: Thu, 19 Nov 2026 19:25:40 +0000
Message-ID: <gh-patroni-patroni-3809@github.com> (raw)

The `check higher failover priority is respected` scenario in `priority_failover.feature` is flaky. It failed in [behave (ubuntu, 3.14, raft)](https://github.com/patroni/patroni/actions/runs/44583303686/job/135769421754) with:

```
ASSERT FAILED: postgres-3 role didn't change to primary after 50 seconds
```

### Cause

- `postgres-3` tried to stream before the primary had created its replication slot (`replication slot "postgres_3" does not exist`), then waited `wal_retrieve_retry_interval` (5s) before retrying.
- The scenario's existing checks didn't catch this:
  - `add_table` calls `pg_switch_wal()`, so the `replication works` table reached `postgres-3` through `restore_command`.
  - `have the same wal position` only compares replay LSNs.
- `postgres-0` was shut down during the retry wait, so only `postgres-2` received the shutdown checkpoint. `postgres-3` logged `Wal position of postgres-2 is ahead of my wal position`, and `postgres-2` (priority 1) took the leader lock. Patroni behaved correctly; the test didn't set up the conditions it assumed.

### Fix

Wait until both replicas report `replication_state=streaming` before comparing WAL positions and shutting down `postgres-0`. This is the same check `site_awareness.feature` uses. The primary doesn't exit until streaming replicas have received the shutdown checkpoint, so both replicas end at the same LSN and failover priority decides which one is promoted.

### Testing

Ran locally with `DCS=raft` and the CI timeout multiplier (PostgreSQL 16):

- **Forced race:** a temporary step (not committed) terminated `postgres-3`'s walsender and dropped its slot, recreating the CI condition.
  - Original feature: fails with the same assertion.
  - Fixed feature: passes. It waits about 5s for streaming to resume, then `postgres-3` is promoted.
- **Normal runs:** the fixed feature passes repeatedly.


inline patch 1: features/priority_failover.feature (+2/-0 lines) download inline-1.patchdiff --git a/features/priority_failover.feature b/features/priority_failover.feature index 1fbcd46c9..a4e244675 100644 --- a/features/priority_failover.feature +++ b/features/priority_failover.feature @@ -16,6 +16,8 @@ Feature: priority replication And I configure and start postgres-3 with a tag failover_priority 2 Then replication works from postgres-0 to postgres-2 after 20 seconds And replication works from postgres-0 to postgres-3 after 20 seconds + And Response on GET http://127.0.0.1:8010/patroni contains replication_state=streaming after 10 seconds + And Response on GET http://127.0.0.1:8011/patroni contains replication_state=streaming after 10 seconds And postgres-2 and postgres-3 have the same wal position after 10 seconds When I shut down postgres-0 Then postgres-3 role is the primary after 10 seconds

view thread (7+ messages)  latest in thread

Message-ID: <gh-patroni-patroni-3809@github.com>
Permalink:  ../gh-patroni-patroni-3809@github.com/

 ·  · 


This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox