pg.ddx.io patroni/patroni GitHub issues and pull requests
help / color / mirror / Atom feed[patroni/patroni] PR #3809: Fix race in priority failover test
7+ messages / 3 participants
[nested] [flat]
* [patroni/patroni] PR #3809: Fix race in priority failover test
@ 2026-11-19 19:25 "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
0 siblings, 0 replies; 7+ messages in thread
From: CyberDem0n (@CyberDem0n) @ 2026-11-19 19:25 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
The `check higher failover priority is respected` scenario in `priority_failover.feature` is flaky. It failed in [behave (ubuntu, 3.14, raft)](https://github.com/patroni/patroni/actions/runs/44583303686/job/135769421754) with:
```
ASSERT FAILED: postgres-3 role didn't change to primary after 50 seconds
```
### Cause
- `postgres-3` tried to stream before the primary had created its replication slot (`replication slot "postgres_3" does not exist`), then waited `wal_retrieve_retry_interval` (5s) before retrying.
- The scenario's existing checks didn't catch this:
- `add_table` calls `pg_switch_wal()`, so the `replication works` table reached `postgres-3` through `restore_command`.
- `have the same wal position` only compares replay LSNs.
- `postgres-0` was shut down during the retry wait, so only `postgres-2` received the shutdown checkpoint. `postgres-3` logged `Wal position of postgres-2 is ahead of my wal position`, and `postgres-2` (priority 1) took the leader lock. Patroni behaved correctly; the test didn't set up the conditions it assumed.
### Fix
Wait until both replicas report `replication_state=streaming` before comparing WAL positions and shutting down `postgres-0`. This is the same check `site_awareness.feature` uses. The primary doesn't exit until streaming replicas have received the shutdown checkpoint, so both replicas end at the same LSN and failover priority decides which one is promoted.
### Testing
Ran locally with `DCS=raft` and the CI timeout multiplier (PostgreSQL 16):
- **Forced race:** a temporary step (not committed) terminated `postgres-3`'s walsender and dropped its slot, recreating the CI condition.
- Original feature: fails with the same assertion.
- Fixed feature: passes. It waits about 5s for streaming to resume, then `postgres-3` is promoted.
- **Normal runs:** the fixed feature passes repeatedly.
inline patch 1: features/priority_failover.feature (+2/-0 lines) download inline-1.patch
diff --git a/features/priority_failover.feature b/features/priority_failover.feature
index 1fbcd46c9..a4e244675 100644
--- a/features/priority_failover.feature
+++ b/features/priority_failover.feature
@@ -16,6 +16,8 @@ Feature: priority replication
And I configure and start postgres-3 with a tag failover_priority 2
Then replication works from postgres-0 to postgres-2 after 20 seconds
And replication works from postgres-0 to postgres-3 after 20 seconds
+ And Response on GET http://127.0.0.1:8010/patroni contains replication_state=streaming after 10 seconds
+ And Response on GET http://127.0.0.1:8011/patroni contains replication_state=streaming after 10 seconds
And postgres-2 and postgres-3 have the same wal position after 10 seconds
When I shut down postgres-0
Then postgres-3 role is the primary after 10 seconds
^ permalink raw [nested|flat] 7+ messages in thread
* Re: [patroni/patroni] PR #3809: Fix race in priority failover test
@ 2026-11-19 19:25 "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
5 siblings, 0 replies; 7+ messages in thread
From: CyberDem0n (@CyberDem0n) @ 2026-11-19 19:25 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
@CyberDem0n requested a review.
^ permalink raw [nested|flat] 7+ messages in thread
* Re: [patroni/patroni] PR #3809: Fix race in priority failover test
@ 2026-11-19 19:39 "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
5 siblings, 0 replies; 7+ messages in thread
From: hughcapet (@hughcapet) @ 2026-11-19 19:39 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
@hughcapet applied label "ci".
^ permalink raw [nested|flat] 7+ messages in thread
* Re: [patroni/patroni] PR #3809: Fix race in priority failover test
@ 2026-11-19 19:40 "codecov[bot] (bot) (@codecov[bot])" <noreply+codecov[bot]@github.com>
5 siblings, 0 replies; 7+ messages in thread
From: codecov[bot] (bot) (@codecov[bot]) @ 2026-11-19 19:40 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
## [Codecov](https://app.codecov.io/gh/patroni/patroni/pull/3809?dropdown=coverage&src=pr&el=h1&utm_m...) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:loudspeaker: Thoughts on this report? [Let us know!](https://github.com/codecov/feedback/issues/255)
^ permalink raw [nested|flat] 7+ messages in thread
* Re: [patroni/patroni] PR #3809 (approved): Fix race in priority failover test
@ 2026-11-19 19:48 "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
5 siblings, 0 replies; 7+ messages in thread
From: hughcapet (@hughcapet) @ 2026-11-19 19:48 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
^ permalink raw [nested|flat] 7+ messages in thread
* Re: [patroni/patroni] PR #3809: Fix race in priority failover test
@ 2026-11-19 20:02 "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
5 siblings, 0 replies; 7+ messages in thread
From: hughcapet (@hughcapet) @ 2026-11-19 20:02 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
@hughcapet closed this.
^ permalink raw [nested|flat] 7+ messages in thread
* Re: [patroni/patroni] PR #3809: Fix race in priority failover test
@ 2026-11-19 20:02 "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
5 siblings, 0 replies; 7+ messages in thread
From: hughcapet (@hughcapet) @ 2026-11-19 20:02 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
@hughcapet merged this (commit 428cb461d867).
^ permalink raw [nested|flat] 7+ messages in thread
end of thread, other threads:[~2026-11-19 20:02 UTC | newest]
Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-11-19 19:25 [patroni/patroni] PR #3809: Fix race in priority failover test "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-19 19:25 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-19 19:39 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
2026-11-19 19:40 ` "codecov[bot] (bot) (@codecov[bot])" <noreply+codecov[bot]@github.com>
2026-11-19 19:48 ` Re: [patroni/patroni] PR #3809 (approved): Fix race in priority failover test "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
2026-11-19 20:02 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
2026-11-19 20:02 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox