agora inbox for pgsql-bugs@postgresql.org
help / color / mirror / Atom feedFrom: PG Bug reporting form <noreply@postgresql.org>
To: pgsql-bugs@lists.postgresql.org
Cc: harshit.singh817775@gmail.com
Subject: BUG #19640: Standby permanently stuck re-requesting old timeline after promotion, never switches to new timeline
Date: Wed, 26 Aug 2026 11:56:11 +0000
Message-ID: <19640-3003103a974cd7dc@postgresql.org> (raw)
The following bug has been logged on the website:
Bug reference: 19640
Logged by: Harshit Singh
Email address: harshit.singh817775@gmail.com
PostgreSQL version: 18.0
Operating system: rhel 10
Description:
PostgreSQL version: Reproduced on both 17.6 and 18.0 — present in the
current release, not something already fixed upstream.
Operating system: Linux x86_64 (RHEL-family, built with Red Hat gcc 14.3.1)
Description:
A standby configured with recovery_target_timeline = 'latest' (the default
under streaming-replication HA managers like Patroni) can get permanently
stuck after a timeline promotion elsewhere in the cluster: it endlessly
re-requests WAL on its own old, now-superseded timeline, is told "end of WAL
reached" by the primary each time, disconnects, and immediately reconnects
requesting the same old timeline again — never advancing to the new one.
This repeats indefinitely (observed for over an hour in one case, tight
~10-20ms reconnect loop), consuming CPU, with no error surfaced to indicate
the process needs manual intervention — patronictl/monitoring tooling on top
of it just reports the node as "starting" forever.
Steps to reproduce:
1. 3-node streaming replication cluster (repro used Patroni-managed
Postgres, but the core issue appears to be in core recovery logic, not
Patroni).
2. Node A is primary; other nodes stream from it.
3. Node A is stopped/killed (or even just a plain systemctl restart of the
current leader — no exotic failure needed). Another node is promoted
(timeline N → N+1).
4. Node A is later restarted and attempts to rejoin as a replica of the new
leader.
5. Node A's local timeline is N; the new leader is on N+1.
6. Node A's log shows, repeating forever:
LOG: started streaming WAL from primary at <LSN> on timeline N
LOG: replication terminated by primary server
DETAIL: End of WAL reached on timeline N at <LSN>.
FATAL: terminating walreceiver process due to administrator command
LOG: waiting for WAL to become available at <LSN>
— new walreceiver PID each cycle, always requesting timeline N, never
N+1.
Reproduced 5 times across different sessions/timelines (N=1 through N=4) and
both PostgreSQL 17.6 and 18.0, including once under active write load (real
WAL divergence existed, not just an idle-DB edge case), and via completely
ordinary systemctl stop/start of the current leader — not an exotic
scenario. The fact that it reproduces identically on 18.0 indicates this
isn't a regression already fixed in the latest release.
Expected behavior:
The standby should detect, via rescanLatestTimeLine(), that a newer timeline
(N+1) now exists and switch its target to follow it, per
recovery_target_timeline = 'latest' semantics.
Suspected root cause / prior art:
This looks closely related to the mechanism described by Dilip Kumar on
-hackers in "Race condition in recovery?"
(https://postgrespro.com/list/thread-id/2526828):
WaitForWALToBecomeAvailable() initializes expectedTLEs from receiveTLI
rather than recoveryTargetTLI. When rescanLatestTimeLine() finds the newest
TLE already matches recoveryTargetTLI, it concludes "nothing to change" —
but expectedTLEs is left referencing the old timeline regardless, so every
subsequent WAL request keeps using it. A patch was proposed there
(initializing from recoveryTargetTLI instead) but the thread doesn't show it
as committed, and given this is still reproducible on 18.0, it appears that
patch — or an equivalent fix — never landed.
Additional context:
A near-identical symptom was reported independently against CloudNativePG
(https://github.com/cloudnative-pg/cloudnative-pg/issues/10419), consistent
with this being a core recovery-logic issue rather than something specific
to any one HA orchestration layer.
view thread (2+ messages) latest in thread
Message-ID: <19640-3003103a974cd7dc@postgresql.org>
Permalink: ../19640-3003103a974cd7dc@postgresql.org/
Also on: postgresql.org/message-id/19640-3003103a974cd7dc@postgresql.org
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: noreply@postgresql.org, pgsql-bugs@lists.postgresql.org, harshit.singh817775@gmail.com
Subject: Re: BUG #19640: Standby permanently stuck re-requesting old timeline after promotion, never switches to new timeline
In-Reply-To: <19640-3003103a974cd7dc@postgresql.org>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox