From: mdzhigarov (@mdzhigarov) <noreply+mdzhigarov@github.com>
To: patroni/patroni <noreply+patroni-patroni@github.com>
Subject: [patroni/patroni] issue #3783: pg_rewind fails on a former primary: restore_command is not in postgresql.conf when the node is not a standby
Date: Mon, 16 Nov 2026 13:18:36 +0000
Message-ID: <gh-patroni-patroni-3783@github.com> (raw)
### What happened?
After an unclean stop of the primary, the former primary could not be rewound because `pg_rewind` needed a WAL file that was no longer in its `pg_wal`, only in the archive. `restore_command` is configured in `postgresql.recovery_conf`, but Patroni did not use it for the rewind:
```
INFO: running pg_rewind from pg-b
INFO: pg_rewind exit code=1
INFO: stderr=pg_rewind: servers diverged at WAL location 0/9000060 on timeline 1
pg_rewind: error: could not open file "/tmp/data/pg_wal/000000010000000000000003": No such file or directory
pg_rewind: error: could not find previous WAL record at 0/3022AB0
ERROR: Failed to rewind from healthy primary: pg-b
INFO: starting as a secondary
...
FATAL: requested timeline 2 is not a child of this server's history
DETAIL: Latest checkpoint in file "pg_control" is at 0/D000028 on timeline 1, but in the history of the requested timeline, the server forked off from that timeline at 0/9000060.
```
The node then loops in `restarting after failure in progress` until it is reinitialized manually.
The cause: on PostgreSQL 12+ `restore_command` is written to `postgresql.conf` only inside the `# recovery.conf` block, which `ConfigHandler.write_postgresql_conf()` writes only while the node has recovery parameters, i.e. while it is a standby. A primary that bootstrapped the cluster never had it in its config files, and a former primary loses it on any `postgresql.conf` rewrite after `remove_recovery_conf()`. `Rewind.pg_rewind()` reads `restore_command` with `postgres -C restore_command`, gets an empty value, so it doesn't pass `--restore-target-wal` and doesn't try `_fetch_missing_wal()` either.
On the leader:
```
$ grep -c restore_command $PGDATA/postgresql.conf
0
$ postgres -D $PGDATA -C restore_command
```
On the replica, `restore_command = 'cp /archive/%f %p'` is in the `# recovery.conf` block as expected.
### How can we reproduce it (as minimally and precisely as possible)?
Two members (pg-a, pg-b), a shared archive directory, the configuration below. pg-a bootstraps the cluster.
1. On the leader (pg-a): `CHECKPOINT;`, then a few `INSERT` + `SELECT pg_switch_wal();` so that the checkpoint's WAL file gets archived and the replica replays everything.
2. Make the leader diverge without the replica being able to catch up from the archive: `ALTER SYSTEM SET archive_command = 'false'; SELECT pg_reload_conf();`, `kill -STOP` the walsender, then a few more `INSERT` + `pg_switch_wal()` and a `CHECKPOINT;`. The last checkpoint removes the common checkpoint's WAL file (`000000010000000000000003`) from `pg_wal`. It stays in the archive.
3. `kill -9` Patroni and all postgres processes on pg-a. pg-b gets promoted (timeline 2).
4. Start Patroni on pg-a again. It does crash recovery in single user mode, runs `pg_rewind` and fails as shown above.
### What did you expect to happen?
`pg_rewind` runs with `--restore-target-wal` (or Patroni fetches the missing file with `restore_command`), rewinds the node, and it joins as a replica on timeline 2.
With a small change that also writes `restore_command` to `postgresql.conf` when there are no recovery parameters (PostgreSQL 13+), the same scenario gives:
```
INFO: pg_rewind exit code=0
INFO: stderr=pg_rewind: servers diverged at WAL location 0/9000060 on timeline 1
pg_rewind: rewinding from last common checkpoint at 0/3020B88 on timeline 1
pg_rewind: Done!
INFO: starting as a secondary
LOG: restored log file "000000010000000000000003" from archive
...
LOG: started streaming WAL from primary at 0/9000000 on timeline 2
INFO: no action. I am (pg-a), a secondary, and following a leader (pg-b)
```
I'll open a PR with it.
### Patroni/PostgreSQL/DCS version
- Patroni version: master (3a89e19c, 4.1.5); also seen with 4.1.3
- PostgreSQL version: 18.6
- DCS (and its version): Kubernetes (ConfigMaps)
### Patroni configuration file
```yaml
scope: demo
name: pg-a
restapi:
listen: 0.0.0.0:8008
connect_address: 10.244.2.32:8008
kubernetes:
namespace: patroni-up
labels: {application: patroni, cluster-name: demo}
use_endpoints: false
pod_ip: 10.244.2.32
bootstrap:
dcs:
ttl: 30
loop_wait: 10
maximum_lag_on_failover: 1073741824
postgresql:
use_pg_rewind: true
use_slots: false
parameters:
archive_mode: "on"
archive_command: "test ! -f /archive/%f && cp %p /archive/%f"
wal_keep_size: 0MB
max_wal_size: 1GB
min_wal_size: 80MB
checkpoint_timeout: 1h
initdb: [{encoding: UTF8}, data-checksums]
postgresql:
listen: 0.0.0.0:5432
connect_address: 10.244.2.32:5432
data_dir: /tmp/data
recovery_conf:
restore_command: "cp /archive/%f %p"
pg_hba:
- local all all trust
- host all all 0.0.0.0/0 trust
- host replication all 0.0.0.0/0 trust
authentication:
superuser: {username: postgres}
replication: {username: postgres}
rewind: {username: postgres}
```
`maximum_lag_on_failover`, `max_wal_size` and `checkpoint_timeout` are set like this only so that the replica is promoted and no extra checkpoint happens in between.
### patronictl show-config
```yaml
loop_wait: 10
maximum_lag_on_failover: 1073741824
postgresql:
parameters:
archive_command: test ! -f /archive/%f && cp %p /archive/%f
archive_mode: 'on'
checkpoint_timeout: 1h
max_wal_size: 1GB
min_wal_size: 80MB
wal_keep_size: 0MB
use_pg_rewind: true
use_slots: false
ttl: 30
```
### Patroni log files
```shell
INFO: doing crash recovery in a single user mode
INFO: Local timeline=1 lsn=0/D000028
INFO: primary_timeline=2
INFO: primary: history=1 0/9000060 no recovery target specified
INFO: running pg_rewind from pg-b
INFO: running pg_rewind from dbname=postgres user=postgres host=10.244.2.33 port=5432 target_session_attrs=read-write
INFO: pg_rewind exit code=1
INFO: stderr=pg_rewind: servers diverged at WAL location 0/9000060 on timeline 1
pg_rewind: error: could not open file "/tmp/data/pg_wal/000000010000000000000003": No such file or directory
pg_rewind: error: could not find previous WAL record at 0/3022AB0
ERROR: Failed to rewind from healthy primary: pg-b
INFO: starting as a secondary
INFO: restarting after failure in progress
```
### PostgreSQL log files
```shell
LOG: restored log file "00000002.history" from archive
FATAL: requested timeline 2 is not a child of this server's history
DETAIL: Latest checkpoint in file "pg_control" is at 0/D000028 on timeline 1, but in the history of the requested timeline, the server forked off from that timeline at 0/9000060.
```
### Have you tried to use GitHub issue search?
- [x] Yes
### Anything else we need to know?
The same happens if `restore_command` is set in `postgresql.parameters`: it is moved to the recovery parameters and written the same way.