pg.ddx.io  patroni/patroni GitHub issues and pull requests  
help / color / mirror / Atom feed
[patroni/patroni] issue #3783: pg_rewind fails on a former primary: restore_command is not in postgresql.conf when the node is not a standby
2+ messages / 2 participants
[nested] [flat]

* [patroni/patroni] issue #3783: pg_rewind fails on a former primary: restore_command is not in postgresql.conf when the node is not a standby
@ 2026-11-16 13:18 "mdzhigarov (@mdzhigarov)" <noreply+mdzhigarov@github.com>
  0 siblings, 0 replies; 2+ messages in thread

From: mdzhigarov (@mdzhigarov) @ 2026-11-16 13:18 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

### What happened?

After an unclean stop of the primary, the former primary could not be rewound because `pg_rewind` needed a WAL file that was no longer in its `pg_wal`, only in the archive. `restore_command` is configured in `postgresql.recovery_conf`, but Patroni did not use it for the rewind:

```
INFO: running pg_rewind from pg-b
INFO: pg_rewind exit code=1
INFO:  stderr=pg_rewind: servers diverged at WAL location 0/9000060 on timeline 1
pg_rewind: error: could not open file "/tmp/data/pg_wal/000000010000000000000003": No such file or directory
pg_rewind: error: could not find previous WAL record at 0/3022AB0
ERROR: Failed to rewind from healthy primary: pg-b
INFO: starting as a secondary
...
FATAL:  requested timeline 2 is not a child of this server's history
DETAIL:  Latest checkpoint in file "pg_control" is at 0/D000028 on timeline 1, but in the history of the requested timeline, the server forked off from that timeline at 0/9000060.
```

The node then loops in `restarting after failure in progress` until it is reinitialized manually.

The cause: on PostgreSQL 12+ `restore_command` is written to `postgresql.conf` only inside the `# recovery.conf` block, which `ConfigHandler.write_postgresql_conf()` writes only while the node has recovery parameters, i.e. while it is a standby. A primary that bootstrapped the cluster never had it in its config files, and a former primary loses it on any `postgresql.conf` rewrite after `remove_recovery_conf()`. `Rewind.pg_rewind()` reads `restore_command` with `postgres -C restore_command`, gets an empty value, so it doesn't pass `--restore-target-wal` and doesn't try `_fetch_missing_wal()` either.

On the leader:

```
$ grep -c restore_command $PGDATA/postgresql.conf
0
$ postgres -D $PGDATA -C restore_command

```

On the replica, `restore_command = 'cp /archive/%f %p'` is in the `# recovery.conf` block as expected.

### How can we reproduce it (as minimally and precisely as possible)?

Two members (pg-a, pg-b), a shared archive directory, the configuration below. pg-a bootstraps the cluster.

1. On the leader (pg-a): `CHECKPOINT;`, then a few `INSERT` + `SELECT pg_switch_wal();` so that the checkpoint's WAL file gets archived and the replica replays everything.
2. Make the leader diverge without the replica being able to catch up from the archive: `ALTER SYSTEM SET archive_command = 'false'; SELECT pg_reload_conf();`, `kill -STOP` the walsender, then a few more `INSERT` + `pg_switch_wal()` and a `CHECKPOINT;`. The last checkpoint removes the common checkpoint's WAL file (`000000010000000000000003`) from `pg_wal`. It stays in the archive.
3. `kill -9` Patroni and all postgres processes on pg-a. pg-b gets promoted (timeline 2).
4. Start Patroni on pg-a again. It does crash recovery in single user mode, runs `pg_rewind` and fails as shown above.

### What did you expect to happen?

`pg_rewind` runs with `--restore-target-wal` (or Patroni fetches the missing file with `restore_command`), rewinds the node, and it joins as a replica on timeline 2.

With a small change that also writes `restore_command` to `postgresql.conf` when there are no recovery parameters (PostgreSQL 13+), the same scenario gives:

```
INFO: pg_rewind exit code=0
INFO:  stderr=pg_rewind: servers diverged at WAL location 0/9000060 on timeline 1
pg_rewind: rewinding from last common checkpoint at 0/3020B88 on timeline 1
pg_rewind: Done!
INFO: starting as a secondary
LOG:  restored log file "000000010000000000000003" from archive
...
LOG:  started streaming WAL from primary at 0/9000000 on timeline 2
INFO: no action. I am (pg-a), a secondary, and following a leader (pg-b)
```

I'll open a PR with it.

### Patroni/PostgreSQL/DCS version

- Patroni version: master (3a89e19c, 4.1.5); also seen with 4.1.3
- PostgreSQL version: 18.6
- DCS (and its version): Kubernetes (ConfigMaps)

### Patroni configuration file

```yaml
scope: demo
name: pg-a
restapi:
  listen: 0.0.0.0:8008
  connect_address: 10.244.2.32:8008
kubernetes:
  namespace: patroni-up
  labels: {application: patroni, cluster-name: demo}
  use_endpoints: false
  pod_ip: 10.244.2.32
bootstrap:
  dcs:
    ttl: 30
    loop_wait: 10
    maximum_lag_on_failover: 1073741824
    postgresql:
      use_pg_rewind: true
      use_slots: false
      parameters:
        archive_mode: "on"
        archive_command: "test ! -f /archive/%f && cp %p /archive/%f"
        wal_keep_size: 0MB
        max_wal_size: 1GB
        min_wal_size: 80MB
        checkpoint_timeout: 1h
  initdb: [{encoding: UTF8}, data-checksums]
postgresql:
  listen: 0.0.0.0:5432
  connect_address: 10.244.2.32:5432
  data_dir: /tmp/data
  recovery_conf:
    restore_command: "cp /archive/%f %p"
  pg_hba:
  - local all all trust
  - host all all 0.0.0.0/0 trust
  - host replication all 0.0.0.0/0 trust
  authentication:
    superuser: {username: postgres}
    replication: {username: postgres}
    rewind: {username: postgres}
```

`maximum_lag_on_failover`, `max_wal_size` and `checkpoint_timeout` are set like this only so that the replica is promoted and no extra checkpoint happens in between.

### patronictl show-config

```yaml
loop_wait: 10
maximum_lag_on_failover: 1073741824
postgresql:
  parameters:
    archive_command: test ! -f /archive/%f && cp %p /archive/%f
    archive_mode: 'on'
    checkpoint_timeout: 1h
    max_wal_size: 1GB
    min_wal_size: 80MB
    wal_keep_size: 0MB
  use_pg_rewind: true
  use_slots: false
ttl: 30
```

### Patroni log files

```shell
INFO: doing crash recovery in a single user mode
INFO: Local timeline=1 lsn=0/D000028
INFO: primary_timeline=2
INFO: primary: history=1	0/9000060	no recovery target specified
INFO: running pg_rewind from pg-b
INFO: running pg_rewind from dbname=postgres user=postgres host=10.244.2.33 port=5432 target_session_attrs=read-write
INFO: pg_rewind exit code=1
INFO:  stderr=pg_rewind: servers diverged at WAL location 0/9000060 on timeline 1
pg_rewind: error: could not open file "/tmp/data/pg_wal/000000010000000000000003": No such file or directory
pg_rewind: error: could not find previous WAL record at 0/3022AB0
ERROR: Failed to rewind from healthy primary: pg-b
INFO: starting as a secondary
INFO: restarting after failure in progress
```

### PostgreSQL log files

```shell
LOG:  restored log file "00000002.history" from archive
FATAL:  requested timeline 2 is not a child of this server's history
DETAIL:  Latest checkpoint in file "pg_control" is at 0/D000028 on timeline 1, but in the history of the requested timeline, the server forked off from that timeline at 0/9000060.
```

### Have you tried to use GitHub issue search?

- [x] Yes

### Anything else we need to know?

The same happens if `restore_command` is set in `postgresql.parameters`: it is moved to the recovery parameters and written the same way.


^ permalink  raw  [nested|flat] 2+ messages in thread

* Re: [patroni/patroni] issue #3783: pg_rewind fails on a former primary: restore_command is not in postgresql.conf when the node is not a standby
@ 2026-11-19 17:48 "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
  0 siblings, 0 replies; 2+ messages in thread

From: hughcapet (@hughcapet) @ 2026-11-19 17:48 UTC (permalink / raw)
  To: patroni/patroni <noreply+patroni-patroni@github.com>

@hughcapet closed this.

^ permalink  raw  [nested|flat] 2+ messages in thread


end of thread, other threads:[~2026-11-19 17:48 UTC | newest]

Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-11-16 13:18 [patroni/patroni] issue #3783: pg_rewind fails on a former primary: restore_command is not in postgresql.conf when the node is not a standby "mdzhigarov (@mdzhigarov)" <noreply+mdzhigarov@github.com>
2026-11-19 17:48 Re: [patroni/patroni] issue #3783: pg_rewind fails on a former primary: restore_command is not in postgresql.conf when the node is not a standby "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox