Subject: pg_rewind with cascade standby doesn't work well

From: Kuwamura Masaki <[email-masked]>
Date: Thu, 7 Sep 2023 15:33:45 +0900

Hi there,

I tested pg_rewind behavior and found a suspicious one.

Consider a scenario like this,

Server A: primary
Server B :replica of A
Server C :replica of B

and somehow A down ,so B gets promoted.

Server A: down
Server B :new primary
Server C :replica of B

In this case, pg_rewind can be used to reconstruct the cascade; the source
is C and the target is A.
However, we get error as belows by running pg_rewind.

```
pg_rewind: fetched file "global/pg_control", length 8192
pg_rewind: source and target cluster are on the same timeline
pg_rewind: no rewind required
```
Though A's timeline is 1 and C's is 2 ideally,  it says they're on the same
timeline.

This is because `pg_rewind` currently uses minRecoveryPointTLI and latest
checkpoint's TimelineID to compare the TLI between source and target[1].
Both C's minRecoveryPointTLI and Latest checkpoint's TimelineID are not
modified until checkpointing. (even though B's are modified).
And then, if you run pg_rewind immediately, pg_rewind won't work because C
and A appear to be on the same timeline. So we have to CHECKPOINT on C
before running pg_rewind;

BTW, immediate pg_rewind with cascade standby seems to be already concerned
in another discussion[2], but unfortunately missed.

Anyway, I don't think this behavior is kind.
To fix this, should we use another variable to compare TLI?
Or, modify the cascade standby's minRecoveryPointTLI somehow?

Masaki Kuwamura

[1]
https://www.postgresql.org/message-id/flat/9f568c97-87fe-a716-bd39-65299b8a60f4%40iki.fi
[2]
https://www.postgresql.org/message-id/flat/aeb5f31a-8de2-40a8-64af-ab659a309d6b%40iki.fi

---

From: Kuwamura Masaki <[email-masked]>
Date: Mon, 11 Sep 2023 17:49:46 +0900

> In this case, pg_rewind can be used to reconstruct the cascade; the
source is C and the target is A.
>  pg_rewind: no rewind required

To fix the above mentioned behavior of pg_rewind, I suggest to change the
cascade standby's (i.e. server C's) minRecoveryPointTLI when it receives
the new timeline information from the new primary (i.e. server B).

When server B is promoted, it creates an end-of-recovery record by calling
CreateEndOfRecoveryRecord(). (in xlog.c)
And also updates B's minRecoveryPoint and minRecoveryPointTLI.
```
/*
      * Update the control file so that crash recovery can follow the
timeline
      * changes to this point.
      */
     LWLockAcquire(ControlFileLock, LW_EXCLUSIVE);
     ControlFile->minRecoveryPoint = recptr;
     ControlFile->minRecoveryPointTLI = xlrec.ThisTimeLineID;
     UpdateControlFile();
     LWLockRelease(ControlFileLock);
```
Since C is a replica of B, the end-of-recovery record is replicated from B
to C, so the record is replayed in C by xlog_redo().
The attached patch updates minRecoveryPoint and minRecoveryPointTLI at this
point by mimicking CreateEndOfRecoveryRecord().
With this patch, you can run pg_rewind with cascade standby immediately.
(without waiting for checkpointing)

Thoughts?

Masaki Kuwamura