Subject: pg_stat_replication.*_lag sometimes shows NULL during active replication

From: Shinya Kato <[email-masked]>
Date: Tue, 24 Feb 2026 15:53:54 +0900

Hi hackers,

I have noticed that pg_stat_replication.*_lag sometimes shows NULL
when inserting a record per second for health checking. This happens
when the startup process replays WAL fast enough before the
walreceiver sends its flush notification to the walsender.

Here is the sequence that triggers the issue: (See normal.svg and
error.svg for diagrams of the normal and problematic cases.)

1. The walreceiver receives, writes, and flushes WAL, then wakes the
startup process via WakeupRecovery().

2. The startup process replays all available WAL quickly, then calls
WalRcvForceReply() to set force_reply = true and wakes the
walreceiver.

3. The walreceiver sends a flush notification to the walsender
(XLogWalRcvSendReply() in XLogWalRcvFlush()). Since the startup has
already replayed the WAL by this point, this message reports the
incremented applyPtr, which equals sentPtr. The walsender processes
this message, consuming the LagTracker samples and setting
fullyAppliedLastTime = true.

4. In the next loop iteration, the walreceiver sees force_reply = true
and sends another reply with the same positions. The walsender sees
applyPtr == sentPtr for the second consecutive time and sets
clearLagTimes = true. Since the LagTracker samples were already
consumed by step 3, all lag values are -1. With clearLagTimes = true,
these -1 values are written to walsnd->*Lag, causing
pg_stat_replication to show NULL.

The comment in ProcessStandbyReplyMessage() says:

     * If the standby reports that it has fully replayed the WAL in two
     * consecutive reply messages, then the second such message must result
     * from wal_receiver_status_interval expiring on the standby.

But as shown above, the second message can also come from
WalRcvForceReply(), violating this assumption.

The attached patch fixes this by adding a check that all lag values
are -1 to the clearLagTimes condition. This ensures that clearLagTimes
only triggers when there are truly no new lag samples in two
consecutive messages (i.e., the system is genuinely idle), and not
when the samples were simply consumed by a preceding message in a
burst of replies.

Regards,

-- 
Best regards,
Shinya Kato
NTT OSS Center

---

From: Fujii Masao <[email-masked]>
Date: Mon, 2 Mar 2026 23:44:05 +0900

> when the samples were simply consumed by a preceding message in a

Thanks for the patch!

With the patch applied, I set up a logical replication and inserted a row every
second. Even with continuous inserts, NULL was shown in the lag columns of
pg_stat_replication. That makes me wonder whether the patch's approach is
sufficient to address the issue.

Relying solely on replies from the standby or subscriber seems a bit fragile to
me. If the goal is to keep showing the last measured lag for some time,
perhaps we should introduce a rate limit on when NULL is displayed in the lag
columns?

For example, if there has been no activity (i.e., sentPtr == applyPtr and
applyPtr has not changed since the previous cycle) for, say, 10 seconds,
then we could allow NULL to be shown. Thought?

Regards,

-- 
Fujii Masao

---

From: Shinya Kato <[email-masked]>
Date: Fri, 6 Mar 2026 16:12:48 +0900

> sufficient to address the issue.

Thank you for the review and testing! I had only considered the issue
in the context of physical replication, but as you pointed out, my
approach is insufficient for logical replication.

> perhaps we should introduce a rate limit on when NULL is displayed in the lag

My primary goal was to ensure that the source code comments match the
actual behavior, as the comment stating "the second such message must
result from wal_receiver_status_interval expiring on the standby" is
inaccurate. However, as you noted, the patch alone is not sufficient
to fully address the issue.

> then we could allow NULL to be shown. Thought?

I considered a time-based rate limit, but it is difficult to choose an
appropriate threshold. Furthermore, the walsender has no way of
knowing the standby's or subscriber's wal_receiver_status_interval
setting.

The attached v2 patch takes a different approach: it additionally
requires that all reported positions (write/flush/apply) remain
unchanged from the previous reply. This directly detects a truly idle
system without relying on timeouts—if any position has advanced, new
WAL activity must have occurred, so we should not clear the lag values
even if the lag tracker is empty.
--
Best regards,
Shinya Kato
NTT OSS Center

---

From: Fujii Masao <[email-masked]>
Date: Mon, 9 Mar 2026 20:21:03 +0900

> even if the lag tracker is empty.

This approach looks good to me.

One comment: currently, the lag becomes NULL basically after about one
wal_receiver_status_interval during periods of no activity. OTOH, with this
approach, it seems it would take about twice wal_receiver_status_interval.
Is this understanding correct?

Regards,

-- 
Fujii Masao

---

From: Shinya Kato <[email-masked]>
Date: Tue, 10 Mar 2026 10:01:45 +0900

> This approach looks good to me.

Thank you for looking into this.

> Is this understanding correct?

Exactly. With this patch, it takes about two
wal_receiver_status_interval cycles to show NULL instead of one. I
think this is an acceptable trade-off because it is better to take a
bit longer to detect inactivity than to incorrectly show NULL during
active replication.

-- 
Best regards,
Shinya Kato
NTT OSS Center

---

From: Fujii Masao <[email-masked]>
Date: Tue, 10 Mar 2026 10:54:13 +0900

> bit longer to detect inactivity than to incorrectly show NULL during

Even with your latest patch, if we remove fullyAppliedLastTime, and set
clearLagTimes to true when applyPtr == sentPtr && noLagSamples &&
positionsUnchanged,
wouldn't the time for the lag to become NULL be almost the same as
wal_receiver_status_interval?

The documentation doesn't clearly specify how long it should take for
the lag to become NULL, so doubling that time might be acceptable.
However, if we can keep it roughly the same without much complexity,
I think that would be preferable.

Thought?

-- 
Fujii Masao

---

From: Shinya Kato <[email-masked]>
Date: Wed, 11 Mar 2026 11:38:23 +0900

> I think that would be preferable.

Thank you for the suggestion. I tested this by removing
fullyAppliedLastTime, but even with synchronous replication, NULL
still appears. Here is why:

- Reply 1 (flush notification): positions = X. Lag samples are
consumed with real values, so noLagSamples = false. clearLagTimes is
not set, and prevPtrs = X is saved.

- Reply 2 (force_reply): positions = X again. Here, noLagSamples =
true and positionsUnchanged = true. Since applyPtr == sentPtr,
clearLagTimes is set to true, resulting in a NULL value.

Therefore, I believe fullyAppliedLastTime is still necessary to ensure
that the previous reply also contained no lag samples.

BTW I noticed an incorrect comment in walreceiver.c and have included
a fix for it. Patch 0001 remains unchanged.

-- 
Best regards,
Shinya Kato
NTT OSS Center

---

From: Fujii Masao <[email-masked]>
Date: Fri, 13 Mar 2026 00:27:32 +0900

> that the previous reply also contained no lag samples.

Thanks for testing and for the clarification! You're right.

However, if we apply this change, the time required for the lag information to
be reset would effectively double. I start wondering if that's really
acceptable, especially for back branches. Although the docs doesn't clearly
specify this timing, doubling it could affect systems that monitor
replication lag, for example. It might still be reasonable to apply
such a change in master, though.

On further thought, the root cause seems to be that walreceiver can send
two consecutive status reply messages with identical WAL locations even
when wal_receiver_status_interval has not yet elapsed. Addressing that
behavior directly might resolve the issue you reported. I've attached a PoC
patch that does this. Thought?

Regards,

-- 
Fujii Masao

---

From: Chao Li <[email-masked]>
Date: Fri, 13 Mar 2026 10:15:46 +0800

> On Mar 12, 2026, at 23:27, Fujii Masao <[email-masked]> wrote:
> 
> <v4-0001-Avoid-sending-duplicate-WAL-locations-in-standby-.patch>

I just read v4. The approach looks good to me overall. I have a few comments about the naming.

This patch changes the old force reply logic to an applied-location-driven reply. Now a reply is sent only if the applied location has advanced. However, this applied-location-driven reply is still triggered from WalRcvForceReply(), so the function has effectively lost its original “force” semantics. Because of that, it might be better to rename WalRcvForceReply() to something like WalRcvRequestApplyStatusUpdate().

Then,
```
static void
XLogWalRcvSendReply(bool force, bool requestReply, bool replyApply)
```

replyApply reads like “send an apply reply”, but in reality it indicates that the applied location should be checked to decide whether to send the reply. So it might be clearer to rename it to something like checkApplyStatus.

Lastly,
```
    sig_atomic_t reply_apply; /* used as a bool */
```

reply_apply sounds like an action of “reply with apply”, but what it actually represents is that the startup process requested an applied-location-driven reply. If applied location is not advanced, the reply won’t be sent. So a name like apply_update_requested might better reflect the meaning.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/

---

From: Shinya Kato <[email-masked]>
Date: Mon, 16 Mar 2026 09:25:52 +0900

> such a change in master, though.

Yes, I agree. Doubling the lag reset time should be avoided in back
branches if possible.

> patch that does this. Thought?

Thank you for the v4 patch. I think this approach is better than mine.
I tested the patch and confirmed that the issue no longer reproduces
with physical replication. However, with logical replication, the lag
columns in pg_stat_replication still show NULL periodically at
wal_receiver_status_interval, since send_feedback() in worker.c can
still send duplicate positions.

+ * previsou update, i.e., when 'replyApply' is true.

One minor thing: there is a typo "previsou". It should be "previous".

-- 
Best regards,
Shinya Kato
NTT OSS Center

---

From: Fujii Masao <[email-masked]>
Date: Tue, 17 Mar 2026 11:00:38 +0900

> still send duplicate positions.

I was thinking that if a feedback message triggered by
wal_receiver_status_interval has the same LSNs as the previous message,
it's expected for the lag columns to become NULL. But you see it differently,
don't you? Sorry, I failed to understand your point...

Regards,

-- 
Fujii Masao

---

From: Shinya Kato <[email-masked]>
Date: Thu, 19 Mar 2026 22:58:18 +0900

> don't you? Sorry, I failed to understand your point...

Sorry for the confusion. I ran a script inserting one row every 0.5
seconds under logical replication and confirmed that NULL still
appears in the lag columns even while replication is actively running.
I was initially mistaken that this was tied to
wal_receiver_status_interval timing — that turned out to be unrelated.

I haven't had time to investigate further, but my current impression
is that the existing approach may not be sufficient for logical
replication.

-- 
Best regards,
Shinya Kato
NTT OSS Center

---

From: Fujii Masao <[email-masked]>
Date: Fri, 20 Mar 2026 02:13:25 +0900

> is that the existing approach may not be sufficient for logical

Thanks for the clarification! I understand your point now.

I think the issue occurs when the positions in the first message point to
the same LSN (e.g., 0/030D5230), and the second message reports the same but
larger LSN (e.g., 0/030D52E0).

I've updated the patch to address this. It removes fullyAppliedLastTime,
tracks the positions from the previous reply, and clears the lag values only
when the positions remain unchanged across two consecutive messages.

Patch attached. Could you test and review this updated patch?

Regards,

-- 
Fujii Masao

---

From: Shinya Kato <[email-masked]>
Date: Sat, 21 Mar 2026 11:04:32 +0900

> larger LSN (e.g., 0/030D52E0).

Thanks for the explanation!

> Patch attached. Could you test and review this updated patch?

The patch works properly. I think it looks nice to me, except for the
typo I sent in the previous message.

-- 
Best regards,
Shinya Kato
NTT OSS Center

---

From: Fujii Masao <[email-masked]>
Date: Tue, 24 Mar 2026 00:31:10 +0900

> typo I sent in the previous message.

Thanks for the review!

I've fixed the typo and attached an updated patch. I also incorporated
Chao's comments from upthread. I'm planning to commit this to master.

As for backpatching, I'm hesitant to backpatch the full patch since it may
reduce the number of replication feedback messages, which feels too invasive
for stable branches.

That said, the patch's changes in walsender.c could be backpatched.
As discussed earlier, they don't fully address the reported issue,
but they do help mitigate cases where lag becomes NULL unexpectedly
in logical replication. So it might be worth considering those changes
for stable branches.

Thoughts?

Regards,

-- 
Fujii Masao

---

From: Chao Li <[email-masked]>
Date: Tue, 24 Mar 2026 13:32:00 +0800

> On Mar 23, 2026, at 23:31, Fujii Masao <[email-masked]> wrote:
> 
> <v6-0001-Avoid-sending-duplicate-WAL-locations-in-standby-.patch>

Thank you for updating the patch. I saw that the variable name and function name were changed to reflect my earlier comments.

v6 looks good to me.

Best regards,

---

From: Shinya Kato <[email-masked]>
Date: Wed, 25 Mar 2026 16:02:53 +0900

> in logical replication. So it might be worth considering those changes

Thanks for the updated patch. LGTM.

Regarding the backpatch, I'd personally appreciate it if the walsender.c
changes were backpatched to stable branches. As you noted, it don't fully
solve the reported issue, but they do help reduce the cases where lag
columns in pg_stat_replication unexpectedly become NULL.

Even a partial mitigation in the back branches would be valuable for users
running stable releases.

--
Best regards,
Shinya Kato

---

From: Fujii Masao <[email-masked]>
Date: Thu, 26 Mar 2026 00:27:37 +0900

Thanks for the review!

Regards,

-- 
Fujii Masao

---

From: Fujii Masao <[email-masked]>
Date: Thu, 26 Mar 2026 00:30:42 +0900

On Wed, Mar 25, 2026 at 4:03 PM Shinya Kato <[email-masked]> wrote:
> Regarding the backpatch, I'd personally appreciate it if the walsender.c changes were backpatched to stable branches. As you noted, it don't fully solve the reported issue, but they do help reduce the cases where lag columns in pg_stat_replication unexpectedly become NULL.
>
> Even a partial mitigation in the back branches would be valuable for users running stable releases.

+1

I've split the changes into two patches.

Patch 0001 fixes premature NULL lag reporting in walsender. I will commit it
and backpatch it to all supported branches.

Patch 0002 avoids sending duplicate WAL locations in standby status replies.
I will commit this to master only.

Regards,

-- 
Fujii Masao

---

From: Shinya Kato <[email-masked]>
Date: Thu, 26 Mar 2026 14:40:37 +0900

> I will commit this to master only.

Thanks, LGTM.

Best regards,
Shinya Kato

---

From: Fujii Masao <[email-masked]>
Date: Thu, 26 Mar 2026 20:56:35 +0900

> Thanks, LGTM.

I've pushed the patches. Thanks!

Regards,

-- 
Fujii Masao