Subject: Re: BF mamba failure

From: Michael Paquier <[email-masked]>
Date: Fri, 15 Nov 2024 13:45:24 +0900

On Thu, Nov 14, 2024 at 10:07:41AM +0000, Bertrand Drouvot wrote:
> === 1
> 
> Maybe use "generation" instead of generation in the comments (where it's not done,
> some comments do it already).

I've tweaked things to be more consistency, and applied that down to 15.

> === 2
> 
> We could think about adding a test. That should be doable with replication slots
> or custom pgstats kinds and probably injection points. But I'm not sure that's
> worth it, as the code in the patch looks "simple" enough. Thoughts?

The tricky part is that this path involves a backend shutdown.  We
should be able to do something with a wait before the dshash_find() in
pgstat_release_entry_ref() with a BackgroundPsql object that gets
stopped, but I'm not sure if we can guarantee a lookup of
pg_stat_activity at this stage.  Let me see..
--
Michael

---

From: Kouber Saparev <[email-masked]>
Date: Mon, 4 Aug 2025 13:00:35 +0300

We just had the same sudden replica shutdown, this time with version 17.3.

2025-08-02 22:10:02.229 UTC,,,473966,,67b3d76c.73b6e,14,,2025-02-18
00:42:20 UTC,8111/0,0,FATAL,XX000,"trying to drop stats entry already
dropped: kind=relation dboid=16420 objoid=4169049057 refcount=1",,,,,"WAL
redo at 6337E/9FF275E8 for Transaction/COMMIT: 2025-08-02
22:10:02.163756+00; dropped stats: 2/16420/4169049057 2/16420/4169049056
2/16420/4169049053 2/16420/4169049071 2/16420/4169049070 2/16420/4169049068
2/16420/4169049064 2/16420/4169049154 2/16420/4169049153 2/16420/4169049152
2/16420/4169049149; inval msgs: catcache 80 c
atcache 79 catcache 80 catcache 79 catcache 55 catcache 54 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 55 catcache 54 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 55 catcache 54
catcache 7 catcache 6 catcache 7 catcache 6 catcache 32 cat
cache 55 catcache 54 catcache 55 catcache 54 catcache 55 catcache 54
catcache 32 catcache 7 catcache 6 catcache 7 catcache 6 catcache 55
catcache 54 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 55 catcache 54 catcache 80 catcache 79 catcache 80 catcache 79
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache
7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 55 catcache
54 catcache 80 catcache 79 catcache 80 catcache 79 catcache 55 catcache 54
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache
7 catcache 6 catcache 7 catcache 6 catcache
7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 55
catcache 54 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 ca
tcache 6 catcache 55 catcache 54 catcache 7 catcache 6 catcache 7 catcache
6 catcache 32 catcache 55 catcache 54 catcache 55 catcache 54 catcache 55
catcache 54 catcache 55 catcache 54 catcache 7 catcache 6 catcache 32
catcache 55 catcache 54 catcache 55 catcache 54 cat
cache 32 catcache 7 catcache 6 catcache 55 catcache 54 catcache 32 catcache
7 catcache 6 catcache 7 catcache 6 catcache 55 catcache 54 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 7 catcache 6 cat
cache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 55
catcache 54 catcache 80 catcache 79 catcache 80 catcache 79 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 7 catcache 6 cat
cache 7 catcache 6 catcache 7 catcache 6 catcache 55 catcache 54 catcache
80 catcache 79 catcache 80 catcache 79 catcache 55 catcache 54 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 7 catcache 6 c
atcache 7 catcache 6 catcache 7 catcache 6 catcache 55 catcache 54 catcache
7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 7 catcache 6 catca
che 55 catcache 54 catcache 7 catcache 6 catcache 7 catcache 6 catcache 32
catcache 55 catcache 54 catcache 55 catcache 54 catcache 55 catcache 54
catcache 55 catcache 54 catcache 7 catcache 6 catcache 32 catcache 55
catcache 54 catcache 55 catcache 54 catcache 32 catca
che 7 catcache 6 catcache 55 catcache 54 catcache 32 catcache 7 catcache 6
catcache 7 catcache 6 catcache 55 catcache 54 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache
7 catcache 6 catcache 7 catcache 6 catcache 7 catcac
he 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 55 catcache 54
catcache 80 catcache 79 catcache 80 catcache 79 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache
7 catcache 6 catcache 7 catcache 6 catcache 7 catcac
he 6 catcache 7 catcache 6 catcache 55 catcache 54 snapshot 2608 relcache
4169049053 snapshot 1214 relcache 4169049056 relcache 4169049057 relcache
4169049056 snapshot 2608 relcache 4169049056 relcache 4169049057 relcache
4169049053 snapshot 2608 relcache 4169049057 rel
cache 4169049056 snapshot 2608 relcache 4169049056 snapshot 2608 snapshot
2608 snapshot 2608 relcache 4169049053 snapshot 2608 snapshot 1214 snapshot
2608 relcache 4169049064 snapshot 1214 relcache 4169049068 relcache
4169049070 relcache 4169049068 snapshot 2608 relcach
e 4169049068 relcache 4169049070 relcache 4169049064 snapshot 2608 relcache
4169049071 relcache 4169049064 snapshot 2608 relcache 4169049064 relcache
4169049071 relcache 4169049071 relcache 4169049064 snapshot 2608 relcache
4169049070 relcache 4169049068 snapshot 2608 r
elcache 4169049068 snapshot 2608 snapshot 2608 snapshot 2608 relcache
4169049064 snapshot 2608 snapshot 1214 snapshot 2608 relcache 4169049149
snapshot 1214 relcache 4169049152 relcache 4169049153 relcache 4169049152
snapshot 2608 relcache 4169049152 relcache 4169049153
 relcache 4169049149 snapshot 2608 relcache 4169049154 relcache 4169049149
snapshot 2608 relcache 4169049149 relcache 4169049154 relcache 4169049154
relcache 4169049149 snapshot 2608 relcache 4169049153 relcache 4169049152
snapshot 2608 relcache 4169049152 snapshot 2608
 snapshot 2608 snapshot 2608 relcache 4169049149 snapshot 2608 snapshot
1214",,,,"","startup",,0
2025-08-02 22:10:02.389 UTC,,,473962,,67b3d76b.73b6a,27,,2025-02-18
00:42:19 UTC,,0,LOG,00000,"startup process (PID 473966) exited with exit
code 1",,,,,,,,,"","postmaster",,0
2025-08-02 22:10:02.389 UTC,,,473962,,67b3d76b.73b6a,28,,2025-02-18
00:42:19 UTC,,0,LOG,00000,"terminating any other active server
processes",,,,,,,,,"","postmaster",,0
2025-08-02 22:10:02.516 UTC,,,473962,,67b3d76b.73b6a,29,,2025-02-18
00:42:19 UTC,,0,LOG,00000,"shutting down due to startup process
failure",,,,,,,,,"","postmaster",,0
2025-08-02 22:10:02.563 UTC,,,473962,,67b3d76b.73b6a,30,,2025-02-18
00:42:19 UTC,,0,LOG,00000,"database system is shut
down",,,,,,,,,"","postmaster",,0

---

From: Michael Paquier <[email-masked]>
Date: Thu, 7 Aug 2025 08:18:26 +0900

> redo at 6337E/9FF275E8 for Transaction/COMMIT: 2025-08-02

You are close to wraparound for this relation.  Is that vanilla
streaming replication with heap?
--
Michael

---

From: Bertrand Drouvot <[email-masked]>
Date: Thu, 7 Aug 2025 05:28:17 +0000

Hi,

> streaming replication with heap?

I was wondering about adding the "generation" (introduced in 818119afccd3) to
the above error log message, thoughts?

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

---

From: Michael Paquier <[email-masked]>
Date: Thu, 7 Aug 2025 16:17:29 +0900

> the above error log message, thoughts?

Yes, that could prove to be useful when debugging such issues.  We do
not have a lot of information here.
--
Michael

---

From: Bertrand Drouvot <[email-masked]>
Date: Thu, 7 Aug 2025 08:35:17 +0000

Hi,

> not have a lot of information here.

Yes, done in the attached.

Regards,

---

From: Michael Paquier <[email-masked]>
Date: Fri, 8 Aug 2025 09:09:14 +0900

> Yes, done in the attached.

Thanks.  With the next minor release happening next week, done that
now so as we will able to gather more information from the field.
--
Michael

---

From: Kouber Saparev <[email-masked]>
Date: Fri, 8 Aug 2025 12:04:52 +0300

Indeed, nothing exotic about our replication.

As for the object 4169049057, I am not able to find it anywhere in the
catalogs. Perhaps it was something that was dropped in the meantime.

---

From: Kouber Saparev <[email-masked]>
Date: Tue, 9 Sep 2025 16:07:45 +0300

Yet again one of our replicas died. Should I file a bug report or
something, what should we do in order to prevent it? Restart the database
every month/week or so?...

2025-09-09 12:08:10.688 UTC,,,1510554,,67b3d8bc.170c9a,337,,2025-02-18
00:47:56 UTC,8111/0,0,FATAL,XX000,"trying to drop stats entry already
dropped: kind=relation dboid=16420 objoid=767325170 refcount=2"
,,,,,"WAL redo at 66E4F/74A0F210 for Transaction/COMMIT: 2025-09-09
12:08:10.657171+00; dropped stats: 2/16420/767325174 2/16420/767325173
2/16420/767325170 2/16420/767325192 2/16420/767325191 2/16420/767
325189 2/16420/767325186 2/16420/767325198 2/16420/767325197
2/16420/767325196 2/16420/767325193; inval msgs: catcache 80 catcache 79
catcache 80 catcache 79 catcache 55 catcache 54 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache
7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 55 catcache 54 catcache 7 catcache 6 catc
ache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 55 catcache 54 catcache
7 catcache 6 catcache 7 catcache 6 catcache 32 catcache 55 catcache 54
catcache 55 catcache 54 catcache 55 catcache 54 catcache 32 catcache 7
catcache 6 catcache 7 catcache 6 catcache 55 catcache 54 catc
ache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 5
5 catcache 54 catcache 80 catcache 79 catcache 80 catcache 79 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 55 catcache 54 catcache 80
catcache 79 catcache 80 catcache 79 catcache 55 catcache 54 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache
7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 55 catcache 54 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 c
atcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 55 catcache 54
catcache 7 catcache 6 catcache 7 catcache 6 catcache 32 catc
ache 55 catcache 54 catcache 55 catcache 54 catcache 55 catcache 54
catcache 55 catcache 54 catcache 7 catcache 6 catcache 32 catcache 55
catcache 54 catcache 55 catcache 54 catcache 32 catcache 7 catcach
e 6 catcache 55 catcache 54 catcache 32 catcache 7 catcache 6 catcache 7
catcache 6 catcache 55 catcache 54 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache
7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 55 catcache 54 catcache 80
catcache 79 catcache 80 catcache 79 catcache 7 catcache 6 catcache
7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 55 catcache 54 catcache 80 catcache 79 catcache 8
0 catcache 79 catcache 55 catcache 54 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 c
atcache 6 catcache 55 catcache 54 catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catca
che 6 catcache 7 catcache 6 catcache 55 catcache 54 catcache 7 catcache 6
catcache 7 catcache 6 catcache 32 catcache 55 catcache 54 catcache 55
catcache 54 catcache 55 catcache 54 catcache 55 catcache 54
catcache 7 catcache 6 catcache 32 catcache 55 catcache 54 catcache 55
catcache 54 catcache 32 catcache 7 catcache 6 catcache 55 catcache 54
catcache 32 catcache 7 catcache 6 catcache 7 catcache 6 catcache
55 catcache 54 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 c
atcache 6 catcache 55 catcache 54 catcache 80 catcache 79 catcache 80
catcache 79 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 c
atcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 55 catcache
54 snapshot 2608 relcache 767325170 snapshot 1214 relcache 767325173
relcache 767325174 relcache 767325173 snapshot 2608 relcache
767325173 relcache 767325174 relcache 767325170 snapshot 2608 relcache
767325174 relcache 767325173 snapshot 2608 relcache 767325173 snapshot 2608
snapshot 2608 snapshot 2608 relcache 767325170 snapshot
2608 snapshot 1214 snapshot 2608 relcache 767325186 snapshot 1214 relcache
767325189 relcache 767325191 relcache 767325189 snapshot 2608 relcache
767325189 relcache 767325191 relcache 767325186 snapshot 2
608 relcache 767325192 relcache 767325186 snapshot 2608 relcache 767325186
relcache 767325192 relcache 767325192 relcache 767325186 snapshot 2608
relcache 767325191 relcache 767325189 snapshot 2608 relcac
he 767325189 snapshot 2608 snapshot 2608 snapshot 2608 relcache 767325186
snapshot 2608 snapshot 1214 snapshot 2608 relcache 767325193 snapshot 1214
relcache 767325196 relcache 767325197 relcache 76732519
6 snapshot 2608 relcache 767325196 relcache 767325197 relcache 767325193
snapshot 2608 relcache 767325198 relcache 767325193 snapshot 2608 relcache
767325193 relcache 767325198 relcache 767325198 relcache
767325193 snapshot 2608 relcache 767325197 relcache 767325196 snapshot 2608
relcache 767325196 snapshot 2608 snapshot 2608 snapshot 2608 relcache
767325193 snapshot 2608 snapshot 1214",,,,"","startup",,0

---

From: Michael Paquier <[email-masked]>
Date: Thu, 11 Sep 2025 08:28:35 +0900

> every month/week or so?...

I don't think we need another bug to report the same problem.  We are
aware that this may be an issue and that it is hard to track, the
problem is to find room to be able to investigate it, at this stage.

I may be able to come back to it soon-ishly, looking at how to trigger
any race condition.  The difficulty is to think how the current code
is able to reach this state, because we have a race condition at hand
in standbys.

As a start, are these failures only in the startup process?  Has the
startup process reached a consistent state when the problem happens
because the replay code is too eager at removing the stats entries?
Has it not reached a consistent state.  These could be useful hints to
extract a reproducible test case, looking for common patterns.

I'll ask around if I have seen cases like that in the user pool I have
an access to.
--
Michael

---

From: Kouber Saparev <[email-masked]>
Date: Thu, 11 Sep 2025 16:35:01 +0300

> extract a reproducible test case, looking for common patterns.

The pattern is the same, although I am not 100% sure that the same replica
is having this - it is a cascaded streaming replica, i.e. a replica of
another replica. Once we had this in October 2024, with version 15.4, then
in August 2025 with 17.3, and now in September again (17.3). The database
is working for month(s) perfectly fine in a heavy production workload (lots
of WALs etc.), and then all of a sudden it shuts down.

Thanks for the feedback, and let me know if I could provide any additional
info.

--
Kouber

---

From: Michael Paquier <[email-masked]>
Date: Fri, 12 Sep 2025 09:37:30 +0900

> of WALs etc.), and then all of a sudden it shuts down.

The shutdown is caused by the startup process choking on redo.  FWIW

> Thanks for the feedback, and let me know if I could provide any additional

Okay, the bit about the cascading standby is a useful piece of
information.  Do you have some data about the relation reported in the
error message this is choking on based on its OID?  Is this actively
used in read-only workloads, with the relation looked at in the
cascading standby?  Is hot_standby_feedback enabled in the cascading
standby?  With which process has this cascading standby been created?
Does the workload of the primary involve a high consumption of OIDs
for relations, say many temporary tables?

Another thing that may help is the WAL record history.  Are you for
example seeing attempts to drop twice the same pgstats entry in WAL
records?  Perhaps the origin of the problem is in this area.  A
refcount of 2 is relevant, of course.

I have looked a bit around but nothing has popped up here, so as far
as I know you seem to be the only one impacted by that.

1d6a03ea4146 and dc5f9054186a are in 17.3, so perhaps something is
still off with the drop when applied to cascading standbys.  A vital
piece of information may also be with "generation", which would show
up in the error report thanks to bdda6ba30cbe, and that's included in
17.6.  A first thing would be to update to 17.6 and see how things
go for these cascading setups.  If it takes a couple of weeks to have
one report, we have a hunt that may take a few months at least, except
if somebody is able to find out the race condition here, me or someone
else.
--
Michael

---

From: Kouber Saparev <[email-masked]>
Date: Tue, 16 Sep 2025 14:45:03 +0300

На пт, 12.09.2025 г. в 3:37 Michael Paquier <[email-masked]> написа:

> cascading standby?

This objoid=767325170 is non-existent, nor was it present in the previous
shutdown (objoid=4169049057). So I guess it is something quasi-temporary
that has been dropped afterwards.

>   Is hot_standby_feedback enabled in the cascading
> standby?

Yes, hot_standby_feedback = on.

> With which process has this cascading standby been created?

Yes, we have around 150 entries added and deleted per second in pg_class,
and around 800 in pg_attribute. So something is actively creating and
dropping tables all the time.

> refcount of 2 is relevant, of course.

How could we dig into this, i.e. inspecting such attempts in the WAL
records?

> if somebody is able to find out the race condition here, me or someone
Is it enough to upgrade the replicas or we need to upgrade the primary as
well?

--
Kouber

---

From: Michael Paquier <[email-masked]>
Date: Wed, 17 Sep 2025 12:32:40 +0900

> dropping tables all the time.

The number of relations would most likely matter, I doubt that
pg_attribute contributes a lot in the frictions.  One of the test
scenarios I am running is exactly that: a bunch of CREATE/DROP tables
running concurrently with backends in a cascading standby accessing
them, while tweaking the OID generated in the control file to force
wraparounds.  I am failing to re-create your problem, for now at
least. 

> How could we dig into this, i.e. inspecting such attempts in the WAL

Yeah, with pg_walinspect or pg_waldump, but I doubt that you retain
enough WAL history to be able to ping at something specific.  One
pattern I am looking for is duplicated drops initiated from the WAL
records, when wraparound hits, or at least if there's a window small 
enough that two WAL records are generated and attempt to generate the
same entry drop.  That's just an assumption at this stage.

> Is it enough to upgrade the replicas or we need to upgrade the primary as

That would be both.  There are always reasons to update to the latest
minor version of Postgres.  See the release notes for a bunch of them,
that's usually convincing enough especially with CVEs addressed.  :)
--
Michael

---

From: Kouber Saparev <[email-masked]>
Date: Tue, 7 Jul 2026 13:13:59 +0300

We had this issue again, the database just stopped - PostgreSQL 17.6.

FATAL,XX000,"trying to drop stats entry already dropped: kind=relation
dboid=16420 objoid=3885511363 refcount=1 generation=0"

There is no such object present in the database.

Do you think the issue might be fixed in PostgreSQL 18.4?... Or PostgreSQL
19 eventually?

---

From: Michael Paquier <[email-masked]>
Date: Wed, 8 Jul 2026 08:59:35 +0900

> Do you think the issue might be fixed in PostgreSQL 18.4?... Or PostgreSQL

Well, honestly, hard to say.  That's a case where a reproducible
scenario would be useful, as it seems to ve very workload dependent in
terms of handling of the relation entries..  (Aka I have not dug in
details how to do that.)

As one example, the buildfarm seems to be quite silent, but it would
help if we need something like a wraparound for better
reproducibility.
--
Michael

---

From: Bertrand Drouvot <[email-masked]>
Date: Wed, 8 Jul 2026 10:53:47 +0000

Hi,

> help if we need something like a wraparound for better

So given that Kouber said that "we have around 150 entries added and deleted
per second in pg_class, and around 800 in pg_attribute. So something is actively
creating and dropping tables all the time.", OID reused is likely the culprit (
doable in months at this rate).

To try to simulate an OID reuse, I created an extension to manually set the next
OID and was able to reproduce the issue on 17.6, that way:

Primary: create table popo (a int);
Standby: select count(*) from popo; (keep this session open)
Primary: SELECT oid FROM pg_class WHERE relname = 'popo'; (17044 here)
Primary: drop table popo;
Primary: checkpoint;
Primary: vacuum;
Primary: SELECT set_next_oid(17044::oid);
Primary: create table popo2 (a int);
Primary: SELECT oid FROM pg_class WHERE relname = 'popo2'; (verify it reused 17044)
Primary: drop table popo2;

Produces:

2026-07-08 10:30:22.566 UTC [2096038] FATAL:  trying to drop stats entry already dropped: kind=relation dboid=5 objoid=17044 refcount=1 generation=0

So, the standby backend holds a local reference to the pgstat entry. When the
first DROP is replayed on the standby, the entry is marked dropped=true and refcount
goes from 2 to 1 so it can't be freed.
Then when the second DROP (for popo2, same OID) is replayed, pgstat_drop_entry_internal()
finds the entry with dropped=true and errors out.

The good news is that it does not reproduce on the 17 STABLE branch, I guess it's
due to 850b9218c8e being backpatched to stable branches. So a fix will be in the
next minor versions.

Michael, I wonder if that would make sense to add this "set_next_oid" kind of
thing to a contrib module to test this kind of OID reuse issue. Same idea as
xid_wraparound.

Regards,

---

From: Michael Paquier <[email-masked]>
Date: Thu, 9 Jul 2026 07:45:54 +0900

> doable in months at this rate).

That would mean keeping around a session on the standby for long time.
With a connection pooler, why not.

> due to 850b9218c8e being backpatched to stable branches. So a fix will be in the

Due to the decision of using missing_ok=true in the end-of-transaction
drops, that could indeed help.  I'd be curious to hear back after the
next minor release.  If I have nothing more to do, I'll take it.

> thing to a contrib module to test this kind of OID reuse issue. Same idea as

That's probably larger than just the OIDs.  Maybe something that's
able to do what pg_resetwal does for the control file could be useful.
Publishing another footgun sounds a bit annoying, but we also have
stuff like pg_surgery, that serve a similar purpose to shortcut the
backend.
--
Michael

---

From: Bertrand Drouvot <[email-masked]>
Date: Thu, 9 Jul 2026 04:34:20 +0000

Hi,

On Thu, Jul 09, 2026 at 07:45:54AM +0900, Michael Paquier wrote:
> next minor release.

Yeah, Kouber do you think you could upgrade to 17.11 or 18.5? (current schedule
is August 13th, 2026 ([1])).

> stuff like pg_surgery, that serve a similar purpose to shortcut the

Yeah. In addition to the OIDs I could think about the mxid (but I think they would
be better suited in xid_wraparound). The others look way more difficult (if doable)
to do outside pg_resetwal.

So I'd be tempted to just add an oid_wraparound one and we could add this test
case too. Thoughts?

[1]: https://www.postgresql.org/developer/roadmap/

Regards,