Date: Wed, 8 Jul 2026 10:53:47 +0000
From: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
To: Michael Paquier <michael@paquier.xyz>
Cc: Kouber Saparev <kouber@gmail.com>, pgsql-hackers@lists.postgresql.org
Subject: Re: BF mamba failure
Message-ID: <ak4sO/IINIBT7y0J@bdtpg>
References: <aJPiwtnsuh9yfFnz@paquier.xyz>
 <CAN4RuQt_Sr9h2NG0-fr7XSmU=DeHkzTeysCuOAWhWrkuhw8Puw@mail.gmail.com>
 <CAN4RuQs6ht4T7ph8NbxHZ+NmDmTrvczAxpyZmcqk+74tzsciuw@mail.gmail.com>
 <aMIJozPEP5OLI3Yj@paquier.xyz>
 <CAN4RuQv3OTJ1Owqn2U+7ZbvL-N-nZYAUjF_NG8o12ayFN-=0GQ@mail.gmail.com>
 <aMNrSgJt6_oRqh45@paquier.xyz>
 <CAN4RuQvQ3ATcYvfTR1LzJnUJXpo_F8mgz-+WxoZsyusLLmCwYA@mail.gmail.com>
 <aMor2AHvdbZWTwCD@paquier.xyz>
 <CAN4RuQv2tk6ELko_M8YC6eM1YgV1Z2r-Lht22b0ObCTPTTG5Aw@mail.gmail.com>
 <ak2S5046X1aaagr4@paquier.xyz>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ak2S5046X1aaagr4@paquier.xyz>
Archived-At: <https://www.postgresql.org/message-id/ak4sO/IINIBT7y0J%40bdtpg>
Precedence: bulk

Hi,

On Wed, Jul 08, 2026 at 08:59:35AM +0900, Michael Paquier wrote:
> On Tue, Jul 07, 2026 at 01:13:59PM +0300, Kouber Saparev wrote:
> > We had this issue again, the database just stopped - PostgreSQL 17.6.
> > 
> > FATAL,XX000,"trying to drop stats entry already dropped: kind=relation
> > dboid=16420 objoid=3885511363 refcount=1 generation=0"
> > 
> > There is no such object present in the database.
> > 
> > Do you think the issue might be fixed in PostgreSQL 18.4?... Or PostgreSQL
> > 19 eventually?
> 
> Well, honestly, hard to say.  That's a case where a reproducible
> scenario would be useful, as it seems to ve very workload dependent in
> terms of handling of the relation entries..  (Aka I have not dug in
> details how to do that.)
> 
> As one example, the buildfarm seems to be quite silent, but it would
> help if we need something like a wraparound for better
> reproducibility.

So given that Kouber said that "we have around 150 entries added and deleted
per second in pg_class, and around 800 in pg_attribute. So something is actively
creating and dropping tables all the time.", OID reused is likely the culprit (
doable in months at this rate).

To try to simulate an OID reuse, I created an extension to manually set the next
OID and was able to reproduce the issue on 17.6, that way:

Primary: create table popo (a int);
Standby: select count(*) from popo; (keep this session open)
Primary: SELECT oid FROM pg_class WHERE relname = 'popo'; (17044 here)
Primary: drop table popo;
Primary: checkpoint;
Primary: vacuum;
Primary: SELECT set_next_oid(17044::oid);
Primary: create table popo2 (a int);
Primary: SELECT oid FROM pg_class WHERE relname = 'popo2'; (verify it reused 17044)
Primary: drop table popo2;

Produces:

2026-07-08 10:30:22.566 UTC [2096038] FATAL:  trying to drop stats entry already dropped: kind=relation dboid=5 objoid=17044 refcount=1 generation=0

So, the standby backend holds a local reference to the pgstat entry. When the
first DROP is replayed on the standby, the entry is marked dropped=true and refcount
goes from 2 to 1 so it can't be freed.
Then when the second DROP (for popo2, same OID) is replayed, pgstat_drop_entry_internal()
finds the entry with dropped=true and errors out.

The good news is that it does not reproduce on the 17 STABLE branch, I guess it's
due to 850b9218c8e being backpatched to stable branches. So a fix will be in the
next minor versions.

Michael, I wonder if that would make sense to add this "set_next_oid" kind of
thing to a contrib module to test this kind of OID reuse issue. Same idea as
xid_wraparound.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com