public inbox for [email protected]  

help / color / mirror / Atom feed

From: Bertrand Drouvot <[email protected]>
To: Michael Paquier <[email protected]>
Cc: Kouber Saparev <[email protected]>
Cc: [email protected]
Subject: Re: BF mamba failure
Date: Wed, 8 Jul 2026 10:53:47 +0000
Message-ID: <ak4sO/IINIBT7y0J@bdtpg> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>
	<CAN4RuQt_Sr9h2NG0-fr7XSmU=DeHkzTeysCuOAWhWrkuhw8Puw@mail.gmail.com>
	<CAN4RuQs6ht4T7ph8NbxHZ+NmDmTrvczAxpyZmcqk+74tzsciuw@mail.gmail.com>
	<[email protected]>
	<CAN4RuQv3OTJ1Owqn2U+7ZbvL-N-nZYAUjF_NG8o12ayFN-=0GQ@mail.gmail.com>
	<[email protected]>
	<CAN4RuQvQ3ATcYvfTR1LzJnUJXpo_F8mgz-+WxoZsyusLLmCwYA@mail.gmail.com>
	<[email protected]>
	<CAN4RuQv2tk6ELko_M8YC6eM1YgV1Z2r-Lht22b0ObCTPTTG5Aw@mail.gmail.com>
	<[email protected]>

Hi,

On Wed, Jul 08, 2026 at 08:59:35AM +0900, Michael Paquier wrote:
> On Tue, Jul 07, 2026 at 01:13:59PM +0300, Kouber Saparev wrote:
> > We had this issue again, the database just stopped - PostgreSQL 17.6.
> > 
> > FATAL,XX000,"trying to drop stats entry already dropped: kind=relation
> > dboid=16420 objoid=3885511363 refcount=1 generation=0"
> > 
> > There is no such object present in the database.
> > 
> > Do you think the issue might be fixed in PostgreSQL 18.4?... Or PostgreSQL
> > 19 eventually?
> 
> Well, honestly, hard to say.  That's a case where a reproducible
> scenario would be useful, as it seems to ve very workload dependent in
> terms of handling of the relation entries..  (Aka I have not dug in
> details how to do that.)
> 
> As one example, the buildfarm seems to be quite silent, but it would
> help if we need something like a wraparound for better
> reproducibility.

So given that Kouber said that "we have around 150 entries added and deleted
per second in pg_class, and around 800 in pg_attribute. So something is actively
creating and dropping tables all the time.", OID reused is likely the culprit (
doable in months at this rate).

To try to simulate an OID reuse, I created an extension to manually set the next
OID and was able to reproduce the issue on 17.6, that way:

Primary: create table popo (a int);
Standby: select count(*) from popo; (keep this session open)
Primary: SELECT oid FROM pg_class WHERE relname = 'popo'; (17044 here)
Primary: drop table popo;
Primary: checkpoint;
Primary: vacuum;
Primary: SELECT set_next_oid(17044::oid);
Primary: create table popo2 (a int);
Primary: SELECT oid FROM pg_class WHERE relname = 'popo2'; (verify it reused 17044)
Primary: drop table popo2;

Produces:

2026-07-08 10:30:22.566 UTC [2096038] FATAL:  trying to drop stats entry already dropped: kind=relation dboid=5 objoid=17044 refcount=1 generation=0

So, the standby backend holds a local reference to the pgstat entry. When the
first DROP is replayed on the standby, the entry is marked dropped=true and refcount
goes from 2 to 1 so it can't be freed.
Then when the second DROP (for popo2, same OID) is replayed, pgstat_drop_entry_internal()
finds the entry with dropped=true and errors out.

The good news is that it does not reproduce on the 17 STABLE branch, I guess it's
due to 850b9218c8e being backpatched to stable branches. So a fix will be in the
next minor versions.

Michael, I wonder if that would make sense to add this "set_next_oid" kind of
thing to a contrib module to test this kind of OID reuse issue. Same idea as
xid_wraparound.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

view thread (20+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected], [email protected]
  Subject: Re: BF mamba failure
  In-Reply-To: <ak4sO/IINIBT7y0J@bdtpg>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox