MIME-Version: 1.0
References: <ZzXL7WkzGyhxJljW@ip-10-97-1-34.eu-west-3.compute.internal>
 <ZzbR5Eu66lP9rt50@paquier.xyz>
 <CAN4RuQvYth942J2+FcLmJKgdpq6fE5eqyFvb_PuskxF2eL=Wzg@mail.gmail.com>
 <aJPiwtnsuh9yfFnz@paquier.xyz>
 <CAN4RuQt_Sr9h2NG0-fr7XSmU=DeHkzTeysCuOAWhWrkuhw8Puw@mail.gmail.com>
 <CAN4RuQs6ht4T7ph8NbxHZ+NmDmTrvczAxpyZmcqk+74tzsciuw@mail.gmail.com>
 <aMIJozPEP5OLI3Yj@paquier.xyz>
 <CAN4RuQv3OTJ1Owqn2U+7ZbvL-N-nZYAUjF_NG8o12ayFN-=0GQ@mail.gmail.com>
 <aMNrSgJt6_oRqh45@paquier.xyz>
 <CAN4RuQvQ3ATcYvfTR1LzJnUJXpo_F8mgz-+WxoZsyusLLmCwYA@mail.gmail.com>
 <aMor2AHvdbZWTwCD@paquier.xyz>
In-Reply-To: <aMor2AHvdbZWTwCD@paquier.xyz>
From: Kouber Saparev <kouber@gmail.com>
Date: Tue, 7 Jul 2026 13:13:59 +0300
Message-ID: 
 <CAN4RuQv2tk6ELko_M8YC6eM1YgV1Z2r-Lht22b0ObCTPTTG5Aw@mail.gmail.com>
Subject: Re: BF mamba failure
To: Michael Paquier <michael@paquier.xyz>
Cc: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>,
 pgsql-hackers@lists.postgresql.org
Content-Type: multipart/alternative; boundary="0000000000003b3ca0065602a7dd"
Archived-At: 
 <https://www.postgresql.org/message-id/CAN4RuQv2tk6ELko_M8YC6eM1YgV1Z2r-Lht22b0ObCTPTTG5Aw%40mail.gmail.com>
Precedence: bulk

--0000000000003b3ca0065602a7dd
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

We had this issue again, the database just stopped - PostgreSQL 17.6.

FATAL,XX000,"trying to drop stats entry already dropped: kind=3Drelation
dboid=3D16420 objoid=3D3885511363 refcount=3D1 generation=3D0"

There is no such object present in the database.

Do you think the issue might be fixed in PostgreSQL 18.4?... Or PostgreSQL
19 eventually?

=D0=9D=D0=B0 =D1=81=D1=80, 17.09.2025=E2=80=AF=D0=B3. =D0=B2 6:33 Michael P=
aquier <michael@paquier.xyz> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0:

> On Tue, Sep 16, 2025 at 02:45:03PM +0300, Kouber Saparev wrote:
> > =D0=9D=D0=B0 =D0=BF=D1=82, 12.09.2025=E2=80=AF=D0=B3. =D0=B2 3:37 Micha=
el Paquier <michael@paquier.xyz>
> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0:
> >> With which process has this cascading standby been created?
> >> Does the workload of the primary involve a high consumption of OIDs
> >> for relations, say many temporary tables?
> >
> > Yes, we have around 150 entries added and deleted per second in pg_clas=
s,
> > and around 800 in pg_attribute. So something is actively creating and
> > dropping tables all the time.
>
> The number of relations would most likely matter, I doubt that
> pg_attribute contributes a lot in the frictions.  One of the test
> scenarios I am running is exactly that: a bunch of CREATE/DROP tables
> running concurrently with backends in a cascading standby accessing
> them, while tweaking the OID generated in the control file to force
> wraparounds.  I am failing to re-create your problem, for now at
> least.
>
> > > Another thing that may help is the WAL record history.  Are you for
> > > example seeing attempts to drop twice the same pgstats entry in WAL
> > > records?  Perhaps the origin of the problem is in this area.  A
> > > refcount of 2 is relevant, of course.
> > >
> >
> > How could we dig into this, i.e. inspecting such attempts in the WAL
> > records?
>
> Yeah, with pg_walinspect or pg_waldump, but I doubt that you retain
> enough WAL history to be able to ping at something specific.  One
> pattern I am looking for is duplicated drops initiated from the WAL
> records, when wraparound hits, or at least if there's a window small
> enough that two WAL records are generated and attempt to generate the
> same entry drop.  That's just an assumption at this stage.
>
>
> > Is it enough to upgrade the replicas or we need to upgrade the primary =
as
> > well?
>
> That would be both.  There are always reasons to update to the latest
> minor version of Postgres.  See the release notes for a bunch of them,
> that's usually convincing enough especially with CVEs addressed.  :)
> --
> Michael
>

--0000000000003b3ca0065602a7dd
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">We had this issue again, the database just stopped - Postg=
reSQL 17.6.<div><br></div><div>FATAL,XX000,&quot;trying to drop stats entry=
 already dropped: kind=3Drelation dboid=3D16420 objoid=3D3885511363 refcoun=
t=3D1 generation=3D0&quot;</div><div><br></div><div>There is no such object=
 present in the database.<br><br>Do you think the issue might be fixed in P=
ostgreSQL 18.4?... Or PostgreSQL 19 eventually?</div></div><br><div class=
=3D"gmail_quote gmail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr=
">=D0=9D=D0=B0 =D1=81=D1=80, 17.09.2025=E2=80=AF=D0=B3. =D0=B2 6:33 Michael=
 Paquier &lt;<a href=3D"mailto:michael@paquier.xyz">michael@paquier.xyz</a>=
&gt; =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0:<br></div><blockquote class=3D"gm=
ail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,=
204,204);padding-left:1ex">On Tue, Sep 16, 2025 at 02:45:03PM +0300, Kouber=
 Saparev wrote:<br>
&gt; =D0=9D=D0=B0 =D0=BF=D1=82, 12.09.2025=E2=80=AF=D0=B3. =D0=B2 3:37 Mich=
ael Paquier &lt;<a href=3D"mailto:michael@paquier.xyz" target=3D"_blank">mi=
chael@paquier.xyz</a>&gt; =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0:<br>
&gt;&gt; With which process has this cascading standby been created?<br>
&gt;&gt; Does the workload of the primary involve a high consumption of OID=
s<br>
&gt;&gt; for relations, say many temporary tables?<br>
&gt; <br>
&gt; Yes, we have around 150 entries added and deleted per second in pg_cla=
ss,<br>
&gt; and around 800 in pg_attribute. So something is actively creating and<=
br>
&gt; dropping tables all the time.<br>
<br>
The number of relations would most likely matter, I doubt that<br>
pg_attribute contributes a lot in the frictions.=C2=A0 One of the test<br>
scenarios I am running is exactly that: a bunch of CREATE/DROP tables<br>
running concurrently with backends in a cascading standby accessing<br>
them, while tweaking the OID generated in the control file to force<br>
wraparounds.=C2=A0 I am failing to re-create your problem, for now at<br>
least. <br>
<br>
&gt; &gt; Another thing that may help is the WAL record history.=C2=A0 Are =
you for<br>
&gt; &gt; example seeing attempts to drop twice the same pgstats entry in W=
AL<br>
&gt; &gt; records?=C2=A0 Perhaps the origin of the problem is in this area.=
=C2=A0 A<br>
&gt; &gt; refcount of 2 is relevant, of course.<br>
&gt; &gt;<br>
&gt; <br>
&gt; How could we dig into this, i.e. inspecting such attempts in the WAL<b=
r>
&gt; records?<br>
<br>
Yeah, with pg_walinspect or pg_waldump, but I doubt that you retain<br>
enough WAL history to be able to ping at something specific.=C2=A0 One<br>
pattern I am looking for is duplicated drops initiated from the WAL<br>
records, when wraparound hits, or at least if there&#39;s a window small <b=
r>
enough that two WAL records are generated and attempt to generate the<br>
same entry drop.=C2=A0 That&#39;s just an assumption at this stage.<br>
<br>
<br>
&gt; Is it enough to upgrade the replicas or we need to upgrade the primary=
 as<br>
&gt; well?<br>
<br>
That would be both.=C2=A0 There are always reasons to update to the latest<=
br>
minor version of Postgres.=C2=A0 See the release notes for a bunch of them,=
<br>
that&#39;s usually convincing enough especially with CVEs addressed.=C2=A0 =
:)<br>
--<br>
Michael<br>
</blockquote></div>

--0000000000003b3ca0065602a7dd--