MIME-Version: 1.0
References: <ZxCRku2jk8Jz0sTU@paquier.xyz> <ZzWs6_pKr8DgEAvs@paquier.xyz>
 <ZzXL7WkzGyhxJljW@ip-10-97-1-34.eu-west-3.compute.internal>
 <ZzbR5Eu66lP9rt50@paquier.xyz>
 <CAN4RuQvYth942J2+FcLmJKgdpq6fE5eqyFvb_PuskxF2eL=Wzg@mail.gmail.com>
 <aJPiwtnsuh9yfFnz@paquier.xyz>
 <CAN4RuQt_Sr9h2NG0-fr7XSmU=DeHkzTeysCuOAWhWrkuhw8Puw@mail.gmail.com>
 <CAN4RuQs6ht4T7ph8NbxHZ+NmDmTrvczAxpyZmcqk+74tzsciuw@mail.gmail.com>
 <aMIJozPEP5OLI3Yj@paquier.xyz>
 <CAN4RuQv3OTJ1Owqn2U+7ZbvL-N-nZYAUjF_NG8o12ayFN-=0GQ@mail.gmail.com>
 <aMNrSgJt6_oRqh45@paquier.xyz>
In-Reply-To: <aMNrSgJt6_oRqh45@paquier.xyz>
From: Kouber Saparev <kouber@gmail.com>
Date: Tue, 16 Sep 2025 14:45:03 +0300
Message-ID: 
 <CAN4RuQvQ3ATcYvfTR1LzJnUJXpo_F8mgz-+WxoZsyusLLmCwYA@mail.gmail.com>
Subject: Re: BF mamba failure
To: Michael Paquier <michael@paquier.xyz>
Cc: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>,
 pgsql-hackers@lists.postgresql.org
Content-Type: multipart/alternative; boundary="000000000000a01129063ee9a7b1"
Archived-At: 
 <https://www.postgresql.org/message-id/CAN4RuQvQ3ATcYvfTR1LzJnUJXpo_F8mgz-%2BWxoZsyusLLmCwYA%40mail.gmail.com>
Precedence: bulk

--000000000000a01129063ee9a7b1
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

=D0=9D=D0=B0 =D0=BF=D1=82, 12.09.2025=E2=80=AF=D0=B3. =D0=B2 3:37 Michael P=
aquier <michael@paquier.xyz> =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0:

> Okay, the bit about the cascading standby is a useful piece of
> information.  Do you have some data about the relation reported in the
> error message this is choking on based on its OID?  Is this actively
> used in read-only workloads, with the relation looked at in the
> cascading standby?


This objoid=3D767325170 is non-existent, nor was it present in the previous
shutdown (objoid=3D4169049057). So I guess it is something quasi-temporary
that has been dropped afterwards.


>   Is hot_standby_feedback enabled in the cascading
> standby?


Yes, hot_standby_feedback =3D on.


> With which process has this cascading standby been created?
> Does the workload of the primary involve a high consumption of OIDs
> for relations, say many temporary tables?
>

Yes, we have around 150 entries added and deleted per second in pg_class,
and around 800 in pg_attribute. So something is actively creating and
dropping tables all the time.


>
> Another thing that may help is the WAL record history.  Are you for
> example seeing attempts to drop twice the same pgstats entry in WAL
> records?  Perhaps the origin of the problem is in this area.  A
> refcount of 2 is relevant, of course.
>

How could we dig into this, i.e. inspecting such attempts in the WAL
records?


>
> I have looked a bit around but nothing has popped up here, so as far
> as I know you seem to be the only one impacted by that.
>
> 1d6a03ea4146 and dc5f9054186a are in 17.3, so perhaps something is
> still off with the drop when applied to cascading standbys.  A vital
> piece of information may also be with "generation", which would show
> up in the error report thanks to bdda6ba30cbe, and that's included in
> 17.6.  A first thing would be to update to 17.6 and see how things
> go for these cascading setups.  If it takes a couple of weeks to have
> one report, we have a hunt that may take a few months at least, except
> if somebody is able to find out the race condition here, me or someone
> else.
>
>
Is it enough to upgrade the replicas or we need to upgrade the primary as
well?

--
Kouber

--000000000000a01129063ee9a7b1
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote g=
mail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">=D0=9D=D0=B0 =
=D0=BF=D1=82, 12.09.2025=E2=80=AF=D0=B3. =D0=B2 3:37 Michael Paquier &lt;<a=
 href=3D"mailto:michael@paquier.xyz">michael@paquier.xyz</a>&gt; =D0=BD=D0=
=B0=D0=BF=D0=B8=D1=81=D0=B0:<br></div><blockquote class=3D"gmail_quote" sty=
le=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);paddi=
ng-left:1ex">Okay, the bit about the cascading standby is a useful piece of=
<br>
information.=C2=A0 Do you have some data about the relation reported in the=
<br>
error message this is choking on based on its OID?=C2=A0 Is this actively<b=
r>
used in read-only workloads, with the relation looked at in the<br>
cascading standby?</blockquote><div><br>This objoid=3D767325170 is non-exis=
tent, nor was it present in the previous shutdown (objoid=3D4169049057). So=
 I guess it is something quasi-temporary that has been dropped afterwards.<=
/div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px=
 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">=C2=
=A0 Is hot_standby_feedback enabled in the cascading<br>
standby?</blockquote><div><br>Yes, hot_standby_feedback =3D on.<br>=C2=A0</=
div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;bor=
der-left:1px solid rgb(204,204,204);padding-left:1ex">With which process ha=
s this cascading standby been created?<br>
Does the workload of the primary involve a high consumption of OIDs<br>
for relations, say many temporary tables?<br></blockquote><div><br>Yes, we =
have around 150 entries added and deleted per second in pg_class, and aroun=
d 800 in pg_attribute. So something is actively creating and dropping table=
s all the time.<br>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"m=
argin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left=
:1ex">
<br>
Another thing that may help is the WAL record history.=C2=A0 Are you for<br=
>
example seeing attempts to drop twice the same pgstats entry in WAL<br>
records?=C2=A0 Perhaps the origin of the problem is in this area.=C2=A0 A<b=
r>
refcount of 2 is relevant, of course.<br></blockquote><div><br>How could we=
 dig into this, i.e. inspecting such attempts in the WAL records?<br>=C2=A0=
</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;b=
order-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
I have looked a bit around but nothing has popped up here, so as far<br>
as I know you seem to be the only one impacted by that.<br>
<br>
1d6a03ea4146 and dc5f9054186a are in 17.3, so perhaps something is<br>
still off with the drop when applied to cascading standbys.=C2=A0 A vital<b=
r>
piece of information may also be with &quot;generation&quot;, which would s=
how<br>
up in the error report thanks to bdda6ba30cbe, and that&#39;s included in<b=
r>
17.6.=C2=A0 A first thing would be to update to 17.6 and see how things<br>
go for these cascading setups.=C2=A0 If it takes a couple of weeks to have<=
br>
one report, we have a hunt that may take a few months at least, except<br>
if somebody is able to find out the race condition here, me or someone<br>
else.<br><br></blockquote><div><br>Is it enough to upgrade the replicas or =
we need to upgrade the primary as well?<br><br>--<br>Kouber=C2=A0</div></di=
v></div>

--000000000000a01129063ee9a7b1--