Feedback-ID: i0fe9450f:Fastmail
Date: Fri, 12 Sep 2025 09:37:30 +0900
From: Michael Paquier <michael@paquier.xyz>
To: Kouber Saparev <kouber@gmail.com>
Cc: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>,
	pgsql-hackers@lists.postgresql.org
Subject: Re: BF mamba failure
Message-ID: <aMNrSgJt6_oRqh45@paquier.xyz>
References: <ZxCRku2jk8Jz0sTU@paquier.xyz>
 <ZzWs6_pKr8DgEAvs@paquier.xyz>
 <ZzXL7WkzGyhxJljW@ip-10-97-1-34.eu-west-3.compute.internal>
 <ZzbR5Eu66lP9rt50@paquier.xyz>
 <CAN4RuQvYth942J2+FcLmJKgdpq6fE5eqyFvb_PuskxF2eL=Wzg@mail.gmail.com>
 <aJPiwtnsuh9yfFnz@paquier.xyz>
 <CAN4RuQt_Sr9h2NG0-fr7XSmU=DeHkzTeysCuOAWhWrkuhw8Puw@mail.gmail.com>
 <CAN4RuQs6ht4T7ph8NbxHZ+NmDmTrvczAxpyZmcqk+74tzsciuw@mail.gmail.com>
 <aMIJozPEP5OLI3Yj@paquier.xyz>
 <CAN4RuQv3OTJ1Owqn2U+7ZbvL-N-nZYAUjF_NG8o12ayFN-=0GQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="bWsaEOg/wKyEDUBS"
Content-Disposition: inline
In-Reply-To: 
 <CAN4RuQv3OTJ1Owqn2U+7ZbvL-N-nZYAUjF_NG8o12ayFN-=0GQ@mail.gmail.com>
Archived-At: 
 <https://www.postgresql.org/message-id/aMNrSgJt6_oRqh45%40paquier.xyz>
Precedence: bulk


--bWsaEOg/wKyEDUBS
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

On Thu, Sep 11, 2025 at 04:35:01PM +0300, Kouber Saparev wrote:
> The pattern is the same, although I am not 100% sure that the same replica
> is having this - it is a cascaded streaming replica, i.e. a replica of
> another replica. Once we had this in October 2024, with version 15.4, then
> in August 2025 with 17.3, and now in September again (17.3). The database
> is working for month(s) perfectly fine in a heavy production workload (lots
> of WALs etc.), and then all of a sudden it shuts down.

The shutdown is caused by the startup process choking on redo.  FWIW

> Thanks for the feedback, and let me know if I could provide any additional
> info.

Okay, the bit about the cascading standby is a useful piece of
information.  Do you have some data about the relation reported in the
error message this is choking on based on its OID?  Is this actively
used in read-only workloads, with the relation looked at in the
cascading standby?  Is hot_standby_feedback enabled in the cascading
standby?  With which process has this cascading standby been created?
Does the workload of the primary involve a high consumption of OIDs
for relations, say many temporary tables?

Another thing that may help is the WAL record history.  Are you for
example seeing attempts to drop twice the same pgstats entry in WAL
records?  Perhaps the origin of the problem is in this area.  A
refcount of 2 is relevant, of course.

I have looked a bit around but nothing has popped up here, so as far
as I know you seem to be the only one impacted by that.

1d6a03ea4146 and dc5f9054186a are in 17.3, so perhaps something is
still off with the drop when applied to cascading standbys.  A vital
piece of information may also be with "generation", which would show
up in the error report thanks to bdda6ba30cbe, and that's included in
17.6.  A first thing would be to update to 17.6 and see how things
go for these cascading setups.  If it takes a couple of weeks to have
one report, we have a hunt that may take a few months at least, except
if somebody is able to find out the race condition here, me or someone
else.
--
Michael

--bWsaEOg/wKyEDUBS
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAmjDa0oACgkQnvQgOdby
QH0LdBAAi/0G5C1P3OVA182TQoQPNQ8yx1Y7m5iiyNpDRRGnxTIP+WlzkG30yNkB
Wdo4NOWN2yrn8gUz7Kg2Lo/yjg8TizqgkoggOlHuyqZ2iIGpgCM3Vs/P9yf/uj5Y
YY55PB57e/8q0BHfBu1C4J+1C4NN1+z+cXhvoGn1Ry71/qec4Ev3o+HwfMS4k76C
ygQK1kG+6AOvR+dAz61JkbjQydN3FHh7j1GyfDkTlR1MPcEiilFS03xHo4EMCjnU
EymK4EuLXloZROCmRnoUQAFETkBqyw+T/ItSNvhmdVjK1lCrARMPUcv57bgpYgNS
67dDvylgTbP0+1gxVCujHaQKEu1wc0+UT4+ZJJcQsDmN0zNpP0Pl7QZpSgwKTe3a
+mgtw4b1NabOb0c6iP7mPvFWe4PHKkbSFtqPi+4+yJirvYDwSnGakHKedjplLpyp
ci4pGMLinrbD5cqrCOW7woME8B2ScmSVk4fMD0KhEg42Mk0dy9z9q2AYjM7wKN3K
rMU+j1HxAShXikH7aHUjPQGi5LX8MFlASme/RD6mWOoF4Qmam1fVkryny8W9o4Is
dLpAqdkhMl5Afw3ukn03q3bD5ji/j4jMwocprY4qRLigyhynOishz8qUwL0OPCA1
AVJsnReGcnGmevFeVBvYUNteQx2VHKvXHqCIIttDcSG1NwmTLzI=
=FHqJ
-----END PGP SIGNATURE-----

--bWsaEOg/wKyEDUBS--