Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rcVC5-000Rw9-Pd for pgsql-bugs@arkaria.postgresql.org; Tue, 20 Feb 2024 18:50:18 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1rcVB3-0019Bt-MI for pgsql-bugs@arkaria.postgresql.org; Tue, 20 Feb 2024 18:49:14 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rcVB3-0019Bk-EA for pgsql-bugs@lists.postgresql.org; Tue, 20 Feb 2024 18:49:13 +0000 Received: from sss.pgh.pa.us ([68.162.161.243]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1rcVB1-000348-Am for pgsql-bugs@lists.postgresql.org; Tue, 20 Feb 2024 18:49:13 +0000 Received: from sss1.sss.pgh.pa.us (localhost [127.0.0.1]) by sss.pgh.pa.us (8.15.2/8.15.2) with ESMTP id 41KIn71T215499; Tue, 20 Feb 2024 13:49:07 -0500 From: Tom Lane To: Floris Van Nee cc: "pgsql-bugs@lists.postgresql.org" , Andres Freund Subject: Re: error "can only drop stats once" brings down database In-reply-to: References: Comments: In-reply-to Floris Van Nee message dated "Tue, 20 Feb 2024 17:38:57 +0000" MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <215497.1708454947.1@sss.pgh.pa.us> Content-Transfer-Encoding: quoted-printable Date: Tue, 20 Feb 2024 13:49:07 -0500 Message-ID: <215498.1708454947@sss.pgh.pa.us> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk Floris Van Nee writes: > Hi, > On a database we have we've recently seen a fatal error occur twice. The= error happened on two different physical replicas (of the same cluster) d= uring a WAL redo action in the recovery process. They're running Postgres = 15.5. > Occurrence 1: > 2024-02-01 06:55:54.476 CET,,,70290,,65a29b60.11292,6,,2024-01-13 15:17:= 04 CET,1/0,0,FATAL,XX000,"can only drop stats once",,,,,"WAL redo at A7BD1= /D6F9B6C0 for Transaction/COMMIT: 2024-02-01 06:55:54.395851+01; ... Hmm. This must be coming from pgstat_drop_entry_internal. I suspect the correct fix is in pgstat_drop_entry, along the lines of - if (shent) + if (shent && !shent->dropped) but it's not clear to me how the already-dropped case ought to affect the function's bool result. Also, how are we getting into a concurrent-drop situation in recovery? regards, tom lane