Message-ID: <77607a48-eb82-f241-a142-21697b6faf4c@gmail.com>
Date: Fri, 3 May 2024 23:00:00 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.4.2
Subject: Re: error "can only drop stats once" brings down database
Content-Language: en-US
To: Floris Van Nee <florisvannee@Optiver.com>, Tom Lane <tgl@sss.pgh.pa.us>
Cc: "pgsql-bugs@lists.postgresql.org" <pgsql-bugs@lists.postgresql.org>,
 Andres Freund <andres@anarazel.de>
References: <b14ae28029f64757bb64613be2549a74@Optiver.com>
 <215498.1708454947@sss.pgh.pa.us>
 <8e5cb534706a4f43bd74405c8fd91437@Optiver.com>
From: Alexander Lakhin <exclusion@gmail.com>
In-Reply-To: <8e5cb534706a4f43bd74405c8fd91437@Optiver.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Archived-At: 
 <https://www.postgresql.org/message-id/77607a48-eb82-f241-a142-21697b6faf4c%40gmail.com>
Precedence: bulk

Hello Floris,

03.05.2024 21:10, Floris Van Nee wrote:
>> Floris Van Nee <florisvannee@Optiver.com> writes:
>>> Hi,
>>> On a database we have we've recently seen a fatal error occur twice. The
>> error happened on two different physical replicas (of the same cluster)
>> during a WAL redo action in the recovery process. They're running Postgres
>> 15.5.
>>
>>> Occurrence 1:
>>> 2024-02-01 06:55:54.476 CET,,,70290,,65a29b60.11292,6,,2024-01-13 15:17:04
>> CET,1/0,0,FATAL,XX000,"can only drop stats once",,,,,"WAL redo at
>> A7BD1/D6F9B6C0 for Transaction/COMMIT: 2024-02-01 06:55:54.395851+01;
>> ...
>>
>> Hmm.  This must be coming from pgstat_drop_entry_internal.
>> I suspect the correct fix is in pgstat_drop_entry, along the lines of
>>
>> -	if (shent)
>> +	if (shent && !shent->dropped)
>>
>> but it's not clear to me how the already-dropped case ought to affect the
>> function's bool result.  Also, how are we getting into a concurrent-drop
>> situation in recovery?
> Anyone has further thoughts on this? This still happens occasionally.
>

Please take a look at bug #17947 (maybe you encounter the same?):
https://www.postgresql.org/message-id/17947-b9554521ad963c9c%40postgresql.org

Best regards,
Alexander