Date: Sun, 3 Mar 2024 15:47:15 -0800
From: Noah Misch <noah@leadboat.com>
To: Ronan Dunklau <ronan.dunklau@aiven.io>
Cc: pgsql-bugs <pgsql-bugs@lists.postgresql.org>
Subject: Re: FSM Corruption (was: Could not read block at end of the relation)
Message-ID: <20240303234715.4d@rfd.leadboat.com>
References: 
 <VtPLBUerbMM8BSDLQwIMV9Co_38iHS1TXE4V09x2ZVFhPkZMNE8LwgPSIzB8PHkVFXykGx04QNAzZVjJFAnU1w==@protonmail.internalid>
 <1878547.tdWV9SEqCh@aivenlaptop>
 <1958255.PYKUYFuaPT@aivenlaptop>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1958255.PYKUYFuaPT@aivenlaptop>
User-Agent: Mutt/2.2.12 (2023-09-09)
Archived-At: 
 <https://www.postgresql.org/message-id/20240303234715.4d%40rfd.leadboat.com>
Precedence: bulk

On Tue, Feb 27, 2024 at 11:34:14AM +0100, Ronan Dunklau wrote:
> - happens during heavy system load
> - lots of concurrent writes happening on a table
> - often (but haven't been able to confirm it is necessary), a vacuum is running 
> on the table at the same time the error is triggered
> 
> Then, several backends get the same error at once "ERROR:  could not read 
> block XXXX in file "base/XXXX/XXXX": read only 0 of 8192 bytes", with different 

What are some of the specific block numbers reported?

> has anybody witnessed something similar ?

https://postgr.es/m/flat/CA%2BhUKGK%2B5DOmLaBp3Z7C4S-Yv6yoROvr1UncjH2S1ZbPT8D%2BZg%40mail.gmail.com
reminded me of this.  Did you upgrade your OS recently?

On Fri, Mar 01, 2024 at 09:56:51AM +0100, Ronan Dunklau wrote:
> I think I may have missed something on my first look. On other affected 
> clusters, the FSM is definitely corrupted.  So it looks like we have an FSM 
> corruption bug on our hands.

What corruption signs did you observe in the FSM?  Since FSM is intentionally
not WAL-logged, corruption is normal, but corruption causing errors is not
normal.  That said, if any crash leaves a state that the freespace/README
"self-correcting measures" don't detect, errors may happen.  Did the clusters
crash recently?

> The occurence of this bug happening makes it hard to reproduce,  but it's 
> definitely frequent enough we witnessed it on a dozen PostgreSQL clusters. 

You could do "ALTER TABLE x SET (vacuum_truncate = off);" and see if the
problem stops happening.  That would corroborate the VACUUM theory.

Can you use backtrace_functions to get a stack track?

> In our case, we need to repair the FSM. The instructions on the wiki do work, 
> but maybe we should add something like the attached patch (modeled after the 
> same feature in pg_visibility) to make it possible to repair the FSM 
> corruption online. What do you think about it ?

That's reasonable in concept.