From: Noah Misch <noah@leadboat.com>
To: Ronan Dunklau <ronan.dunklau@aiven.io>
Cc: pgsql-bugs <pgsql-bugs@lists.postgresql.org>
Subject: Re: FSM Corruption (was: Could not read block at end of the relation)
Date: Sun, 3 Mar 2024 15:47:15 -0800
Message-ID: <20240303234715.4d@rfd.leadboat.com> (raw)
In-Reply-To: <1958255.PYKUYFuaPT@aivenlaptop>
References: <VtPLBUerbMM8BSDLQwIMV9Co_38iHS1TXE4V09x2ZVFhPkZMNE8LwgPSIzB8PHkVFXykGx04QNAzZVjJFAnU1w==@protonmail.internalid>
<1878547.tdWV9SEqCh@aivenlaptop>
<1958255.PYKUYFuaPT@aivenlaptop>
On Tue, Feb 27, 2024 at 11:34:14AM +0100, Ronan Dunklau wrote:
> - happens during heavy system load
> - lots of concurrent writes happening on a table
> - often (but haven't been able to confirm it is necessary), a vacuum is running
> on the table at the same time the error is triggered
>
> Then, several backends get the same error at once "ERROR: could not read
> block XXXX in file "base/XXXX/XXXX": read only 0 of 8192 bytes", with different
What are some of the specific block numbers reported?
> has anybody witnessed something similar ?
https://postgr.es/m/flat/CA%2BhUKGK%2B5DOmLaBp3Z7C4S-Yv6yoROvr1UncjH2S1ZbPT8D%2BZg%40mail.gmail.com
reminded me of this. Did you upgrade your OS recently?
On Fri, Mar 01, 2024 at 09:56:51AM +0100, Ronan Dunklau wrote:
> I think I may have missed something on my first look. On other affected
> clusters, the FSM is definitely corrupted. So it looks like we have an FSM
> corruption bug on our hands.
What corruption signs did you observe in the FSM? Since FSM is intentionally
not WAL-logged, corruption is normal, but corruption causing errors is not
normal. That said, if any crash leaves a state that the freespace/README
"self-correcting measures" don't detect, errors may happen. Did the clusters
crash recently?
> The occurence of this bug happening makes it hard to reproduce, but it's
> definitely frequent enough we witnessed it on a dozen PostgreSQL clusters.
You could do "ALTER TABLE x SET (vacuum_truncate = off);" and see if the
problem stops happening. That would corroborate the VACUUM theory.
Can you use backtrace_functions to get a stack track?
> In our case, we need to repair the FSM. The instructions on the wiki do work,
> but maybe we should add something like the attached patch (modeled after the
> same feature in pg_visibility) to make it possible to repair the FSM
> corruption online. What do you think about it ?
That's reasonable in concept.
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: noah@leadboat.com, ronan.dunklau@aiven.io, pgsql-bugs@lists.postgresql.org
Subject: Re: FSM Corruption (was: Could not read block at end of the relation)
In-Reply-To: <20240303234715.4d@rfd.leadboat.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox