MIME-Version: 1.0
References: <CADofcAV8xu3hCNHq7-7x56KrP9rD6=A04=qjTr3nETh-gptF8w@mail.gmail.com>
 <8aa1d1d7-645f-404b-a8f8-7c49be9acd27@vondra.me>
In-Reply-To: <8aa1d1d7-645f-404b-a8f8-7c49be9acd27@vondra.me>
From: Michael Harris <harmic@gmail.com>
Date: Tue, 10 Dec 2024 10:00:43 +1100
Message-ID: <CADofcAUfhMer0SjthZbG8AoXaYDH6eGDpw+jzkO8UH-ZqkbmNQ@mail.gmail.com>
Subject: Re: FileFallocate misbehaving on XFS
To: Tomas Vondra <tomas@vondra.me>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Archived-At: <https://www.postgresql.org/message-id/CADofcAUfhMer0SjthZbG8AoXaYDH6eGDpw%2BjzkO8UH-ZqkbmNQ%40mail.gmail.com>
Precedence: bulk

Hi Tomas

On Mon, 9 Dec 2024 at 21:06, Tomas Vondra <tomas@vondra.me> wrote:
> Sounds more like an XFS bug/behavior, so it's not clear to me what we
> could do about it. I mean, if the filesystem reports bogus out-of-space,
> is there even something we can do?

I don't disagree that it's most likely an XFS issue. However, XFS is
pretty widely used - it's the default FS for RHEL & the default in
SUSE for non-root partitions - so maybe some action should be taken.

Some things we could consider:

 - Providing a way to configure PG not to use posix_fallocate at runtime

 - Detecting the use of XFS (probably nasty and complex to do in a
platform independent way) and disable posix_fallocate

 - In the case of posix_fallocate failing with ENOSPC, fall back to
FileZero (worst case that will fail as well, in which case we will
know that we really are out of space)

 - Documenting that XFS might not be a good choice, at least for some
kernel versions

> What is not clear to me is why would this affect pg_upgrade at all. We
> have the data files split into 1GB segments, and the copy/clone/... goes
> one by one. So there shouldn't be more than 1GB "extra" space needed.
> Surely you have more free space on the system?

Yes, that also confused me. It actually fails during the schema
restore phase - where pg_upgrade calls pg_restore to restore a
schema-only dump that it takes earlier in the process. At this stage
it is only trying to restore the schema, not any actual table data.
Note that we use the --link  option to pg_upgrade, so it should not be
using much space even when the table data is being upgraded.

The filesystems have >1TB free space when this has occurred.

It does continue to give this error after the upgrade, at apparently
random intervals, when data is being loaded into the DB using COPY
commands, so it might be best not to focus too much on the fact that
we first encounter it during the upgrade.

Cheers
Mike.