public inbox for [email protected]
help / color / mirror / Atom feedFrom: Thomas Munro <[email protected]>
To: Andres Freund <[email protected]>
Cc: [email protected]
Subject: Re: odd buildfarm failure - "pg_ctl: control file appears to be corrupt"
Date: Thu, 24 Nov 2022 10:59:26 +1300
Message-ID: <CA+hUKGKr0TBwFMZ_BxcOps4_WDxhX5tS2D1Mwb1fAHyMKxo-UA@mail.gmail.com> (raw)
In-Reply-To: <CA+hUKGL9w0VhZ=ceMDjXgCXy1BC5VzjwXc-HetSxfNzvBLge3A@mail.gmail.com>
References: <[email protected]>
<CA+hUKGL9w0VhZ=ceMDjXgCXy1BC5VzjwXc-HetSxfNzvBLge3A@mail.gmail.com>
On Wed, Nov 23, 2022 at 11:03 PM Thomas Munro <[email protected]> wrote:
> On Wed, Nov 23, 2022 at 2:42 PM Andres Freund <[email protected]> wrote:
> > The failure has to be happening in wait_for_postmaster_promote(), because the
> > standby2 is actually successfully promoted.
>
> I assume this is ext4. Presumably anything that reads the
> controlfile, like pg_ctl, pg_checksums, pg_resetwal,
> pg_control_system(), ... by reading without interlocking against
> writes could see garbage. I have lost track of the versions and the
> thread, but I worked out at some point by experimentation that this
> only started relatively recently for concurrent read() and write(),
> but always happened with concurrent pread() and pwrite(). The control
> file uses the non-p variants which didn't mash old/new data like
> grated cheese under concurrency due to some implementation detail, but
> now does.
As for what to do about it, some ideas:
1. Use advisory range locking. (This would be an advisory version of
what many other filesystems do automatically, AFAIK. Does Windows
have a thing like POSIX file locking, or need it here?)
2. Retry after a short time on checksum failure. The probability is
already miniscule, and becomes pretty close to 0 if we read thrice
100ms apart.
3. Some scheme that involves renaming the file into place. (That
might be a pain on Windows; it only works for the relmap thing because
all readers and writers are in the backend and use an LWLock to avoid
silly handle semantics.)
4. ???
First thought is that 2 is appropriate level of complexity for this
rare and stupid problem.
view thread (8+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected]
Subject: Re: odd buildfarm failure - "pg_ctl: control file appears to be corrupt"
In-Reply-To: <CA+hUKGKr0TBwFMZ_BxcOps4_WDxhX5tS2D1Mwb1fAHyMKxo-UA@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox