From: Tom Lane <tgl@sss.pgh.pa.us>
To: Thomas Munro <thomas.munro@gmail.com>
cc: Andres Freund <andres@anarazel.de>, pgsql-hackers@postgresql.org
Subject: Re: odd buildfarm failure - "pg_ctl: control file appears to be
 corrupt"
In-reply-to: 
 <CA+hUKGKr0TBwFMZ_BxcOps4_WDxhX5tS2D1Mwb1fAHyMKxo-UA@mail.gmail.com>
References: <20221123014224.xisi44byq3cf5psi@awork3.anarazel.de>
 <CA+hUKGL9w0VhZ=ceMDjXgCXy1BC5VzjwXc-HetSxfNzvBLge3A@mail.gmail.com>
 <CA+hUKGKr0TBwFMZ_BxcOps4_WDxhX5tS2D1Mwb1fAHyMKxo-UA@mail.gmail.com>
Comments: In-reply-to Thomas Munro <thomas.munro@gmail.com>
	message dated "Thu, 24 Nov 2022 10:59:26 +1300"
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <3748782.1669241103.1@sss.pgh.pa.us>
Content-Transfer-Encoding: quoted-printable
Date: Wed, 23 Nov 2022 17:05:03 -0500
Message-ID: <3748783.1669241103@sss.pgh.pa.us>
Archived-At: 
 <https://www.postgresql.org/message-id/3748783.1669241103%40sss.pgh.pa.us>
Precedence: bulk

Thomas Munro <thomas.munro@gmail.com> writes:
> On Wed, Nov 23, 2022 at 11:03 PM Thomas Munro <thomas.munro@gmail.com> w=
rote:
>> I assume this is ext4.  Presumably anything that reads the
>> controlfile, like pg_ctl, pg_checksums, pg_resetwal,
>> pg_control_system(), ... by reading without interlocking against
>> writes could see garbage.  I have lost track of the versions and the
>> thread, but I worked out at some point by experimentation that this
>> only started relatively recently for concurrent read() and write(),
>> but always happened with concurrent pread() and pwrite().  The control
>> file uses the non-p variants which didn't mash old/new data like
>> grated cheese under concurrency due to some implementation detail, but
>> now does.

Ugh.

> As for what to do about it, some ideas:
> 2.  Retry after a short time on checksum failure.  The probability is
> already miniscule, and becomes pretty close to 0 if we read thrice
> 100ms apart.

> First thought is that 2 is appropriate level of complexity for this
> rare and stupid problem.

Yeah, I was thinking the same.  A variant could be "repeat until
we see the same calculated checksum twice".

			regards, tom lane