MIME-Version: 1.0
References: <20221123014224.xisi44byq3cf5psi@awork3.anarazel.de>
 <CA+hUKGL9w0VhZ=ceMDjXgCXy1BC5VzjwXc-HetSxfNzvBLge3A@mail.gmail.com>
 <CA+hUKGKr0TBwFMZ_BxcOps4_WDxhX5tS2D1Mwb1fAHyMKxo-UA@mail.gmail.com>
 <3748783.1669241103@sss.pgh.pa.us>
In-Reply-To: <3748783.1669241103@sss.pgh.pa.us>
From: Thomas Munro <thomas.munro@gmail.com>
Date: Thu, 24 Nov 2022 14:02:54 +1300
Message-ID: 
 <CA+hUKGK-BEe38aKNqHJDQ86LUW-CMwF5F9bo1JtJVg71FoDv_w@mail.gmail.com>
Subject: Re: odd buildfarm failure - "pg_ctl: control file appears to be
 corrupt"
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Andres Freund <andres@anarazel.de>, pgsql-hackers@postgresql.org
Content-Type: multipart/mixed; boundary="000000000000459f6e05ee2cf982"
Archived-At: 
 <https://www.postgresql.org/message-id/CA%2BhUKGK-BEe38aKNqHJDQ86LUW-CMwF5F9bo1JtJVg71FoDv_w%40mail.gmail.com>
Precedence: bulk

--000000000000459f6e05ee2cf982
Content-Type: text/plain; charset="UTF-8"

On Thu, Nov 24, 2022 at 11:05 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Thomas Munro <thomas.munro@gmail.com> writes:
> > On Wed, Nov 23, 2022 at 11:03 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> > As for what to do about it, some ideas:
> > 2.  Retry after a short time on checksum failure.  The probability is
> > already miniscule, and becomes pretty close to 0 if we read thrice
> > 100ms apart.
>
> > First thought is that 2 is appropriate level of complexity for this
> > rare and stupid problem.
>
> Yeah, I was thinking the same.  A variant could be "repeat until
> we see the same calculated checksum twice".

Hmm.  While writing a comment to explain why that's good enough, I
realised it's not really true for a standby that control file writes
are always expected to be far apart in time.  XLogFlush->
UpdateMinRecoveryPoint() could coincide badly with our N attempts for
any small N and for any nap time, which I think makes your idea better
than mine.

With some cartoon-level understanding of what's going on (to wit: I
think the kernel just pins the page but doesn't use a page-level
content lock or range lock, so what you're seeing is raw racing memcpy
calls and unsynchronised cache line effects), I guess you'd be fairly
likely to make "progress" in seeing more new data even if you didn't
sleep in between, but who knows.  So I have a 10ms sleep to make
progress very likely; given your algorithm it doesn't matter if you
didn't make all the progress, just some.  Since this is reachable from
SQL, I think we also need a CFI call so you can't get uninterruptibly
stuck here?

I wrote a stupid throw-away function to force a write.  If you have an
ext4 system to hand (xfs, zfs, apfs, ufs, others don't suffer from
this) you can do:

 do $$ begin for i in 1..100000000 do loop perform
pg_update_control_file(); end loop; end; $$;

... while you also do:

 select pg_control_system();
 \watch 0.001

... and you'll soon see:

ERROR:  calculated CRC checksum does not match value stored in file

The attached draft patch fixes it.

--000000000000459f6e05ee2cf982
Content-Type: text/x-patch; charset="US-ASCII";
	name="0001-XXX-Dirty-hack-to-clobber-control-file-for-testing.patch"
Content-Disposition: attachment;
	filename="0001-XXX-Dirty-hack-to-clobber-control-file-for-testing.patch"
Content-Transfer-Encoding: base64
Content-ID: <f_lauddif40>
X-Attachment-Id: f_lauddif40

RnJvbSA5ZjE4NTZhZWI1NmJlODg4MTIzNzAyZmU0NzFkODM4OGRhNjY0MzlmIE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpGcm9tOiBUaG9tYXMgTXVucm8gPHRob21hcy5tdW5yb0BnbWFpbC5jb20+
CkRhdGU6IFRodSwgMjQgTm92IDIwMjIgMDA6NTU6MDMgKzAwMDAKU3ViamVjdDogW1BBVENIIDEv
Ml0gWFhYIERpcnR5IGhhY2sgdG8gY2xvYmJlciBjb250cm9sIGZpbGUgZm9yIHRlc3RpbmcKCi0t
LQogc3JjL2JhY2tlbmQvYWNjZXNzL3RyYW5zYW0veGxvZy5jIHwgMTAgKysrKysrKysrKwogc3Jj
L2luY2x1ZGUvY2F0YWxvZy9wZ19wcm9jLmRhdCAgIHwgIDggKysrKysrKysKIDIgZmlsZXMgY2hh
bmdlZCwgMTggaW5zZXJ0aW9ucygrKQoKZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL2FjY2Vzcy90
cmFuc2FtL3hsb2cuYyBiL3NyYy9iYWNrZW5kL2FjY2Vzcy90cmFuc2FtL3hsb2cuYwppbmRleCBh
MzFmYmJmZjc4Li44OGRlMDVhYjM1IDEwMDY0NAotLS0gYS9zcmMvYmFja2VuZC9hY2Nlc3MvdHJh
bnNhbS94bG9nLmMKKysrIGIvc3JjL2JhY2tlbmQvYWNjZXNzL3RyYW5zYW0veGxvZy5jCkBAIC0y
NTAyLDYgKzI1MDIsMTYgQEAgVXBkYXRlTWluUmVjb3ZlcnlQb2ludChYTG9nUmVjUHRyIGxzbiwg
Ym9vbCBmb3JjZSkKIAlMV0xvY2tSZWxlYXNlKENvbnRyb2xGaWxlTG9jayk7CiB9CiAKK0RhdHVt
CitwZ191cGRhdGVfY29udHJvbF9maWxlKCkKK3sKKwlMV0xvY2tBY3F1aXJlKENvbnRyb2xGaWxl
TG9jaywgTFdfRVhDTFVTSVZFKTsKKwlDb250cm9sRmlsZS0+bWluUmVjb3ZlcnlQb2ludCsrOwkJ
LyogWFhYIGNoYW5nZSBzb21ldGhpbmcgdG8gYWZmZWN0IENSQyEgKi8KKwlVcGRhdGVDb250cm9s
RmlsZSgpOworCUxXTG9ja1JlbGVhc2UoQ29udHJvbEZpbGVMb2NrKTsKKwlQR19SRVRVUk5fVk9J
RCgpOworfQorCiAvKgogICogRW5zdXJlIHRoYXQgYWxsIFhMT0cgZGF0YSB0aHJvdWdoIHRoZSBn
aXZlbiBwb3NpdGlvbiBpcyBmbHVzaGVkIHRvIGRpc2suCiAgKgpkaWZmIC0tZ2l0IGEvc3JjL2lu
Y2x1ZGUvY2F0YWxvZy9wZ19wcm9jLmRhdCBiL3NyYy9pbmNsdWRlL2NhdGFsb2cvcGdfcHJvYy5k
YXQKaW5kZXggZjE1YWEyZGJiMS4uODE3N2MxNjU3YyAxMDA2NDQKLS0tIGEvc3JjL2luY2x1ZGUv
Y2F0YWxvZy9wZ19wcm9jLmRhdAorKysgYi9zcmMvaW5jbHVkZS9jYXRhbG9nL3BnX3Byb2MuZGF0
CkBAIC0xMTY3OSw2ICsxMTY3OSwxNCBAQAogICBwcm9hcmduYW1lcyA9PiAne3BnX2NvbnRyb2xf
dmVyc2lvbixjYXRhbG9nX3ZlcnNpb25fbm8sc3lzdGVtX2lkZW50aWZpZXIscGdfY29udHJvbF9s
YXN0X21vZGlmaWVkfScsCiAgIHByb3NyYyA9PiAncGdfY29udHJvbF9zeXN0ZW0nIH0sCiAKK3sg
b2lkID0+ICc4ODg4JywKKyAgZGVzY3IgPT4gJ3VwZGF0ZSBjb250cm9sIGZpbGUnLAorICBwcm9u
YW1lID0+ICdwZ191cGRhdGVfY29udHJvbF9maWxlJywgcHJvdm9sYXRpbGUgPT4gJ3YnLCBwcm9y
ZXR0eXBlID0+ICd2b2lkJywKKyAgcHJvYXJndHlwZXMgPT4gJycsIHByb2FsbGFyZ3R5cGVzID0+
ICcnLAorICBwcm9hcmdtb2RlcyA9PiAne30nLAorICBwcm9hcmduYW1lcyA9PiAne30nLAorICBw
cm9zcmMgPT4gJ3BnX3VwZGF0ZV9jb250cm9sX2ZpbGUnIH0sCisKIHsgb2lkID0+ICczNDQyJywK
ICAgZGVzY3IgPT4gJ3BnX2NvbnRyb2xkYXRhIGNoZWNrcG9pbnQgc3RhdGUgaW5mb3JtYXRpb24g
YXMgYSBmdW5jdGlvbicsCiAgIHByb25hbWUgPT4gJ3BnX2NvbnRyb2xfY2hlY2twb2ludCcsIHBy
b3ZvbGF0aWxlID0+ICd2JywKLS0gCjIuMzUuMQoK
--000000000000459f6e05ee2cf982
Content-Type: text/x-patch; charset="US-ASCII";
	name="0002-Try-to-tolerate-concurrent-reads-and-writes-of-contr.patch"
Content-Disposition: attachment;
	filename="0002-Try-to-tolerate-concurrent-reads-and-writes-of-contr.patch"
Content-Transfer-Encoding: base64
Content-ID: <f_lauddkhm1>
X-Attachment-Id: f_lauddkhm1

RnJvbSBhNjM4MThhMzJkNjYxZGJhNTYzY2VkZmRiODU3MzFlNTIyYjNjNmE5IE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpGcm9tOiBUaG9tYXMgTXVucm8gPHRob21hcy5tdW5yb0BnbWFpbC5jb20+
CkRhdGU6IFRodSwgMjQgTm92IDIwMjIgMTM6Mjg6MjIgKzEzMDAKU3ViamVjdDogW1BBVENIIDIv
Ml0gVHJ5IHRvIHRvbGVyYXRlIGNvbmN1cnJlbnQgcmVhZHMgYW5kIHdyaXRlcyBvZiBjb250cm9s
CiBmaWxlLgoKVmFyaW91cyBmcm9udGVuZCBwcm9ncmFtcyBhbmQgU1FMLWNhbGxhYmxlIGJhY2tl
bmQgZnVuY3Rpb25zIHJlYWQgdGhlCmNvbnRyb2wgZmlsZSB3aXRob3V0IGFueSBraW5kIG9mIGlu
dGVybG9ja2luZyBhZ2FpbnN0IGNvbmN1cnJlbnQgd3JpdGVzLgpMaW51eCBleHQ0IGRvZXNuJ3Qg
aW1wbGVtZW50IHRoZSBhdG9taWNpdHkgcmVxdWlyZWQgYnkgUE9TSVggaGVyZSwgc28gYQpjb25j
dXJyZW50IHJlYWRlciBjYW4gc2VlIG9ubHkgcGFydGlhbCBlZmZlY3RzIG9mIGFuIGluLXByb2dy
ZXNzIHdyaXRlLgoKVG9sZXJhdGUgdGhpcyBieSByZXRyeWluZyB1bnRpbCB3ZSBnZXQgdHdvIHJl
YWRzIGluIGEgcm93IHdpdGggdGhlIHNhbWUKY2hlY2tzdW0sIGFmdGVyIGFuIGlkZWEgZnJvbSBU
b20gTGFuZS4KClJlcG9ydGVkLWJ5OiBBbmRyZXMgRnJldW5kIDxhbmRyZXNAYW5hcmF6ZWwuZGU+
CkRpc2N1c3Npb246IGh0dHBzOi8vcG9zdGdyLmVzL20vMjAyMjExMjMwMTQyMjQueGlzaTQ0Ynlx
M2NmNXBzaSU0MGF3b3JrMy5hbmFyYXplbC5kZQotLS0KIHNyYy9jb21tb24vY29udHJvbGRhdGFf
dXRpbHMuYyB8IDI1ICsrKysrKysrKysrKysrKysrKysrKysrKysKIDEgZmlsZSBjaGFuZ2VkLCAy
NSBpbnNlcnRpb25zKCspCgpkaWZmIC0tZ2l0IGEvc3JjL2NvbW1vbi9jb250cm9sZGF0YV91dGls
cy5jIGIvc3JjL2NvbW1vbi9jb250cm9sZGF0YV91dGlscy5jCmluZGV4IDJkMWYzNWJiZDEuLjIw
MGQyNGRmMDIgMTAwNjQ0Ci0tLSBhL3NyYy9jb21tb24vY29udHJvbGRhdGFfdXRpbHMuYworKysg
Yi9zcmMvY29tbW9uL2NvbnRyb2xkYXRhX3V0aWxzLmMKQEAgLTU2LDEyICs1NiwxOSBAQCBnZXRf
Y29udHJvbGZpbGUoY29uc3QgY2hhciAqRGF0YURpciwgYm9vbCAqY3JjX29rX3ApCiAJY2hhcgkJ
Q29udHJvbEZpbGVQYXRoW01BWFBHUEFUSF07CiAJcGdfY3JjMzJjCWNyYzsKIAlpbnQJCQlyOwor
CWJvb2wJCWZpcnN0X3RyeTsKKwlwZ19jcmMzMmMJbGFzdF9jcmM7CiAKIAlBc3NlcnQoY3JjX29r
X3ApOwogCiAJQ29udHJvbEZpbGUgPSBwYWxsb2Nfb2JqZWN0KENvbnRyb2xGaWxlRGF0YSk7CiAJ
c25wcmludGYoQ29udHJvbEZpbGVQYXRoLCBNQVhQR1BBVEgsICIlcy9nbG9iYWwvcGdfY29udHJv
bCIsIERhdGFEaXIpOwogCisJZmlyc3RfdHJ5ID0gdHJ1ZTsKKwlJTklUX0NSQzMyQyhsYXN0X2Ny
Yyk7CisKK3JldHJ5OgorCiAjaWZuZGVmIEZST05URU5ECiAJaWYgKChmZCA9IE9wZW5UcmFuc2ll
bnRGaWxlKENvbnRyb2xGaWxlUGF0aCwgT19SRE9OTFkgfCBQR19CSU5BUlkpKSA9PSAtMSkKIAkJ
ZXJlcG9ydChFUlJPUiwKQEAgLTExNyw2ICsxMjQsMjQgQEAgZ2V0X2NvbnRyb2xmaWxlKGNvbnN0
IGNoYXIgKkRhdGFEaXIsIGJvb2wgKmNyY19va19wKQogCiAJKmNyY19va19wID0gRVFfQ1JDMzJD
KGNyYywgQ29udHJvbEZpbGUtPmNyYyk7CiAKKwkvKgorCSAqIFdpdGggdW5sdWNreSB0aW1pbmcg
b24gZmlsZXN5c3RlbXMgdGhhdCBkb24ndCBpbXBsZW1lbnQgYXRvbWljaXR5IG9mCisJICogY29u
Y3VycmVudCByZWFkcyBhbmQgd3JpdGVzIChzdWNoIGFzIExpbnV4IGV4dDQpLCB3ZSBtaWdodCBo
YXZlIHNlZW4KKwkgKiBnYXJiYWdlIGlmIHRoZSBzZXJ2ZXIgd2FzIHdyaXRpbmcgdG8gdGhlIGZp
bGUgYXQgdGhlIHNhbWUgdGltZS4gIEtlZXAKKwkgKiByZXRyeWluZyB1bnRpbCB3ZSBzZWUgdGhl
IHNhbWUgQ1JDIHR3aWNlLgorCSAqLworCWlmICghKmNyY19va19wICYmIChmaXJzdF90cnkgfHwg
IUVRX0NSQzMyQyhjcmMsIGxhc3RfY3JjKSkpCisJeworCQlmaXJzdF90cnkgPSBmYWxzZTsKKwkJ
bGFzdF9jcmMgPSBjcmM7CisJCXBnX3VzbGVlcCgxMDAwMCk7CisKKyNpZm5kZWYgRlJPTlRFTkQK
KwkJQ0hFQ0tfRk9SX0lOVEVSUlVQVFMoKTsKKyNlbmRpZgorCQlnb3RvIHJldHJ5OworCX0KKwog
CS8qIE1ha2Ugc3VyZSB0aGUgY29udHJvbCBmaWxlIGlzIHZhbGlkIGJ5dGUgb3JkZXIuICovCiAJ
aWYgKENvbnRyb2xGaWxlLT5wZ19jb250cm9sX3ZlcnNpb24gJSA2NTUzNiA9PSAwICYmCiAJCUNv
bnRyb2xGaWxlLT5wZ19jb250cm9sX3ZlcnNpb24gLyA2NTUzNiAhPSAwKQotLSAKMi4zNS4xCgo=
--000000000000459f6e05ee2cf982--