Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oqem0-0007H8-Vn for pgsql-hackers@arkaria.postgresql.org; Thu, 03 Nov 2022 18:17:05 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1oqelz-0004wI-Nj for pgsql-hackers@arkaria.postgresql.org; Thu, 03 Nov 2022 18:17:03 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oqelz-0004w9-Eo for pgsql-hackers@lists.postgresql.org; Thu, 03 Nov 2022 18:17:03 +0000 Received: from relay5-d.mail.gandi.net ([2001:4b98:dc4:8::225]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oqelw-0000c6-Jd for pgsql-hackers@postgresql.org; Thu, 03 Nov 2022 18:17:02 +0000 Received: (Authenticated sender: adsend@dunslane.net) by mail.gandi.net (Postfix) with ESMTPSA id A75F81C000B; Thu, 3 Nov 2022 18:16:53 +0000 (UTC) Content-Type: multipart/mixed; boundary="------------8DyEHGNCodPOClIlWlQEvTO0" Message-ID: Date: Thu, 3 Nov 2022 14:16:51 -0400 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.5.0 Subject: Re: ssl tests aren't concurrency safe due to get_free_port() Content-Language: en-US From: Andrew Dunstan To: Andres Freund , pgsql-hackers@postgresql.org, Tom Lane References: <20221002164931.d57hlutrcz4d2zi7@awork3.anarazel.de> <7579f627-aad4-7a04-1a98-033cc6b3cca5@dunslane.net> <52266bf8-bf38-fdfa-9a77-1568f82b63b3@dunslane.net> <0891f64a-d30f-a1b7-9d0c-e447684cd8cd@dunslane.net> In-Reply-To: <0891f64a-d30f-a1b7-9d0c-e447684cd8cd@dunslane.net> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk This is a multi-part message in MIME format. --------------8DyEHGNCodPOClIlWlQEvTO0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On 2022-11-02 We 15:09, Andrew Dunstan wrote: > On 2022-10-17 Mo 10:59, Andrew Dunstan wrote: >> On 2022-10-04 Tu 01:39, Andrew Dunstan wrote: >>> On 2022-10-02 Su 12:49, Andres Freund wrote: >>>> 2) Use a lockfile containing a pid to protect the choice of a port within a >>>> build directory. Before accepting a port get_free_port() would check if the >>>> a lockfile exists for the port and if so, if the test using it is still >>>> alive. That will protect against racyness between multiple tests inside a >>>> build directory, but won't protect against races between multiple builds >>>> running concurrently on a machine (e.g. on a buildfarm host) >>>> >>>> >>> I think this is the right solution. To deal with the last issue, the >>> lockdir should be overrideable, like this: >>> >>> >>>   my $port_lockdir = $ENV{PG_PORT_LOCKDIR} || $build_dir; >>> >>> >>> Buildfarm animals could set this, probably to the global lockdir (see >>> run_branches.pl). Prior to that, buildfarm owners could do that manually. >>> >>> >> The problem here is that Cluster.pm doesn't have any idea where the >> build directory is, or even if there is one present at all. >> >> meson does appear to let us know that, however, with MESON_BUILD_ROOT, >> so probably the best thing would be to use PG_PORT_LOCKDIR if it's set, >> otherwise MESON_BUILD_ROOT if it's set, otherwise the tmp_check >> directory. If we want to backport to the make system we could export >> top_builddir somewhere. >> >> > Here's a patch which I think does the right thing. > > > Updated with a couple of thinkos fixed. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com --------------8DyEHGNCodPOClIlWlQEvTO0 Content-Type: text/x-patch; charset=UTF-8; name="prevent-concurrent-free-port-v2.patch" Content-Disposition: attachment; filename="prevent-concurrent-free-port-v2.patch" Content-Transfer-Encoding: base64 ZGlmZiAtLWdpdCBhL3NyYy90ZXN0L3BlcmwvUG9zdGdyZVNRTC9UZXN0L0NsdXN0ZXIucG0g Yi9zcmMvdGVzdC9wZXJsL1Bvc3RncmVTUUwvVGVzdC9DbHVzdGVyLnBtCmluZGV4IGQ4MDEz NGIyNmYuLmFjZWNhMzUzZDMgMTAwNjQ0Ci0tLSBhL3NyYy90ZXN0L3BlcmwvUG9zdGdyZVNR TC9UZXN0L0NsdXN0ZXIucG0KKysrIGIvc3JjL3Rlc3QvcGVybC9Qb3N0Z3JlU1FML1Rlc3Qv Q2x1c3Rlci5wbQpAQCAtOTMsOSArOTMsOSBAQCB1c2Ugd2FybmluZ3M7CiAKIHVzZSBDYXJw OwogdXNlIENvbmZpZzsKLXVzZSBGY250bCBxdyg6bW9kZSk7Cit1c2UgRmNudGwgcXcoOm1v ZGUgOmZsb2NrIDpzZWVrIE9fQ1JFQVQgT19SRFdSKTsKIHVzZSBGaWxlOjpCYXNlbmFtZTsK LXVzZSBGaWxlOjpQYXRoIHF3KHJtdHJlZSk7Cit1c2UgRmlsZTo6UGF0aCBxdyhybXRyZWUg bWtwYXRoKTsKIHVzZSBGaWxlOjpTcGVjOwogdXNlIEZpbGU6OnN0YXQgcXcoc3RhdCk7CiB1 c2UgRmlsZTo6VGVtcCAoKTsKQEAgLTEwOSw3ICsxMDksNyBAQCB1c2UgVGltZTo6SGlSZXMg cXcodXNsZWVwKTsKIHVzZSBTY2FsYXI6OlV0aWwgcXcoYmxlc3NlZCk7CiAKIG91ciAoJHVz ZV90Y3AsICR0ZXN0X2xvY2FsaG9zdCwgJHRlc3RfcGdob3N0LCAkbGFzdF9ob3N0X2Fzc2ln bmVkLAotCSRsYXN0X3BvcnRfYXNzaWduZWQsIEBhbGxfbm9kZXMsICRkaWVkKTsKKwkkbGFz dF9wb3J0X2Fzc2lnbmVkLCBAYWxsX25vZGVzLCAkZGllZCwgJHBvcnRkaXIpOwogCiAjIHRo ZSBtaW5pbXVtIHZlcnNpb24gd2UgYmVsaWV2ZSB0byBiZSBjb21wYXRpYmxlIHdpdGggdGhp cyBwYWNrYWdlIHdpdGhvdXQKICMgc3ViY2xhc3NpbmcuCkBAIC0xNDAsNiArMTQwLDI3IEBA IElOSVQKIAogCSMgVHJhY2tpbmcgb2YgbGFzdCBwb3J0IHZhbHVlIGFzc2lnbmVkIHRvIGFj Y2VsZXJhdGUgZnJlZSBwb3J0IGxvb2t1cC4KIAkkbGFzdF9wb3J0X2Fzc2lnbmVkID0gaW50 KHJhbmQoKSAqIDE2Mzg0KSArIDQ5MTUyOworCisJIyBTZXQgdGhlIHBvcnQgbG9jayBkaXJl Y3RvcnkKKworCSMgSWYgd2UncmUgdG9sZCB0byB1c2UgYSBkaXJlY3RvcnkgKGUuZy4gZnJv bSBhIGJ1aWxkZmFybSBjbGllbnQpCisJIyBleHBsaWNpdGx5LCB1c2UgdGhhdAorCSRwb3J0 ZGlyID0gJEVOVntQR19URVNUX1BPUlRfRElSfTsKKwkjIE90aGVyd2lzZSwgdHJ5IHRvIHVz ZSBhIGRpcmVjdG9yeSBhdCB0aGUgdG9wIG9mIHRoZSBidWlsZCB0cmVlCisJaWYgKCEgJHBv cnRkaXIgJiYgJEVOVntNRVNPTl9CVUlMRF9ST09UfSkKKwl7CisJCSRwb3J0ZGlyID0gJEVO VntNRVNPTl9CVUlMRF9ST09UfSAuICcvcG9ydGxvY2snCisJfQorCWVsc2lmICghICRwb3J0 ZGlyICYmICgkRU5We1RFU1REQVRBRElSfSB8fCAiIikgPX4gL1xXKHNyY3xjb250cmliKVxX L3ApCisJeworCQlteSAkZGlyID0gJHteUFJFTUFUQ0h9OworCQkkcG9ydGRpciA9ICIkZGly L3BvcnRsb2NrIiBpZiAkZGlyOworCX0KKwkjIEFzIGEgbGFzdCByZXNvcnQgdXNlIGEgZGly ZWN0b3J5IHVuZGVyIHRtcF9jaGVjaworCSRwb3J0ZGlyIHx8PSAkUG9zdGdyZVNRTDo6VGVz dDo6VXRpbHM6OnRtcF9jaGVjayAuICcvcG9ydGxvY2snOworCSRwb3J0ZGlyID1+IHMhXFwh LyFnOworCSMgTWFrZSBzdXJlIHRoZSBkaXJlY3RvcnkgZXhpc3RzCisJbWtwYXRoKCRwb3J0 ZGlyKSB1bmxlc3MgLWQgJHBvcnRkaXI7CiB9CiAKID1wb2QKQEAgLTE1MDUsNiArMTUyNiw3 IEBAIHN1YiBnZXRfZnJlZV9wb3J0CiAJCQkJCWxhc3Q7CiAJCQkJfQogCQkJfQorCQkJJGZv dW5kID0gX3Jlc2VydmVfcG9ydCgkcG9ydCkgaWYgJGZvdW5kOwogCQl9CiAJfQogCkBAIC0x NTM1LDYgKzE1NTcsMzggQEAgc3ViIGNhbl9iaW5kCiAJcmV0dXJuICRyZXQ7CiB9CiAKKyMg SW50ZXJuYWwgcm91dGluZSB0byByZXNlcnZlIGEgcG9ydCBudW1iZXIKKyMgUmV0dXJucyAx IGlmIHN1Y2Nlc3NmdWwsIDAgaWYgcG9ydCBpcyBhbHJlYWR5IHJlc2VydmVkLgorc3ViIF9y ZXNlcnZlX3BvcnQKK3sKKwlteSAkcG9ydCA9IHNoaWZ0OworCSMgb3BlbiBpbiBydyBtb2Rl IHNvIHdlIGRvbid0IGhhdmUgdG8gcmVvcGVuIGl0IGFuZCBsb3NlIHRoZSBsb2NrCisJc3lz b3BlbihteSAkcG9ydGZpbGUsICIkcG9ydGRpci8kcG9ydC5yc3YiLCBPX1JEV1J8T19DUkVB VCkKKwkgIHx8IGRpZSAib3BlbmluZyBwb3J0IGZpbGUiOworCSMgdGFrZSBhbiBleGNsdXNp dmUgbG9jayB0byBhdm9pZCBjb25jdXJyZW50IGFjY2VzcworCWZsb2NrKCRwb3J0ZmlsZSwg TE9DS19FWCkgfHwgZGllICJsb2NraW5nIHBvcnQgZmlsZSI7CisJIyBzZWUgaWYgc29tZW9u ZSBlbHNlIGhhcyBvciBoYWQgYSByZXNlcnZhdGlvbiBvZiB0aGlzIHBvcnQKKwlteSAkcGlk ID0gPCRwb3J0ZmlsZT47CisJY2hvbXAgJHBpZDsKKwlpZiAoJHBpZCArMCA+IDApCisJewor CQlpZiAoa2lsbCAwLCAkcGlkKQorCQl7CisJCQkjIHByb2Nlc3MgZXhpc3RzIGFuZCBpcyBv d25lZCBieSB1cywgc28gd2UgY2FuJ3QgcmVzZXJ2ZSB0aGlzIHBvcnQKKwkJCWNsb3NlKCRw b3J0ZmlsZSk7CisJCQlyZXR1cm4gMDsKKwkJfQorCX0KKwkjIEFsbCBnb29kLCBnbyBhaGVh ZCBhbmQgcmVzZXJ2ZSB0aGUgcG9ydCwgZmlyc3QgcmV3aW5kIGFuZCB0cnVuY2F0ZS4KKwkj IElmIHRydW5jYXRpb24gZmFpbHMgaXQncyBub3QgYSB0cmFnZWR5LCBpdCBqdXN0IG1pZ2h0 IGxlYXZlIHNvbWUKKwkjIHRyYWlsaW5nIGp1bmsgaW4gdGhlIGZpbGUgdGhhdCB3b24ndCBh ZmZlY3QgdXMuCisJc2VlaygkcG9ydGZpbGUsIDAsIFNFRUtfU0VUKTsKKwl0cnVuY2F0ZSgk cG9ydGZpbGUsIDApOworCXByaW50ICRwb3J0ZmlsZSAiJCRcbiI7CisJY2xvc2UoJHBvcnRm aWxlKTsKKwlyZXR1cm4gMTsKK30KKwogIyBBdXRvbWF0aWNhbGx5IHNodXQgZG93biBhbnkg c3RpbGwtcnVubmluZyBub2RlcyAoaW4gdGhlIHNhbWUgb3JkZXIgdGhlIG5vZGVzCiAjIHdl cmUgY3JlYXRlZCBpbikgd2hlbiB0aGUgdGVzdCBzY3JpcHQgZXhpdHMuCiBFTkQK --------------8DyEHGNCodPOClIlWlQEvTO0--