Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oqJ8e-0001BF-6x for pgsql-hackers@arkaria.postgresql.org; Wed, 02 Nov 2022 19:11:00 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1oqJ7e-0007GL-UK for pgsql-hackers@arkaria.postgresql.org; Wed, 02 Nov 2022 19:09:58 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oqJ7e-0007Dj-Kn for pgsql-hackers@lists.postgresql.org; Wed, 02 Nov 2022 19:09:58 +0000 Received: from relay1-d.mail.gandi.net ([217.70.183.193]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oqJ7b-0006Fo-Ce for pgsql-hackers@postgresql.org; Wed, 02 Nov 2022 19:09:57 +0000 Received: (Authenticated sender: adsend@dunslane.net) by mail.gandi.net (Postfix) with ESMTPSA id D88F5240002; Wed, 2 Nov 2022 19:09:49 +0000 (UTC) Content-Type: multipart/mixed; boundary="------------dSgAXmmFck2DgVVMuDa67LNh" Message-ID: <0891f64a-d30f-a1b7-9d0c-e447684cd8cd@dunslane.net> Date: Wed, 2 Nov 2022 15:09:47 -0400 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.5.0 Subject: Re: ssl tests aren't concurrency safe due to get_free_port() Content-Language: en-US From: Andrew Dunstan To: Andres Freund , pgsql-hackers@postgresql.org, Tom Lane References: <20221002164931.d57hlutrcz4d2zi7@awork3.anarazel.de> <7579f627-aad4-7a04-1a98-033cc6b3cca5@dunslane.net> <52266bf8-bf38-fdfa-9a77-1568f82b63b3@dunslane.net> In-Reply-To: <52266bf8-bf38-fdfa-9a77-1568f82b63b3@dunslane.net> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk This is a multi-part message in MIME format. --------------dSgAXmmFck2DgVVMuDa67LNh Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On 2022-10-17 Mo 10:59, Andrew Dunstan wrote: > On 2022-10-04 Tu 01:39, Andrew Dunstan wrote: >> On 2022-10-02 Su 12:49, Andres Freund wrote: >>> 2) Use a lockfile containing a pid to protect the choice of a port within a >>> build directory. Before accepting a port get_free_port() would check if the >>> a lockfile exists for the port and if so, if the test using it is still >>> alive. That will protect against racyness between multiple tests inside a >>> build directory, but won't protect against races between multiple builds >>> running concurrently on a machine (e.g. on a buildfarm host) >>> >>> >> I think this is the right solution. To deal with the last issue, the >> lockdir should be overrideable, like this: >> >> >>   my $port_lockdir = $ENV{PG_PORT_LOCKDIR} || $build_dir; >> >> >> Buildfarm animals could set this, probably to the global lockdir (see >> run_branches.pl). Prior to that, buildfarm owners could do that manually. >> >> > The problem here is that Cluster.pm doesn't have any idea where the > build directory is, or even if there is one present at all. > > meson does appear to let us know that, however, with MESON_BUILD_ROOT, > so probably the best thing would be to use PG_PORT_LOCKDIR if it's set, > otherwise MESON_BUILD_ROOT if it's set, otherwise the tmp_check > directory. If we want to backport to the make system we could export > top_builddir somewhere. > > Here's a patch which I think does the right thing. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com --------------dSgAXmmFck2DgVVMuDa67LNh Content-Type: text/x-patch; charset=UTF-8; name="prevent-concurrent-free-port.patch" Content-Disposition: attachment; filename="prevent-concurrent-free-port.patch" Content-Transfer-Encoding: base64 ZGlmZiAtLWdpdCBhL3NyYy90ZXN0L3BlcmwvUG9zdGdyZVNRTC9UZXN0L0NsdXN0ZXIucG0g Yi9zcmMvdGVzdC9wZXJsL1Bvc3RncmVTUUwvVGVzdC9DbHVzdGVyLnBtCmluZGV4IGQ4MDEz NGIyNmYuLjI1OTQ2OThmMWYgMTAwNjQ0Ci0tLSBhL3NyYy90ZXN0L3BlcmwvUG9zdGdyZVNR TC9UZXN0L0NsdXN0ZXIucG0KKysrIGIvc3JjL3Rlc3QvcGVybC9Qb3N0Z3JlU1FML1Rlc3Qv Q2x1c3Rlci5wbQpAQCAtOTMsOSArOTMsOSBAQCB1c2Ugd2FybmluZ3M7CiAKIHVzZSBDYXJw OwogdXNlIENvbmZpZzsKLXVzZSBGY250bCBxdyg6bW9kZSk7Cit1c2UgRmNudGwgcXcoOm1v ZGUgOmZsb2NrIDpzZWVrIE9fQ1JFQVQgT19SRFdSKTsKIHVzZSBGaWxlOjpCYXNlbmFtZTsK LXVzZSBGaWxlOjpQYXRoIHF3KHJtdHJlZSk7Cit1c2UgRmlsZTo6UGF0aCBxdyhybXRyZWUg bWtwYXRoKTsKIHVzZSBGaWxlOjpTcGVjOwogdXNlIEZpbGU6OnN0YXQgcXcoc3RhdCk7CiB1 c2UgRmlsZTo6VGVtcCAoKTsKQEAgLTEwOSw3ICsxMDksNyBAQCB1c2UgVGltZTo6SGlSZXMg cXcodXNsZWVwKTsKIHVzZSBTY2FsYXI6OlV0aWwgcXcoYmxlc3NlZCk7CiAKIG91ciAoJHVz ZV90Y3AsICR0ZXN0X2xvY2FsaG9zdCwgJHRlc3RfcGdob3N0LCAkbGFzdF9ob3N0X2Fzc2ln bmVkLAotCSRsYXN0X3BvcnRfYXNzaWduZWQsIEBhbGxfbm9kZXMsICRkaWVkKTsKKwkkbGFz dF9wb3J0X2Fzc2lnbmVkLCBAYWxsX25vZGVzLCAkZGllZCwgJHBvcnRkaXIpOwogCiAjIHRo ZSBtaW5pbXVtIHZlcnNpb24gd2UgYmVsaWV2ZSB0byBiZSBjb21wYXRpYmxlIHdpdGggdGhp cyBwYWNrYWdlIHdpdGhvdXQKICMgc3ViY2xhc3NpbmcuCkBAIC0xNDAsNiArMTQwLDI5IEBA IElOSVQKIAogCSMgVHJhY2tpbmcgb2YgbGFzdCBwb3J0IHZhbHVlIGFzc2lnbmVkIHRvIGFj Y2VsZXJhdGUgZnJlZSBwb3J0IGxvb2t1cC4KIAkkbGFzdF9wb3J0X2Fzc2lnbmVkID0gaW50 KHJhbmQoKSAqIDE2Mzg0KSArIDQ5MTUyOworCisJIyBTZXQgdGhlIHBvcnQgbG9jayBkaXJl Y3RvcnkKKworCSMgSWYgd2UncmUgdG9sZCB0byB1c2UgYSBkaXJlY3RvcnkgKGUuZy4gZnJv bSBhIGJ1aWxkZmFybSBjbGllbnQpCisJIyBleHBsaWNpdGx5LCB1c2UgdGhhdAorCSRwb3J0 ZGlyID0gJEVOVntQR19URVNUX1BPUlRfRElSfTsKKwkjIE90aGVyd2lzZSwgdHJ5IHRvIHVz ZSBhIGRpcmVjdG9yeSBhdCB0aGUgdG9wIG9mIHRoZSBidWlsZCB0cmVlCisJaWYgKCEgJHBv cnRkaXIgJiYgZXhpc3RzICRFTlZ7TUVTT05fQlVJTERfUk9PVH0pCisJeworCQlteSAkZGly ID0gJEVOVntNRVNPTl9CVUlMRF9ST09UfTsKKwkJJGRpciA9fiBzIVxcIS8hZzsKKwkJJHBv cnRkaXIgPSAiJGRpci9wb3J0bG9jayIgaWYgJGRpcjsKKwl9CisJZWxzaWYgKCEgJHBvcnRk aXIgJiYgJEVOVntURVNUREFUQURJUn0gPX4gL1xXKHNyY3xjb250cmliKVxXL3ApCisJewor CQlteSAkZGlyID0gJHteUFJFTUFUQ0h9OworCQkkZGlyID1+IHMhXFwhLyFnOworCQkkcG9y dGRpciA9ICIkZGlyL3BvcnRsb2NrIiBpZiAkZGlyOworCX0KKwkjIEFzIGEgbGFzdCByZXNv cnQgdXNlIGEgdGVtcCBkaXJlY3RvcnkgdW5kZXIgdG1wX2NoZWNrCisJJHBvcnRkaXIgfHw9 IFBvc3RncmVTUUw6OlRlc3Q6OlV0aWxzOjp0ZW1wZGlyKCdwb3J0bG9jaycpOworCSMgTWFr ZSBzdXJlIHRoZSBkaXJlY3RvcnkgZXhpc3RzCisJbWtwYXRoKCRwb3J0ZGlyKSB1bmxlc3Mg LWQgJHBvcnRkaXI7CiB9CiAKID1wb2QKQEAgLTE1MDUsNiArMTUyOCw3IEBAIHN1YiBnZXRf ZnJlZV9wb3J0CiAJCQkJCWxhc3Q7CiAJCQkJfQogCQkJfQorCQkJJGZvdW5kID0gX3Jlc2Vy dmVfcG9ydCgkcG9ydCkgaWYgJGZvdW5kOwogCQl9CiAJfQogCkBAIC0xNTM1LDYgKzE1NTks MzcgQEAgc3ViIGNhbl9iaW5kCiAJcmV0dXJuICRyZXQ7CiB9CiAKKyMgaW50ZXJuYWwgcm91 dGluZSB0byByZXNlcnZlIGEgcG9ydCBudW1iZXIKKyMgcmV0dXJucyAxIGlmIHN1Y2Nlc3Nm dWwsIDAgaWYgcG9ydCBpcyBhbHJlYWR5IHJlc2VydmVkLgorc3ViIF9yZXNlcnZlX3BvcnQK K3sKKwlteSAkcG9ydCA9IHNoaWZ0OworCSMgb3BlbiBpbiBydyBtb2RlIHNvIHdlIGRvbid0 IGhhdmUgdG8gcmVvcGVuIGl0IGFuZCBsb3NlIHRoZSBsb2NrCisJc3lzb3BlbihteSAkcG9y dGZpbGUsICIkcG9ydGRpci8kcG9ydC5yc3YiLCBPX1JEV1J8T19DUkVBVCkKKwkgIHx8IGRp ZSAib3BlbmluZyBwb3J0IGZpbGUiOworCSMgdGFrZSBhbiBleGNsdXNpdmUgbG9jayB0byBh dm9pZCBjb25jdXJyZW50IGFjY2VzcworCWZsb2NrKCRwb3J0ZmlsZSwgTE9DS19FWCkgfHwg ZGllICJsb2NraW5nIHBvcnQgZmlsZSI7CisJIyBzZWUgaWYgc29tZW9uZSBlbHNlIGhhcyBv ciBoYWQgYSByZXNlcnZhdGlvbiBvZiB0aGlzIHBvcnQKKwlteSAkcGlkID0gPCRwb3J0Zmls ZT47CisJY2hvbXAgJHBpZDsKKwlpZiAoJHBpZCArMCA+IDApCisJeworCQlpZiAoa2lsbCAw LCAkcGlkKQorCQl7CisJCQkjIHByb2Nlc3MgZXhpc3RzIGFuZCBpcyBvd25lZCBieSB1cywg c28gd2UgY2FuJ3QgcmVzZXJ2ZSB0aGlzIHBvcnQKKwkJCWNsb3NlKCRwb3J0ZmlsZSk7CisJ CQlyZXR1cm4gMDsKKwkJfQorCX0KKwkjIGFsbCBnb29kLCBnbyBhaGVhZCBhbmQgcmVzZXJ2 ZSB0aGUgcG9ydAorCSMgZmlyc3QgcmV3aW5kIGFuZCB0cnVuY2F0ZSB0aGUgZmlsZQorCXNl ZWsoJHBvcnRmaWxlLCAwLCBTRUVLX1NFVCk7CisJdHJ1bmNhdGUoJHBvcnRmaWxlLCAwKTsK KwlwcmludCAkcG9ydGZpbGUgIiQkXG4iOworCWNsb3NlKCRwb3J0ZmlsZSk7CisJcmV0dXJu IDE7Cit9CisKICMgQXV0b21hdGljYWxseSBzaHV0IGRvd24gYW55IHN0aWxsLXJ1bm5pbmcg bm9kZXMgKGluIHRoZSBzYW1lIG9yZGVyIHRoZSBub2RlcwogIyB3ZXJlIGNyZWF0ZWQgaW4p IHdoZW4gdGhlIHRlc3Qgc2NyaXB0IGV4aXRzLgogRU5ECg== --------------dSgAXmmFck2DgVVMuDa67LNh--