Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jhcNB-0006qI-H9 for pgsql-bugs@arkaria.postgresql.org; Sat, 06 Jun 2020 17:12:45 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1jhcN8-0006ll-Ra for pgsql-bugs@arkaria.postgresql.org; Sat, 06 Jun 2020 17:12:42 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jhcN8-0006le-Jv for pgsql-bugs@lists.postgresql.org; Sat, 06 Jun 2020 17:12:42 +0000 Received: from courriel-2.mines-paristech.fr ([77.158.173.19] helo=antispam-1.ensmp.fr) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jhcN4-0000Ta-C5 for pgsql-bugs@lists.postgresql.org; Sat, 06 Jun 2020 17:12:41 +0000 Received: from smtp-5.mines-paristech.fr (smtp.mines-paristech.fr [77.158.173.146]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by antispam-1.ensmp.fr (antispam.mines-paristech.fr) with ESMTPS id 68FF65614E; Sat, 6 Jun 2020 19:12:36 +0200 (CEST) Received: from pseudo (159.237.219.88.rev.sfr.net [88.219.237.159]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp-5.mines-paristech.fr (Postfix) with ESMTPSA id 6036520280; Sat, 6 Jun 2020 19:12:35 +0200 (CEST) Date: Sat, 6 Jun 2020 19:12:33 +0200 (CEST) From: Fabien COELHO X-X-Sender: fabien@pseudo To: Tom Lane cc: Andres Freund , PostgreSQL Bugs , David Rowley , "Jawarilal, Manish" Subject: Re: pgbench bug / limitation In-Reply-To: <1673150.1591455433@sss.pgh.pa.us> Message-ID: References: <384920.1591027137@sss.pgh.pa.us> <512846.1591057287@sss.pgh.pa.us> <1507639.1591375547@sss.pgh.pa.us> <1509305.1591378355@sss.pgh.pa.us> <20200605174418.zbb4egfe5wxhv2yd@alap3.anarazel.de> <1510961.1591380652@sss.pgh.pa.us> <1673150.1591455433@sss.pgh.pa.us> User-Agent: Alpine 2.22 (DEB 394 2020-01-19) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="2056645051-291096523-1591463556=:32228" X-CLAMAV-SCAN: ok X-VRSPAM-SCORE: -100 X-VRSPAM-STATE: legit X-VRSPAM-CAUSE: gggruggvucftvghtrhhoucdtuddrgeduhedrudeghedguddugecutefuodetggdotefrucfrrhhofhhilhgvmecuggftfghnshhusghstghrihgsvgenuceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmnecujfgurhepfffhvffujgfkfhgfgggtsehmtderredtreelnecuhfhrohhmpefhrggsihgvnhcuvefqgffnjffquceotghovghlhhhosegtrhhirdgvnhhsmhhprdhfrheqnecuggftrfgrthhtvghrnhepffdvheekveegtefhjeduheelkeefueeujeejfeejhffggfehteekkeetveeileelnecukfhppeekkedrvdduledrvdefjedrudehleenucfrrghrrghmpehmohguvgepshhmthhpohhuth X-VRSPAM-EXTCAUSE: mhhouggvpehsmhhtphhouhht List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --2056645051-291096523-1591463556=:32228 Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 8BIT Hello Tom, My 0.02 €, which might not convince you: >> ISTM that the patch your propose, a full reimplementation of the abstract >> poll/select fd set interface for windows, is overkill because it adds >> about 80 lines of code. ISTM that just redefining some macros is enough to >> get the same performance benefit without adding much to the code base, but >> maybe I'm missing something. At least, I do not think that getting rid of >> the FD_SETSIZE redefinition for windows is worth all that trouble. > > I think it's worth doing because it puts the Windows implementation at > full feature parity with Unixen, ie, there's no hard-wired limit on the > number of connections. That is a point, although I do not find it desirable: In most pgbench settings ISTM that one thread should handle a few clients, which means waiting at most for a few handles, so that performance are not limited by pgbench itself. > I'm also not that thrilled with trying to make one code path work with > two fundamentally different implementations. Yep, but with the same interface. > We've found out that just because Microsoft claims their implementation > is compatible with POSIX doesn't make it so, Sure! > and I'd rather not have warts in our source code from trying to paper > over that. Hmmm. > Lastly, I think it's entirely likely that someone will come up with a > Windows-native implementation at some point, so that we'd have the extra > code lines anyway. When reading WaitForMultipleObjects doc, the suggestion seems to create more threads to do the waiting, by chunks of 64, and then you wait for these threads, possibly recursively. In pgbench context, it would make much more sense to advise the user to create more threads for benchmarking from the beginning, instead of having one thread for benchmarking and possibly creating more just to wait for events, so I cannot see any windows implementation that would make sense and would be worth having, at least based on this interface. So it seems to me that we would be better off with a minimal patch to work around the FD_SET check, not bother with the FD_SET overhead, not fiddle with FD_SETSIZE at all (which given the underlying nČ handling seems like a bad idea anyway), and advise the user to create more threads when the limit is reached, as the attached version does. I cannot foresee a setting where over 60 clients per thread (ISTM there is at most one wait per client) would make much sense, so let's not help the user run ineffective benchmarks. Maybe we should even issue a warning when #client/#thread is high. Even if there is a bench where a thread could really handle 1000 clients (implying 1000 pg processes server side), probably the user could suffer to run that bench with 16 threads. -- Fabien. --2056645051-291096523-1591463556=:32228 Content-Type: text/x-diff; name=pgbench-select-win-4.patch Content-Transfer-Encoding: BASE64 Content-ID: Content-Description: Content-Disposition: attachment; filename=pgbench-select-win-4.patch ZGlmZiAtLWdpdCBhL3NyYy9iaW4vcGdiZW5jaC9wZ2JlbmNoLmMgYi9zcmMv YmluL3BnYmVuY2gvcGdiZW5jaC5jDQppbmRleCAwOGE1OTQ3YTllLi4yY2Q2 OThjMDI3IDEwMDY0NA0KLS0tIGEvc3JjL2Jpbi9wZ2JlbmNoL3BnYmVuY2gu Yw0KKysrIGIvc3JjL2Jpbi9wZ2JlbmNoL3BnYmVuY2guYw0KQEAgLTI3LDEw ICsyNyw2IEBADQogICoNCiAgKi8NCiANCi0jaWZkZWYgV0lOMzINCi0jZGVm aW5lIEZEX1NFVFNJWkUgMTAyNAkJCS8qIG11c3Qgc2V0IGJlZm9yZSB3aW5z b2NrMi5oIGlzIGluY2x1ZGVkICovDQotI2VuZGlmDQotDQogI2luY2x1ZGUg InBvc3RncmVzX2ZlLmgiDQogDQogI2luY2x1ZGUgPGN0eXBlLmg+DQpAQCAt NjY5OCwxNiArNjY5NCwzMCBAQCBjbGVhcl9zb2NrZXRfc2V0KHNvY2tldF9z ZXQgKnNhKQ0KIAlzYS0+bWF4ZmQgPSAtMTsNCiB9DQogDQorI2lmbmRlZiBX SU4zMg0KKyNkZWZpbmUgRkRfSVNfVkFMSUQoZmQsIHNldCkgKChmZCkgPj0g MCAmJiAoZmQpIDwgRkRfU0VUU0laRSkNCisjZWxzZQ0KKy8qDQorICogV2lu ZG93cyBpbXBsZW1lbnRhdGlvbiBpcyBib2d1czogaXQgZG9lcyBub3QgYWxs b2NhdGUgZmQgd2l0aCB0aGUgbG93ZXN0DQorICogcG9zc2libGUgbnVtYmVy LCBhbmQgRkRfU0VUIGlzIGltcGxlbWVudGVkIGFzIGEgc2ltcGxlIGFycmF5 Lg0KKyAqDQorICogRmlsbGluZyB0aGUgc2V0IGlzIGluIE8oTl4yKSBiZWNh dXNlIG9mIGRlZHVwbGljYXRpb24uDQorICoNCisgKiBGRF9TRVRTSVpFIGlz IGRlZmluZWQgdG8gNjQgYnkgZGVmYXVsdCBvbiB3aW5kb3dzLg0KKyAqLw0K KyNkZWZpbmUgRkRfSVNfVkFMSUQoZmQsIHNldCkgKChmZCkgPj0gMCAmJiAo c2V0KS0+ZmRfY291bnQgPCBGRF9TRVRTSVpFKQ0KKyNlbmRpZiAvKiBXSU4z MiAqLw0KKw0KIHN0YXRpYyB2b2lkDQogYWRkX3NvY2tldF90b19zZXQoc29j a2V0X3NldCAqc2EsIGludCBmZCwgaW50IGlkeCkNCiB7DQotCWlmIChmZCA8 IDAgfHwgZmQgPj0gRkRfU0VUU0laRSkNCisJaWYgKCFGRF9JU19WQUxJRChm ZCwgJnNhLT5mZHMpKQ0KIAl7DQogCQkvKg0KIAkJICogRG9pbmcgYSBoYXJk IGV4aXQgaGVyZSBpcyBhIGJpdCBncm90dHksIGJ1dCBpdCBkb2Vzbid0IHNl ZW0gd29ydGgNCiAJCSAqIGNvbXBsaWNhdGluZyB0aGUgQVBJIHRvIG1ha2Ug aXQgbGVzcyBncm90dHkuDQogCQkgKi8NCi0JCXBnX2xvZ19mYXRhbCgidG9v IG1hbnkgY2xpZW50IGNvbm5lY3Rpb25zIGZvciBzZWxlY3QoKSIpOw0KKwkJ cGdfbG9nX2ZhdGFsKCJ0b28gbWFueSBjbGllbnQgY29ubmVjdGlvbnMgZm9y IHNlbGVjdCgpLCBjb25zaWRlciB1c2luZyBtb3JlIHRocmVhZHMiKTsNCiAJ CWV4aXQoMSk7DQogCX0NCiAJRkRfU0VUKGZkLCAmc2EtPmZkcyk7DQo= --2056645051-291096523-1591463556=:32228--