Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jhGoa-0006ZE-Fh for pgsql-bugs@arkaria.postgresql.org; Fri, 05 Jun 2020 18:11:36 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1jhGo3-0004XC-Oq for pgsql-bugs@arkaria.postgresql.org; Fri, 05 Jun 2020 18:11:03 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jhGo3-0004X5-IQ for pgsql-bugs@lists.postgresql.org; Fri, 05 Jun 2020 18:11:03 +0000 Received: from sss.pgh.pa.us ([66.207.139.130]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jhGo0-0006kq-I6 for pgsql-bugs@lists.postgresql.org; Fri, 05 Jun 2020 18:11:02 +0000 Received: from sss1.sss.pgh.pa.us (localhost [127.0.0.1]) by sss.pgh.pa.us (8.15.2/8.15.2) with ESMTP id 055IAqBN1510962; Fri, 5 Jun 2020 14:10:52 -0400 From: Tom Lane To: Andres Freund cc: pgsql-bugs@lists.postgresql.org, David Rowley , Fabien COELHO , "Jawarilal, Manish" Subject: Re: pgbench bug / limitation In-reply-to: <20200605174418.zbb4egfe5wxhv2yd@alap3.anarazel.de> References: <384920.1591027137@sss.pgh.pa.us> <512846.1591057287@sss.pgh.pa.us> <1507639.1591375547@sss.pgh.pa.us> <1509305.1591378355@sss.pgh.pa.us> <20200605174418.zbb4egfe5wxhv2yd@alap3.anarazel.de> Comments: In-reply-to Andres Freund message dated "Fri, 05 Jun 2020 10:44:18 -0700" MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----- =_aaaaaaaaaa0" Content-ID: <1510769.1591380267.0@sss.pgh.pa.us> Date: Fri, 05 Jun 2020 14:10:52 -0400 Message-ID: <1510961.1591380652@sss.pgh.pa.us> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk ------- =_aaaaaaaaaa0 Content-Type: text/plain; charset="us-ascii" Content-ID: <1510769.1591380267.1@sss.pgh.pa.us> Andres Freund writes: > I'm fairly sure one can easily scale to large numbers of sockets on > windows by using completion based APIs, but that doesn't seem like a > realistic thing to use for pgbench anytime soon. Agreed. Seems like the best answer is to get in bed with the Windows representation of fd_set, since we cannot avoid knowing that it is different from other platforms' versions. I suggest that we might as well get all the way in and dodge the FD_SETSIZE limitation altogether, as per the attached utterly-untested draft patch. A remaining problem with this is that in theory, repeatedly applying socket_has_input() after the wait could also be O(N^2) (unless FD_ISSET is way smarter than I suspect it is). In practice, that's probably not a huge problem since there would normally be only a few FDs in the result fd_set. Still, I'd rather see us get out from under select() altogether on Windows ... but maybe they *don't* have a better API anywhere. In any case, this should be a good bit better than what we have now. regards, tom lane ------- =_aaaaaaaaaa0 Content-Type: text/x-diff; name="make-pgbench-understand-windows-fd_set-1.patch"; charset="us-ascii" Content-ID: <1510769.1591380267.2@sss.pgh.pa.us> Content-Description: make-pgbench-understand-windows-fd_set-1.patch diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c index 08a5947a9e..e37f833879 100644 --- a/src/bin/pgbench/pgbench.c +++ b/src/bin/pgbench/pgbench.c @@ -27,10 +27,6 @@ * */ -#ifdef WIN32 -#define FD_SETSIZE 1024 /* must set before winsock2.h is included */ -#endif - #include "postgres_fe.h" #include @@ -51,7 +47,11 @@ #include #endif #else /* no ppoll(), so use select() */ +#ifdef WIN32 +#define POLL_USING_WINDOWS_SELECT +#else #define POLL_USING_SELECT +#endif #ifdef HAVE_SYS_SELECT_H #include #endif @@ -108,6 +108,19 @@ typedef struct socket_set #endif /* POLL_USING_SELECT */ +#ifdef POLL_USING_WINDOWS_SELECT +#define SOCKET_WAIT_METHOD "select" + +/*---------- + * On Windows, fd_set is a struct that's effectively + * u_int fd_count; + * SOCKET fd_array[FLEXIBLE_ARRAY_MEMBER]; + *---------- + */ +typedef fd_set socket_set; + +#endif /* POLL_USING_WINDOWS_SELECT */ + /* * Multi-platform pthread implementations */ @@ -6740,6 +6753,70 @@ socket_has_input(socket_set *sa, int fd, int idx) #endif /* POLL_USING_SELECT */ +#ifdef POLL_USING_WINDOWS_SELECT + +static socket_set * +alloc_socket_set(int count) +{ + socket_set *sa; + + sa = (socket_set *) pg_malloc(offsetof(socket_set, fd_array) + + sizeof(SOCKET) * count); + sa->fd_count = 0; + return sa; +} + +static void +free_socket_set(socket_set *sa) +{ + pg_free(sa); +} + +static void +clear_socket_set(socket_set *sa) +{ + FD_ZERO(sa); +} + +static void +add_socket_to_set(socket_set *sa, int fd, int idx) +{ + /* + * We do not use FD_SET() here, first because it enforces a maximum array + * length of FD_SETSIZE which is not relevant, and second because it + * uselessly de-duplicates entries, requiring O(N^2) time to add N FDs to + * the set. + */ + Assert(idx == sa->fd_count); + sa->fd_array[sa->fd_count++] = fd; +} + +static int +wait_on_socket_set(socket_set *sa, int64 usecs) +{ + if (usecs > 0) + { + struct timeval timeout; + + timeout.tv_sec = usecs / 1000000; + timeout.tv_usec = usecs % 1000000; + /* Note that Windows ignores select's first argument */ + return select(0, sa, NULL, NULL, &timeout); + } + else + { + return select(0, sa, NULL, NULL, NULL); + } +} + +static bool +socket_has_input(socket_set *sa, int fd, int idx) +{ + return (FD_ISSET(fd, sa) != 0); +} + +#endif /* POLL_USING_WINDOWS_SELECT */ + /* partial pthread implementation for Windows */ ------- =_aaaaaaaaaa0--