From: Tom Lane <tgl@sss.pgh.pa.us>
To: Andres Freund <andres@anarazel.de>
Cc: pgsql-bugs@lists.postgresql.org, David Rowley <dgrowleyml@gmail.com>
Cc: Fabien COELHO <coelho@cri.ensmp.fr>
Cc: Jawarilal, Manish <Manish.Jawarilal@dell.com>
Subject: Re: pgbench bug / limitation
Date: Fri, 05 Jun 2020 14:10:52 -0400
Message-ID: <1510961.1591380652@sss.pgh.pa.us> (raw)
In-Reply-To: <20200605174418.zbb4egfe5wxhv2yd@alap3.anarazel.de>
References: <BL0PR1901MB198583EAC6C8A3B770D20F1AED8A0@BL0PR1901MB1985.namprd19.prod.outlook.com>
<384920.1591027137@sss.pgh.pa.us>
<CAApHDvrjWtJa9oQ98oH+9PCvTgOkGZP+gcmottNX95sjn6o9fQ@mail.gmail.com>
<512846.1591057287@sss.pgh.pa.us>
<CAApHDvpMEAaZMxtzixbnb7ySo=Kb1eUfshEB-y4YOm8SgGrpiw@mail.gmail.com>
<alpine.DEB.2.22.394.2006021258170.87735@pseudo>
<CAApHDvpDBkjj6a+Ld-9YGW=_Bcxss8Yyz_QYjtjus3WUsTb+1g@mail.gmail.com>
<1507639.1591375547@sss.pgh.pa.us>
<D973F4AF-8949-42C1-ABF1-E87F7C63E746@anarazel.de>
<1509305.1591378355@sss.pgh.pa.us>
<20200605174418.zbb4egfe5wxhv2yd@alap3.anarazel.de>
Andres Freund <andres@anarazel.de> writes:
> I'm fairly sure one can easily scale to large numbers of sockets on
> windows by using completion based APIs, but that doesn't seem like a
> realistic thing to use for pgbench anytime soon.
Agreed. Seems like the best answer is to get in bed with the Windows
representation of fd_set, since we cannot avoid knowing that it is
different from other platforms' versions. I suggest that we might as
well get all the way in and dodge the FD_SETSIZE limitation altogether,
as per the attached utterly-untested draft patch.
A remaining problem with this is that in theory, repeatedly applying
socket_has_input() after the wait could also be O(N^2) (unless FD_ISSET
is way smarter than I suspect it is). In practice, that's probably not
a huge problem since there would normally be only a few FDs in the
result fd_set. Still, I'd rather see us get out from under select()
altogether on Windows ... but maybe they *don't* have a better API
anywhere. In any case, this should be a good bit better than what we
have now.
regards, tom lane
Attachments:make-pgbench-understand-windows-fd_set-1.patch (text/x-diff, 2.4K) downloaddiff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.cindex 08a5947a9e..e37f833879 100644--- a/src/bin/pgbench/pgbench.c+++ b/src/bin/pgbench/pgbench.c@@ -27,10 +27,6 @@
*
*/
-#ifdef WIN32-#define FD_SETSIZE 1024 /* must set before winsock2.h is included */-#endif-
#include "postgres_fe.h"
#include <ctype.h>
@@ -51,7 +47,11 @@
#include <poll.h>
#endif
#else /* no ppoll(), so use select() */
+#ifdef WIN32+#define POLL_USING_WINDOWS_SELECT+#else
#define POLL_USING_SELECT
+#endif
#ifdef HAVE_SYS_SELECT_H
#include <sys/select.h>
#endif
@@ -108,6 +108,19 @@ typedef struct socket_set
#endif /* POLL_USING_SELECT */
+#ifdef POLL_USING_WINDOWS_SELECT+#define SOCKET_WAIT_METHOD "select"++/*----------+ * On Windows, fd_set is a struct that's effectively+ * u_int fd_count;+ * SOCKET fd_array[FLEXIBLE_ARRAY_MEMBER];+ *----------+ */+typedef fd_set socket_set;++#endif /* POLL_USING_WINDOWS_SELECT */+
/*
* Multi-platform pthread implementations
*/
@@ -6740,6 +6753,70 @@ socket_has_input(socket_set *sa, int fd, int idx)
#endif /* POLL_USING_SELECT */
+#ifdef POLL_USING_WINDOWS_SELECT++static socket_set *+alloc_socket_set(int count)+{+ socket_set *sa;++ sa = (socket_set *) pg_malloc(offsetof(socket_set, fd_array) ++ sizeof(SOCKET) * count);+ sa->fd_count = 0;+ return sa;+}++static void+free_socket_set(socket_set *sa)+{+ pg_free(sa);+}++static void+clear_socket_set(socket_set *sa)+{+ FD_ZERO(sa);+}++static void+add_socket_to_set(socket_set *sa, int fd, int idx)+{+ /*+ * We do not use FD_SET() here, first because it enforces a maximum array+ * length of FD_SETSIZE which is not relevant, and second because it+ * uselessly de-duplicates entries, requiring O(N^2) time to add N FDs to+ * the set.+ */+ Assert(idx == sa->fd_count);+ sa->fd_array[sa->fd_count++] = fd;+}++static int+wait_on_socket_set(socket_set *sa, int64 usecs)+{+ if (usecs > 0)+ {+ struct timeval timeout;++ timeout.tv_sec = usecs / 1000000;+ timeout.tv_usec = usecs % 1000000;+ /* Note that Windows ignores select's first argument */+ return select(0, sa, NULL, NULL, &timeout);+ }+ else+ {+ return select(0, sa, NULL, NULL, NULL);+ }+}++static bool+socket_has_input(socket_set *sa, int fd, int idx)+{+ return (FD_ISSET(fd, sa) != 0);+}++#endif /* POLL_USING_WINDOWS_SELECT */+
/* partial pthread implementation for Windows */
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: tgl@sss.pgh.pa.us, andres@anarazel.de, dgrowleyml@gmail.com, coelho@cri.ensmp.fr, Manish.Jawarilal@dell.com
Subject: Re: pgbench bug / limitation
In-Reply-To: <1510961.1591380652@sss.pgh.pa.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox