pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Tom Lane <tgl@sss.pgh.pa.us>
To: Andres Freund <andres@anarazel.de>
Cc: pgsql-bugs@lists.postgresql.org, David Rowley <dgrowleyml@gmail.com>
Cc: Fabien COELHO <coelho@cri.ensmp.fr>
Cc: Jawarilal, Manish <Manish.Jawarilal@dell.com>
Subject: Re: pgbench bug / limitation
Date: Fri, 05 Jun 2020 14:10:52 -0400
Message-ID: <1510961.1591380652@sss.pgh.pa.us> (raw)
In-Reply-To: <20200605174418.zbb4egfe5wxhv2yd@alap3.anarazel.de>
References: <BL0PR1901MB198583EAC6C8A3B770D20F1AED8A0@BL0PR1901MB1985.namprd19.prod.outlook.com>
	<384920.1591027137@sss.pgh.pa.us>
	<CAApHDvrjWtJa9oQ98oH+9PCvTgOkGZP+gcmottNX95sjn6o9fQ@mail.gmail.com>
	<512846.1591057287@sss.pgh.pa.us>
	<CAApHDvpMEAaZMxtzixbnb7ySo=Kb1eUfshEB-y4YOm8SgGrpiw@mail.gmail.com>
	<alpine.DEB.2.22.394.2006021258170.87735@pseudo>
	<CAApHDvpDBkjj6a+Ld-9YGW=_Bcxss8Yyz_QYjtjus3WUsTb+1g@mail.gmail.com>
	<1507639.1591375547@sss.pgh.pa.us>
	<D973F4AF-8949-42C1-ABF1-E87F7C63E746@anarazel.de>
	<1509305.1591378355@sss.pgh.pa.us>
	<20200605174418.zbb4egfe5wxhv2yd@alap3.anarazel.de>

Andres Freund <andres@anarazel.de> writes:
> I'm fairly sure one can easily scale to large numbers of sockets on
> windows by using completion based APIs, but that doesn't seem like a
> realistic thing to use for pgbench anytime soon.

Agreed.  Seems like the best answer is to get in bed with the Windows
representation of fd_set, since we cannot avoid knowing that it is
different from other platforms' versions.  I suggest that we might as
well get all the way in and dodge the FD_SETSIZE limitation altogether,
as per the attached utterly-untested draft patch.

A remaining problem with this is that in theory, repeatedly applying
socket_has_input() after the wait could also be O(N^2) (unless FD_ISSET
is way smarter than I suspect it is).  In practice, that's probably not
a huge problem since there would normally be only a few FDs in the
result fd_set.  Still, I'd rather see us get out from under select()
altogether on Windows ... but maybe they *don't* have a better API
anywhere.  In any case, this should be a good bit better than what we
have now.

			regards, tom lane

Attachments:

make-pgbench-understand-windows-fd_set-1.patch (text/x-diff, 2.4K) download diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c index 08a5947a9e..e37f833879 100644 --- a/src/bin/pgbench/pgbench.c +++ b/src/bin/pgbench/pgbench.c @@ -27,10 +27,6 @@ * */ -#ifdef WIN32 -#define FD_SETSIZE 1024 /* must set before winsock2.h is included */ -#endif - #include "postgres_fe.h" #include <ctype.h> @@ -51,7 +47,11 @@ #include <poll.h> #endif #else /* no ppoll(), so use select() */ +#ifdef WIN32 +#define POLL_USING_WINDOWS_SELECT +#else #define POLL_USING_SELECT +#endif #ifdef HAVE_SYS_SELECT_H #include <sys/select.h> #endif @@ -108,6 +108,19 @@ typedef struct socket_set #endif /* POLL_USING_SELECT */ +#ifdef POLL_USING_WINDOWS_SELECT +#define SOCKET_WAIT_METHOD "select" + +/*---------- + * On Windows, fd_set is a struct that's effectively + * u_int fd_count; + * SOCKET fd_array[FLEXIBLE_ARRAY_MEMBER]; + *---------- + */ +typedef fd_set socket_set; + +#endif /* POLL_USING_WINDOWS_SELECT */ + /* * Multi-platform pthread implementations */ @@ -6740,6 +6753,70 @@ socket_has_input(socket_set *sa, int fd, int idx) #endif /* POLL_USING_SELECT */ +#ifdef POLL_USING_WINDOWS_SELECT + +static socket_set * +alloc_socket_set(int count) +{ + socket_set *sa; + + sa = (socket_set *) pg_malloc(offsetof(socket_set, fd_array) + + sizeof(SOCKET) * count); + sa->fd_count = 0; + return sa; +} + +static void +free_socket_set(socket_set *sa) +{ + pg_free(sa); +} + +static void +clear_socket_set(socket_set *sa) +{ + FD_ZERO(sa); +} + +static void +add_socket_to_set(socket_set *sa, int fd, int idx) +{ + /* + * We do not use FD_SET() here, first because it enforces a maximum array + * length of FD_SETSIZE which is not relevant, and second because it + * uselessly de-duplicates entries, requiring O(N^2) time to add N FDs to + * the set. + */ + Assert(idx == sa->fd_count); + sa->fd_array[sa->fd_count++] = fd; +} + +static int +wait_on_socket_set(socket_set *sa, int64 usecs) +{ + if (usecs > 0) + { + struct timeval timeout; + + timeout.tv_sec = usecs / 1000000; + timeout.tv_usec = usecs % 1000000; + /* Note that Windows ignores select's first argument */ + return select(0, sa, NULL, NULL, &timeout); + } + else + { + return select(0, sa, NULL, NULL, NULL); + } +} + +static bool +socket_has_input(socket_set *sa, int fd, int idx) +{ + return (FD_ISSET(fd, sa) != 0); +} + +#endif /* POLL_USING_WINDOWS_SELECT */ + /* partial pthread implementation for Windows */

view thread (26+ messages)  latest in thread

Message-ID: <1510961.1591380652@sss.pgh.pa.us>
Permalink:  ../1510961.1591380652@sss.pgh.pa.us/
Also on:    postgresql.org/message-id/1510961.1591380652@sss.pgh.pa.us

 ·  · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-bugs@postgresql.org
  Cc: tgl@sss.pgh.pa.us, andres@anarazel.de, dgrowleyml@gmail.com, coelho@cri.ensmp.fr, Manish.Jawarilal@dell.com
  Subject: Re: pgbench bug / limitation
  In-Reply-To: <1510961.1591380652@sss.pgh.pa.us>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox