pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Fabien COELHO <coelho@cri.ensmp.fr>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Andres Freund <andres@anarazel.de>
Cc: PostgreSQL Bugs <pgsql-bugs@lists.postgresql.org>
Cc: David Rowley <dgrowleyml@gmail.com>
Cc: Jawarilal, Manish <Manish.Jawarilal@dell.com>
Subject: Re: pgbench bug / limitation
Date: Sat, 6 Jun 2020 19:12:33 +0200 (CEST)
Message-ID: <alpine.DEB.2.22.394.2006061819450.32228@pseudo> (raw)
In-Reply-To: <1673150.1591455433@sss.pgh.pa.us>
References: <BL0PR1901MB198583EAC6C8A3B770D20F1AED8A0@BL0PR1901MB1985.namprd19.prod.outlook.com>
	<384920.1591027137@sss.pgh.pa.us>
	<CAApHDvrjWtJa9oQ98oH+9PCvTgOkGZP+gcmottNX95sjn6o9fQ@mail.gmail.com>
	<512846.1591057287@sss.pgh.pa.us>
	<CAApHDvpMEAaZMxtzixbnb7ySo=Kb1eUfshEB-y4YOm8SgGrpiw@mail.gmail.com>
	<alpine.DEB.2.22.394.2006021258170.87735@pseudo>
	<CAApHDvpDBkjj6a+Ld-9YGW=_Bcxss8Yyz_QYjtjus3WUsTb+1g@mail.gmail.com>
	<1507639.1591375547@sss.pgh.pa.us>
	<D973F4AF-8949-42C1-ABF1-E87F7C63E746@anarazel.de>
	<1509305.1591378355@sss.pgh.pa.us>
	<20200605174418.zbb4egfe5wxhv2yd@alap3.anarazel.de>
	<1510961.1591380652@sss.pgh.pa.us>
	<alpine.DEB.2.22.394.2006060851480.32228@pseudo>
	<1673150.1591455433@sss.pgh.pa.us>


Hello Tom,

My 0.02 €, which might not convince you:

>> ISTM that the patch your propose, a full reimplementation of the abstract
>> poll/select fd set interface for windows, is overkill because it adds
>> about 80 lines of code. ISTM that just redefining some macros is enough to
>> get the same performance benefit without adding much to the code base, but
>> maybe I'm missing something. At least, I do not think that getting rid of
>> the FD_SETSIZE redefinition for windows is worth all that trouble.
>
> I think it's worth doing because it puts the Windows implementation at
> full feature parity with Unixen, ie, there's no hard-wired limit on the
> number of connections.

That is a point, although I do not find it desirable: In most pgbench 
settings ISTM that one thread should handle a few clients, which means 
waiting at most for a few handles, so that performance are not limited by 
pgbench itself.

> I'm also not that thrilled with trying to make one code path work with 
> two fundamentally different implementations.

Yep, but with the same interface.

> We've found out that just because Microsoft claims their implementation
> is compatible with POSIX doesn't make it so,

Sure!

> and I'd rather not have warts in our source code from trying to paper 
> over that.

Hmmm.

> Lastly, I think it's entirely likely that someone will come up with a
> Windows-native implementation at some point, so that we'd have the extra
> code lines anyway.

When reading WaitForMultipleObjects doc, the suggestion seems to create 
more threads to do the waiting, by chunks of 64, and then you wait for 
these threads, possibly recursively. In pgbench context, it would make 
much more sense to advise the user to create more threads for benchmarking 
from the beginning, instead of having one thread for benchmarking and 
possibly creating more just to wait for events, so I cannot see any 
windows implementation that would make sense and would be worth having, at 
least based on this interface.

So it seems to me that we would be better off with a minimal patch to work 
around the FD_SET check, not bother with the FD_SET overhead, not fiddle 
with FD_SETSIZE at all (which given the underlying n² handling seems like 
a bad idea anyway), and advise the user to create more threads when the 
limit is reached, as the attached version does.

I cannot foresee a setting where over 60 clients per thread (ISTM there is 
at most one wait per client) would make much sense, so let's not help the 
user run ineffective benchmarks. Maybe we should even issue a warning when 
#client/#thread is high. Even if there is a bench where a thread could 
really handle 1000 clients (implying 1000 pg processes server side), 
probably the user could suffer to run that bench with 16 threads.

-- 
Fabien.

Attachments:

pgbench-select-win-4.patch (text/x-diff, 1.4K) download diff --git a/src/bin/pgbench/pgbench.c b/src/bin/pgbench/pgbench.c index 08a5947a9e..2cd698c027 100644 --- a/src/bin/pgbench/pgbench.c +++ b/src/bin/pgbench/pgbench.c @@ -27,10 +27,6 @@ * */ -#ifdef WIN32 -#define FD_SETSIZE 1024 /* must set before winsock2.h is included */ -#endif - #include "postgres_fe.h" #include <ctype.h> @@ -6698,16 +6694,30 @@ clear_socket_set(socket_set *sa) sa->maxfd = -1; } +#ifndef WIN32 +#define FD_IS_VALID(fd, set) ((fd) >= 0 && (fd) < FD_SETSIZE) +#else +/* + * Windows implementation is bogus: it does not allocate fd with the lowest + * possible number, and FD_SET is implemented as a simple array. + * + * Filling the set is in O(N^2) because of deduplication. + * + * FD_SETSIZE is defined to 64 by default on windows. + */ +#define FD_IS_VALID(fd, set) ((fd) >= 0 && (set)->fd_count < FD_SETSIZE) +#endif /* WIN32 */ + static void add_socket_to_set(socket_set *sa, int fd, int idx) { - if (fd < 0 || fd >= FD_SETSIZE) + if (!FD_IS_VALID(fd, &sa->fds)) { /* * Doing a hard exit here is a bit grotty, but it doesn't seem worth * complicating the API to make it less grotty. */ - pg_log_fatal("too many client connections for select()"); + pg_log_fatal("too many client connections for select(), consider using more threads"); exit(1); } FD_SET(fd, &sa->fds);

view thread (26+ messages)  latest in thread

Message-ID: <alpine.DEB.2.22.394.2006061819450.32228@pseudo>
Permalink:  ../alpine.DEB.2.22.394.2006061819450.32228@pseudo/
Also on:    postgresql.org/message-id/alpine.DEB.2.22.394.2006061819450.32228@pseudo

 ·  · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-bugs@postgresql.org
  Cc: coelho@cri.ensmp.fr, tgl@sss.pgh.pa.us, andres@anarazel.de, pgsql-bugs@lists.postgresql.org, dgrowleyml@gmail.com, Manish.Jawarilal@dell.com
  Subject: Re: pgbench bug / limitation
  In-Reply-To: <alpine.DEB.2.22.394.2006061819450.32228@pseudo>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox