Date: Wed, 8 Sep 2010 12:56:54 -0700
From: David Kerr <dmk@mr-paradox.net>
To: Greg Smith <greg@2ndquadrant.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, pgsql-performance@postgresql.org
Subject: Re: pgbench could not send data to client: Broken pipe
Message-ID: <20100908195654.GB49941@mr-paradox.net>
References: <20100908174629.GD40339@mr-paradox.net>
	<8037.1283973026@sss.pgh.pa.us> <4C87E3A6.8060800@2ndquadrant.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <4C87E3A6.8060800@2ndquadrant.com>
User-Agent: Mutt/1.4.2.3i

On Wed, Sep 08, 2010 at 03:27:34PM -0400, Greg Smith wrote:
- Tom Lane wrote:
- >As of the 9.0 release, it's possible to run pgbench in a "multi thread"
- >mode, and if you forced the subprocess rather than thread model it looks
- >like the select() limit would be per subprocess rather than global.
- >So I think you could get above the FD_SETSIZE limit with a bit of
- >hacking if you were using 9.0's pgbench.  No chance with 8.3 though.
- >  
- 
- I believe David can do this easily enough by compiling a 9.0 source code 
- tree with the "--disable-thread-safety" option.  That's the simplest way 
- to force the pgbench client to build itself using the multi-process 
- model, rather than the multi-threaded one.
- 
- It's kind of futile to run pgbench simulating much more than a hundred 
- or two clients before 9.0 anyway.  Without multiple workers, you're 
- likely to just run into the process switching limitations within pgbench 
- itself rather than testing server performance usefully.  I've watched 
- the older pgbench program fail to come close to saturating an 8 core 
- server without running into its own limitations first.
- 
- You might run a 9.0 pgbench client against an 8.3 server though, if you 
- did the whole thing starting from pgbench database initialization over 
- again--the built-in tables like "accounts" changed to "pgbench_accounts" 
- in 8.4.  That might work, can't recall any changes that would prevent 
- it; but as I haven't tested it yet I can't say for sure.

Thanks, I compiled the 9.0 RC1 branch with the --disable-thread-safety option
and ran PG bench on my 8.3 DB it seemed to work fine, 

However, MAXCLIENTS is still 1024, if i hack it to switch it up to 2048 i 
get this:
starting vacuum...end.
select failed: Bad file descriptor  <---------------
transaction type: TPC-B (sort of)
scaling factor: 1
query mode: simple
number of clients: 1900
number of threads: 1
number of transactions per client: 10
number of transactions actually processed: 3723/19000
tps = 52.007642 (including connections establishing)
tps = 82.579077 (excluding connections establishing)


I'm not sure what Tom is referring to with the select(2) limitation, maybe I'm running
into it (where do i find that? /usr/include/sys/select.h? )

should i be running pgbench differently? I tried increasing the # of threads
but that didn't increase the number of backend's and i'm trying to simulate
2000 physical backend processes.

thanks

Dave