X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org
Received: from localhost (unknown [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 94FFB537F0
	for <pgsql-performance-postgresql.org@localhost.postgresql.org>;
	Fri, 15 Apr 2005 18:25:46 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 17091-10
	for <pgsql-performance-postgresql.org@localhost.postgresql.org>;
	Fri, 15 Apr 2005 21:25:37 +0000 (GMT)
Received: from mail.tecarta.com (66.238.115.135.ptr.us.xo.net
 [66.238.115.135])
	by svr1.postgresql.org (Postfix) with ESMTP id 3F37053B83
	for <pgsql-performance@postgresql.org>;
	Fri, 15 Apr 2005 18:25:36 -0300 (ADT)
Received: from mail pickup service by mail.tecarta.com with Microsoft SMTPSVC;
	Fri, 15 Apr 2005 14:28:41 -0700
Received: from mail.tecarta.com ([192.168.160.2]) by mail.tecarta.com with
	Microsoft SMTPSVC(6.0.3790.0); Fri, 15 Apr 2005 14:28:35 -0700
Received: from barracuda.tecarta.com ([192.168.160.200])
	by mail.tecarta.com (SAVSMTP 3.1.0.29) with SMTP id
	M2005041514283405044
	for <pgsql_performance_postgresql_org@tecarta.com>;
	Fri, 15 Apr 2005 14:28:34 -0700
X-ASG-Debug-ID: 1113600332-10868-0-0
X-Barracuda-URL: http://192.168.160.200:8000/cgi-bin/mark.cgi
Received: from mail1 (mail1.hq.corp [192.168.160.5])
	by barracuda.tecarta.com (Spam Firewall) with SMTP id BD225200D0FD
	for <pgsql_performance_postgresql_org@tecarta.com>;
	Fri, 15 Apr 2005 14:25:32 -0700 (PDT)
Received: from [192.168.0.13] ([63.206.203.145]) by mail.tecarta.com with
	Microsoft SMTPSVC(6.0.3790.0); Fri, 15 Apr 2005 14:28:26 -0700
Message-ID: <42603127.5070108@sfnet.cc>
Date: Fri, 15 Apr 2005 14:24:55 -0700
From: Steve Poe <spoe@sfnet.cc>
User-Agent: Mozilla Thunderbird 1.0 (X11/20041228)
X-Accept-Language: en-us, en
MIME-Version: 1.0
To: "Thomas F.O'Connell" <tfo@sitening.com>
Cc: PgSQL - Performance <pgsql-performance@postgresql.org>
X-ASG-Orig-Subj: Re: [PERFORM] pgbench Comparison of 7.4.7 to 8.0.2
Subject: Re: pgbench Comparison of 7.4.7 to 8.0.2
References: <2784476415a96bb0912ebca41edb3d84@sitening.com>
In-Reply-To: <2784476415a96bb0912ebca41edb3d84@sitening.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
X-OriginalArrivalTime: 15 Apr 2005 21:28:26.0547 (UTC)
	FILETIME=[0F637030:01C54202]
X-Virus-Scanned: by Barracuda Spam Firewall at tecarta.com
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of TAG_LEVEL=4.0
	QUARANTINE_LEVEL=1000.0 KILL_LEVEL=7.0 tests=BAYES_50
X-Barracuda-Spam-Report: Code version 3.02, rules version 3.0.316
	Rule breakdown below pts rule name              description
	---- ----------------------
	--------------------------------------------------
	0.00 BAYES_50 BODY: Bayesian spam probability is 40 to 60%
	[score: 0.5000]
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.724 tagged_above=0 required=5 tests=AWL,
	FORGED_RCVD_HELO, NO_DNS_FOR_FROM
X-Spam-Level: 
X-Archive-Number: 200504/368
X-Sequence-Number: 11765

Tom,

People's opinions on pgbench may vary, so take what I say with a grain 
of salt. Here are my thoughts:

1) Test with no less than 200 transactions per client. I've heard with 
less than this, your results will vary too much with the direction of 
the wind blowing. A high enough value will help rule out some "noise" 
factor. If I am wrong, please let me know.


2) How is the database going to  be used?  What percentage will be 
read/write if you had to guess? Pgbench is like a TPC-B with will help 
guage the potential throughput of your tps. However, it may not stress 
the server enough to help you make key performance changes. However, 
benchmarks are like statistics...full of lies <g>.

3) Run not just a couple pgbench runs, but *many* (I do between 20-40 
runs) so you can rule out noise and guage improvement on median results.

4) Find something that you test OLTP-type transactions. I used OSDB 
since it is simple to implement and use. Although OSDL's OLTP testing 
will closer to reality.

Steve Poe