Date: Thu, 28 Apr 2011 23:58:33 -0500 (CDT)
From: Joshua Berkus <josh@agliodbs.com>
To: Greg Smith <greg@2ndquadrant.com>
Cc: pgsql-performance@postgresql.org, Tomas Vondra <tv@fuzzy.cz>
Message-ID: <30006282.226917.1304053113799.JavaMail.root@mail-1.01.com>
In-Reply-To: <4DB890D8.5000604@2ndquadrant.com>
Subject: Re: Performance
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

All,

> The easiest place to start is by re-using the work already done by the
> TPC for benchmarking commercial databases. There are ports of the TPC
> workloads to PostgreSQL available in the DBT-2, DBT-3, and DBT-5
> tests;

Also EAStress, which I think the project still has a license for.

The drawback to these is that they're quite difficult and time-consuming to run, making them unsuitable for doing, say, incremental tuning tests which need to run 100 iterations.  At least, now that we don't have access to the OSDL or Sun labs anymore.  

On the other hand, Greg has made the first steps in a benchmark constructor kit by making it possible for pgBench to run arbitrary workloads.  Someone could build on Greg's foundation by:

a) building a more complex database model with random data generators, and
b) designing a wide series of queries designed to test specific performance problems, i.e, "large object reads", "complex nested subqueries", "mass bulk correllated updates"
c) finally creating scripts which generate benchmarks by choosing a database size and a "mix" of the query menu

This would give us kit which would be capable of testing performance regressions and improvements for PostgreSQL.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
San Francisco