DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com;
	h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
	b=n63djGuU4zYKmZ63DK1BuGhUN++EA+JLMHAGymHEayscD6ySEXmKZI2eNDppdlFVS+dNheWbnDb23DS+0DNDwEpq2BPQAr35rkNGpHZxILCyxenr8oQUTPxcogvzzYridFaOkUcIAEGY4pE0GrYZVkkUd9Dzr3/zqBHQBR2j8XE=
Message-ID: <36e682920604181310u740ef76hbc5bd5b63ed75cd2@mail.gmail.com>
Date: Tue, 18 Apr 2006 13:10:09 -0700
From: "Jonah H. Harris" <jonah.harris@gmail.com>
To: "Jim C. Nasby" <jnasby@pervasive.com>
Subject: Re: Google SoC--Idea Request
Cc: "Mark Wong" <markw@osdl.org>,
	"Pgsql Hackers" <pgsql-hackers@postgresql.org>
In-Reply-To: <20060418191149.GD49405@pervasive.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
References: <36e682920604151205i75a08030x7c05a6f52fccf095@mail.gmail.com>
	<20060417205455.GP49405@pervasive.com> <44452F9C.8000401@osdl.org>
	<20060418191149.GD49405@pervasive.com>

On 4/18/06, Jim C. Nasby <jnasby@pervasive.com> wrote:
> On Tue, Apr 18, 2006 at 11:27:40AM -0700, Mark Wong wrote:
> > Jim C. Nasby wrote:
> > >On Sat, Apr 15, 2006 at 03:05:20PM -0400, Jonah H. Harris wrote:
> > >>All ideas welcome!
> > >
> > >I know it's not directly PostgreSQL related, but I'd love to see the
> > >dbt* code improved. Items on my wish-list:
> > >
> > >- make it easy to run the test framework and clients on a seperate
> > >  machine from the database server
> > >- keep results in a database
> > >- provide a front-end to allow users to schedule tests in a queue
> > >- add support for windows, at least for the database (theoretically
> > >  possible to run that way now, but you have to do everything by hand)
> > >
> > >Another idea: afaik, spikesource is still offering a bounty for
> > >improvements to OSS test suites, something that'd fit well with SoC.
> >
> > I second this. :)  There are also the TPC-App (Java) fair-use
> > implementation that I've started and the TPC-E (next gen OLTP) that I
> > would like to start.
>
> Maybe before starting on TPC-E it makes sense to try and get a common
> framework for all the different tests built? AFAIK most of the
> benchmarks all use a fairly standard client-server infrastructure, so we
> should hopefully be able to share that between the different tests...

I agree with Jim.  A framework would really help out here.  All of the
tests are basically the same and would benefit from a framework.

However, Mark, do you think Java is a reliable benchmarking platform?=20
At EnterpriseDB, we've tried several Java benchmarks and could never
get as repeatable or reliable of a benchmark as DBT2 gives you.

--
Jonah H. Harris, Database Internals Architect
EnterpriseDB Corporation
732.331.1324