User-Agent: Microsoft-Entourage/11.2.1.051004
Date: Sat, 26 Nov 2005 12:15:28 -0800
Subject: Re: Open request for benchmarking input
From: "Luke Lonergan" <llonergan@greenplum.com>
To: "Qingqing Zhou" <zhouqq@cs.toronto.edu>, pgsql-performance@postgresql.org
Message-ID: <BFAE0260.147E6%llonergan@greenplum.com>
Thread-Topic: [PERFORM] Open request for benchmarking input
Thread-Index: AcXyxiSDYvHeB165EdqEvAANk63kWA==
In-Reply-To: <dmab5n$226n$1@news.hub.org>
MIME-Version: 1.0
Content-Type: text/plain;
 charset=us-ascii
Content-Transfer-Encoding: 7bit

Jeff, Qingqing,

On 11/26/05 10:57 AM, "Qingqing Zhou" <zhouqq@cs.toronto.edu> wrote:

> 
> "Jeff Frost" <jeff@frostconsultingllc.com> wrote
>> 
>> Did you folks see this article on Slashdot with a fellow requesting input
>> on what sort of benchmarks to run to get a good Postgresql vs Mysql
>> dataset? Perhaps this would be a good opportunity for us to get some good
>> benchmarking done.
>>  "The hardware I have available is as follows:
>> 
>>     * 2x dual Opteron 8G ram, 2x144G 15Krpm SCSI
>>     * 2x dual Opteron 8G ram, 2x72G 15Krpm SCSI
>>     * 1x dual Opteron 16G ram, 2x36G 15Krpm SCSI 16x400G 7200rpm SATA
>> 

I suggest specifying a set of basic system / HW benchmarks to baseline the
hardware before each benchmark is run.  This has proven to be a major issue
with most performance tests.  My pick for I/O is bonnie++.

Your equipment allows you the opportunity to benchmark all 5 machines
running together as a cluster - this is important to measure maturity of
solutions for high performance warehousing.  Greenplum can provide you a
license for Bizgres MPP for this purpose.

> (2) The hardware configuration may not reflect all potentials of PostgreSQL.
> For example, so far, PostgreSQL does not pay much attention in reducing I/O
> cost, so a stronger RAID definitely will benefit PostgreSQL performance.

The 16x SATA drives should be great, provided you have a high performance
RAID adapter configured properly.  You should be able to get 800MB/s of
sequential scan performance by using a card like the 3Ware 9550SX.  I've
also heard that the Areca cards are good (how good?).  Configuration of the
I/O must be validated though - I've seen as low as 25MB/s from a
misconfigured system.

>> For my own interests, I would like to at least cover the following bases:
>> 32 bit vs 64 bit vs 64 bit kernel + 32 bit user-space; data warehouse type
>> tests (data >> memory); and web prefs test (active data RAM)
>> 
> 
> Don't forget TPCC (data > memory, with intensive updates). So the benchmarks
> in my mind include TPCC, TPCH and TPCW.

I agree with Qingqing, though I think the OSTG DBT-3 (very similar to TPC-H)
is sufficient for data warehousing.

This is a fairly ambitious project - one problem I see is that MySQL may not
run all of these benchmarks, particularly the DBT-3.  Also - would the rules
allow for mixing / matching pluggable features of the DBMS?  Innodb versus
MyISAM?

- Luke