User-Agent: Microsoft-Entourage/11.2.1.051004
Date: Fri, 18 Nov 2005 16:04:00 -0800
Subject: Re: Hardware/OS recommendations for large databases (
From: "Luke Lonergan" <llonergan@greenplum.com>
To: "Mark Kirkwood" <markir@paradise.net.nz>
cc: "Dave Cramer" <pg@fastcrypt.com>, "Greg Stark" <gsstark@mit.edu>,
	"Joshua Marsh" <icub3d@gmail.com>, pgsql-performance@postgresql.org
Message-ID: <BFA3ABF0.14124%llonergan@greenplum.com>
Thread-Topic: [PERFORM] Hardware/OS recommendations for large databases
 (
Thread-Index: AcXsmmQi55EBnDCuTAK/LsJ/SGHiVAAAloQR
In-Reply-To: <437E67EE.4070605@paradise.net.nz>
MIME-Version: 1.0
Content-Type: text/plain;
 charset=us-ascii
Content-Transfer-Encoding: 7bit

Mark,

On 11/18/05 3:46 PM, "Mark Kirkwood" <markir@paradise.net.nz> wrote:
 
> If you alter this to involve more complex joins (e.g 4. way star) and
> (maybe add a small number of concurrent executors too) - is it still the
> case?

4-way star, same result, that's part of my point.  With Bizgres MPP, the
4-way star uses 4 concurrent scanners, though not all are active all the
time.  And that's per segment instance - we normally use one segment
instance per CPU, so our concurrency is NCPUs plus some.

The trick is the "small number of concurrent executors" part.  The only way
to get this with normal postgres is to have concurrent users, and normally
they are doing different things, scanning different parts of the disk.
These are competing things, and for concurrency enhancement something like
"sync scan" would be an effective optimization.

But in reporting, business analytics and warehousing in general, there are
reports that take hours to run.  If you can knock that down by factors of 10
using parallelism, it's a big win.  That's the reason that Teradata did $1.5
Billion in business last year.

More importantly - that's the kind of work that everyone using internet data
for analytics wants right now.

- Luke