User-Agent: Microsoft-Entourage/11.2.0.050811
Date: Thu, 29 Sep 2005 10:06:52 -0700
Subject: Re: [PERFORM] A Better External Sort?
From: "Luke Lonergan" <llonergan@greenplum.com>
To: "Josh Berkus" <josh@agliodbs.com>,
	"Jeffrey W. Baker" <jwbaker@acm.org>
Cc: "Ron Peacetree" <rjpeace@earthlink.net>,
	pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org
Message-ID: <BF616D3C.104C3%llonergan@greenplum.com>
Thread-Topic: [HACKERS] [PERFORM] A Better External Sort?
Thread-Index: AcXFGC+xbkyDzDELEdq3eAANk63kWA==
In-Reply-To: <433C1C2D.7090908@agliodbs.com>
MIME-Version: 1.0
Content-Type: text/plain;
 charset=us-ascii
Content-Transfer-Encoding: 7bit

Josh,

On 9/29/05 9:54 AM, "Josh Berkus" <josh@agliodbs.com> wrote:

> Following an index creation, we see that 95% of the time required is the
> external sort, which averages 2mb/s.  This is with seperate drives for
> the WAL, the pg_tmp, the table and the index.  I've confirmed that
> increasing work_mem beyond a small minimum (around 128mb) had no benefit
> on the overall index creation speed.

Yuuuup!  That about sums it up - regardless of taking 1 or 2 passes through
the heap being sorted, 1.5 - 2 MB/s is the wrong number.  This is not
necessarily an algorithmic problem, but is a optimization problem with
Postgres that must be fixed before it can be competitive.

We read/write to/from disk at 240MB/s and so 2 passes would run at a net
rate of 120MB/s through the sort set if it were that efficient.

Anyone interested in tackling the real performance issue? (flame bait, but
for a worthy cause :-)

- Luke