From: David Fetter <david@fetter.org>
To: Luke Lonergan <llonergan@greenplum.com>
Cc: Josh Berkus <josh@agliodbs.com>
Cc: Jeffrey W. Baker <jwbaker@acm.org>
Cc: Ron Peacetree <rjpeace@earthlink.net>
Cc: pgsql-hackers@postgresql.org
Cc: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] A Better External Sort?
Date: Thu, 29 Sep 2005 10:17:57 -0700
Message-ID: <20050929171757.GC7534@fetter.org> (raw)
In-Reply-To: <BF616D3C.104C3%llonergan@greenplum.com>
References: <433C1C2D.7090908@agliodbs.com>
<BF616D3C.104C3%llonergan@greenplum.com>
On Thu, Sep 29, 2005 at 10:06:52AM -0700, Luke Lonergan wrote:
> Josh,
>
> On 9/29/05 9:54 AM, "Josh Berkus" <josh@agliodbs.com> wrote:
>
> > Following an index creation, we see that 95% of the time required
> > is the external sort, which averages 2mb/s. This is with seperate
> > drives for the WAL, the pg_tmp, the table and the index. I've
> > confirmed that increasing work_mem beyond a small minimum (around
> > 128mb) had no benefit on the overall index creation speed.
>
> Yuuuup! That about sums it up - regardless of taking 1 or 2 passes
> through the heap being sorted, 1.5 - 2 MB/s is the wrong number.
> This is not necessarily an algorithmic problem, but is a
> optimization problem with Postgres that must be fixed before it can
> be competitive.
>
> We read/write to/from disk at 240MB/s and so 2 passes would run at a
> net rate of 120MB/s through the sort set if it were that efficient.
>
> Anyone interested in tackling the real performance issue? (flame
> bait, but for a worthy cause :-)
I'm not sure that it's flamebait, but what do I know? Apart from the
nasty number (1.5-2 MB/s), what other observations do you have to
hand? Any ideas about what things are not performing here? Parts of
the code that could bear extra scrutiny? Ideas on how to fix same in
a cross-platform way?
Cheers,
D
--
David Fetter david@fetter.org http://fetter.org/
phone: +1 510 893 6100 mobile: +1 415 235 3778
Remember to vote!
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: david@fetter.org, llonergan@greenplum.com, josh@agliodbs.com, jwbaker@acm.org, rjpeace@earthlink.net, pgsql-hackers@postgresql.org
Subject: Re: [PERFORM] A Better External Sort?
In-Reply-To: <20050929171757.GC7534@fetter.org>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox