From: Josh Berkus <josh@agliodbs.com>
To: Jeffrey W. Baker <jwbaker@acm.org>
Cc: Luke Lonergan <llonergan@greenplum.com>
Cc: Ron Peacetree <rjpeace@earthlink.net>
Cc: pgsql-hackers@postgresql.org
Cc: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] A Better External Sort?
Date: Thu, 29 Sep 2005 11:03:27 -0700
Message-ID: <200509291103.27344.josh@agliodbs.com> (raw)
In-Reply-To: <1128015863.11474.9.camel@noodles>
References: <BF616D3C.104C3%llonergan@greenplum.com>
<1128015863.11474.9.camel@noodles>
Jeff,
> Josh, do you happen to know how many passes are needed in the multiphase
> merge on your 60GB table?
No, any idea how to test that?
> I think the largest speedup will be to dump the multiphase merge and
> merge all tapes in one pass, no matter how large M. Currently M is
> capped at 6, so a sort of 60GB with 1GB sort memory needs 13 passes over
> the tape. It could be done in a single pass heap merge with N*log(M)
> comparisons, and, more importantly, far less input and output.
Yes, but the evidence suggests that we're actually not using the whole 1GB
of RAM ... maybe using only 32MB of it which would mean over 200 passes
(I'm not sure of the exact match). Just fixing our algorithm so that it
used all of the work_mem permitted might improve things tremendously.
> I would also recommend using an external processes to asynchronously
> feed the tuples into the heap during the merge.
>
> What's the timeframe for 8.2?
Too far out to tell yet. Probably 9mo to 1 year, that's been our history.
--
--Josh
Josh Berkus
Aglio Database Solutions
San Francisco
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: josh@agliodbs.com, jwbaker@acm.org, llonergan@greenplum.com, rjpeace@earthlink.net, pgsql-hackers@postgresql.org
Subject: Re: [PERFORM] A Better External Sort?
In-Reply-To: <200509291103.27344.josh@agliodbs.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox