From: Tom Lane <tgl@sss.pgh.pa.us>
To: Gregory Stark <gsstark@mit.edu>
Cc: Alvaro Herrera <alvherre@commandprompt.com>
Cc: Luke Lonergan <llonergan@greenplum.com>
Cc: Chad Wagner <chad.wagner@gmail.com>
Cc: Simon Riggs <simon@2ndquadrant.com>
Cc: Igor Lobanov <ilobanov@swsoft.com>
Cc: Richard Huxton <dev@archonet.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: Querying distinct values from a large table
Date: Wed, 31 Jan 2007 10:15:10 -0500
Message-ID: <25904.1170256510@sss.pgh.pa.us> (raw)
In-Reply-To: <877iv3miqu.fsf@stark.xeocode.com>
References: <C1E49CB9.199A4%llonergan@greenplum.com>
<87d54wmpvd.fsf@stark.xeocode.com>
<20070130170443.GJ2303@alvh.no-ip.org>
<16278.1170222919@sss.pgh.pa.us>
<877iv3miqu.fsf@stark.xeocode.com>
Gregory Stark <gsstark@mit.edu> writes:
> On the other there are some common situations where you could see
> atypical increases. Consider joining a bunch of small tables to
> generate a large result set. The small tables are probably all in
> memory and the result set may only have a small number of distinct
> values. If you throw out the duplicates early you save *all* the
> I/O. If you have to do a disk sort it could be many orders slower.
Right, we already have support for doing that well, in the form of
hashed aggregation. What needs to happen is to get that to work for
DISTINCT as well as GROUP BY. IIRC, DISTINCT is currently rather
thoroughly intertwined with ORDER BY, and we'd have to figure out
some way to decouple them --- without breaking DISTINCT ON, which
makes it a lot harder :-(
regards, tom lane
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: tgl@sss.pgh.pa.us, gsstark@mit.edu, alvherre@commandprompt.com, llonergan@greenplum.com, chad.wagner@gmail.com, simon@2ndquadrant.com, ilobanov@swsoft.com, dev@archonet.com
Subject: Re: Querying distinct values from a large table
In-Reply-To: <25904.1170256510@sss.pgh.pa.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox