From: Mark Mielke <mark@mark.mielke.cc>
To: PFC <lists@peufeu.com>
Cc: Francisco Reyes <lists@stringsutils.com>
Cc: Gregory Stark <stark@enterprisedb.com>
Cc: Pgsql performance <pgsql-performance@postgresql.org>
Subject: Re: Group by more efficient than distinct?
Date: Mon, 21 Apr 2008 21:39:15 -0400
Message-ID: <480D41C3.3070608@mark.mielke.cc> (raw)
In-Reply-To: <480D283E.1060602@mark.mielke.cc>
References: <cone.1208490368.136870.53107.1000@zoraida.natserv.net>
<87ve2f7bel.fsf@oxford.xeocode.com>
<op.t9sycqo6cigqcu@apollo13.peufeu.com>
<cone.1208704536.428347.85914.1000@zoraida.natserv.net>
<op.t9zif2yocigqcu@apollo13.peufeu.com>
<480D283E.1060602@mark.mielke.cc>
Mark Mielke wrote:
> PFC wrote:
>> Actually, the memory used by the hash depends on the number of
>> distinct values, not the number of rows which are processed...
>> Consider :
>>
>> SELECT a GROUP BY a
>> SELECT a,count(*) GROUP BY a
>>
>> In both cases the hash only holds discinct values. So if you have
>> 1 million rows to process but only 10 distinct values of "a", the
>> hash will only contain those 10 values (and the counts), so it will
>> be very small and fast, it will absorb a huge seq scan without
>> problem. If however, you have (say) 100 million distinct values for
>> a, using a hash would be a bad idea. As usual, divide the size of
>> your RAM by the number of concurrent connections or something.
>> Note that "a" could be a column, several columns, anything, the
>> size of the hash will be proportional to the number of distinct
>> values, ie. the number of rows returned by the query, not the number
>> of rows processed (read) by the query. Same with hash joins etc,
>> that's why when you join a very small table to a large one Postgres
>> likes to use seq scan + hash join on the small table.
>
> This surprises me - hash values are lossy, so it must still need to
> confirm against the real list of values, which at a minimum should
> require references to the rows to check against?
>
> Is PostgreSQL doing something beyond my imagination? :-)
Hmmm... You did say distinct values, so I can see how that would work
for distinct. What about seq scan + hash join, though? To complete the
join, wouldn't it need to have a reference to each of the rows to join
against? If there is 20 distinct values and 200 rows in the small table
- wouldn't it need 200 references to be stored?
Cheers,
mark
--
Mark Mielke <mark@mielke.cc>
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: mark@mark.mielke.cc, lists@peufeu.com, lists@stringsutils.com, stark@enterprisedb.com
Subject: Re: Group by more efficient than distinct?
In-Reply-To: <480D41C3.3070608@mark.mielke.cc>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox