pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Andy Colson <andy@squeakycode.net>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Jon Nelson <jnelson+pgsql@jamponi.net>
Cc: pgsql-performance@postgresql.org
Subject: Re: queries with lots of UNIONed relations
Date: Thu, 13 Jan 2011 16:52:10 -0600
Message-ID: <4D2F821A.1000803@squeakycode.net> (raw)
In-Reply-To: <AANLkTikg+DmHbRrxdQbeX3B5PyXD0rRziT0QowQ-_LnM@mail.gmail.com>
References: <AANLkTiktjfaR9_sDU+F3UZhAQn7qrBktXE1EiOX=inzQ@mail.gmail.com>
	<19302.1294938802@sss.pgh.pa.us>
	<AANLkTikM9=JXPwYpAbfQ5KZgenWBCn5_J2=W6MRhq0U0@mail.gmail.com>
	<4879.1294949144@sss.pgh.pa.us>
	<AANLkTimx00ADoFDbOyFixLOSy4ypKjhQrHnTzanepbJD@mail.gmail.com>
	<AANLkTikYEX25b4E_AoZk=WWspv2OC7ZKct2VeZb1NCMj@mail.gmail.com>
	<11563.1294957582@sss.pgh.pa.us>
	<AANLkTikH4eyp5JwJpUEkfOX4u2hOemGDCoXPEf01dkAv@mail.gmail.com>
	<AANLkTinXs1v8zPJiL-di7TZok6CNgOfv-Dan8H+Ot2DN@mail.gmail.com>
	<4D2F8103.7010805@squeakycode.net>
	<AANLkTikg+DmHbRrxdQbeX3B5PyXD0rRziT0QowQ-_LnM@mail.gmail.com>

On 1/13/2011 4:49 PM, Robert Haas wrote:
> On Thu, Jan 13, 2011 at 5:47 PM, Andy Colson<andy@squeakycode.net>  wrote:
>>>>> I don't believe there is any case where hashing each individual relation
>>>>> is a win compared to hashing them all together.  If the optimizer were
>>>>> smart enough to be considering the situation as a whole, it would always
>>>>> do the latter.
>>>>
>>>> You might be right, but I'm not sure.  Suppose that there are 100
>>>> inheritance children, and each has 10,000 distinct values, but none of
>>>> them are common between the tables.  In that situation, de-duplicating
>>>> each individual table requires a hash table that can hold 10,000
>>>> entries.  But deduplicating everything at once requires a hash table
>>>> that can hold 1,000,000 entries.
>>>>
>>>> Or am I all wet?
>>>
>>> Yeah, I'm all wet, because you'd still have to re-de-duplicate at the
>>> end.  But then why did the OP get a speedup?  *scratches head*
>>
>> Because it all fix it memory and didnt swap to disk?
>
> Doesn't make sense.  The re-de-duplication at the end should use the
> same amount of memory regardless of whether the individual relations
> have already been de-duplicated.
>

Unless I missed something in the thread:

distinctList + distinctList + ... -> [fit in mem] -> last distinct -> 
[fit in mem]

vs.

fullList + fullList + ... -> [swapped to disk] -> last distinct -> [fit 
in mem]


-Andy



view thread (22+ messages)  latest in thread

Message-ID: <4D2F821A.1000803@squeakycode.net>
Permalink:  ../4D2F821A.1000803@squeakycode.net/
Also on:    postgresql.org/message-id/4D2F821A.1000803@squeakycode.net

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-performance@postgresql.org
  Cc: andy@squeakycode.net, robertmhaas@gmail.com, tgl@sss.pgh.pa.us, jnelson+pgsql@jamponi.net
  Subject: Re: queries with lots of UNIONed relations
  In-Reply-To: <4D2F821A.1000803@squeakycode.net>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox