Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id 6DD5A1337BA6 for ; Thu, 13 Jan 2011 18:52:18 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 38010-03 for ; Thu, 13 Jan 2011 22:52:11 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from marajade.localdomain (vcs5.camavision.com [63.228.164.252]) by mail.postgresql.org (Postfix) with ESMTP id 3027F1337B96 for ; Thu, 13 Jan 2011 18:52:10 -0400 (AST) Received: from [192.168.10.84] (unknown [192.168.10.84]) (Authenticated sender: andy) by marajade.localdomain (Postfix) with ESMTPA id 4F92343002; Thu, 13 Jan 2011 16:52:08 -0600 (CST) Message-ID: <4D2F821A.1000803@squeakycode.net> Date: Thu, 13 Jan 2011 16:52:10 -0600 From: Andy Colson User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101207 Lightning/1.0b2 Thunderbird/3.1.7 MIME-Version: 1.0 To: Robert Haas CC: Tom Lane , Jon Nelson , pgsql-performance@postgresql.org Subject: Re: queries with lots of UNIONed relations References: <19302.1294938802@sss.pgh.pa.us> <4879.1294949144@sss.pgh.pa.us> <11563.1294957582@sss.pgh.pa.us> <4D2F8103.7010805@squeakycode.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.9 tagged_above=-10 required=5 tests=BAYES_00=-1.9 X-Spam-Level: X-Archive-Number: 201101/89 X-Sequence-Number: 41876 On 1/13/2011 4:49 PM, Robert Haas wrote: > On Thu, Jan 13, 2011 at 5:47 PM, Andy Colson wrote: >>>>> I don't believe there is any case where hashing each individual relation >>>>> is a win compared to hashing them all together. If the optimizer were >>>>> smart enough to be considering the situation as a whole, it would always >>>>> do the latter. >>>> >>>> You might be right, but I'm not sure. Suppose that there are 100 >>>> inheritance children, and each has 10,000 distinct values, but none of >>>> them are common between the tables. In that situation, de-duplicating >>>> each individual table requires a hash table that can hold 10,000 >>>> entries. But deduplicating everything at once requires a hash table >>>> that can hold 1,000,000 entries. >>>> >>>> Or am I all wet? >>> >>> Yeah, I'm all wet, because you'd still have to re-de-duplicate at the >>> end. But then why did the OP get a speedup? *scratches head* >> >> Because it all fix it memory and didnt swap to disk? > > Doesn't make sense. The re-de-duplication at the end should use the > same amount of memory regardless of whether the individual relations > have already been de-duplicated. > Unless I missed something in the thread: distinctList + distinctList + ... -> [fit in mem] -> last distinct -> [fit in mem] vs. fullList + fullList + ... -> [swapped to disk] -> last distinct -> [fit in mem] -Andy