Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id 45E7A1337B9E for ; Thu, 13 Jan 2011 23:19:26 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 50338-03 for ; Fri, 14 Jan 2011 03:19:20 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from hrndva-omtalb.mail.rr.com (hrndva-omtalb.mail.rr.com [71.74.56.124]) by mail.postgresql.org (Postfix) with ESMTP id C48361337B96 for ; Thu, 13 Jan 2011 23:19:18 -0400 (AST) X-Authority-Analysis: v=1.1 cv=dquaJDitHqzHCdqWSoZ6IgapSuTzW/4TaRYx9N9k4W8= c=1 sm=0 a=VTwSuj-UqSoA:10 a=8nJEP1OIZ-IA:10 a=RAii6je2cdnAH6ATCOpc6g==:17 a=1XWaLZrsAAAA:8 a=_Ixf_7qHAAAA:8 a=4slNvdSJSWWl7xjI0IkA:9 a=nIfeTUzX4MYIpTHWNIEA:7 a=Dz3xEk2cZKoW-2WQX47PIXDLYEQA:4 a=wPNLvfGTeEIA:10 a=ryAo9hy1-OkA:10 a=6vsjEBoLwlAA:10 a=Eflo-ka-lPoA:10 a=0tfO7MgyNwYA:10 a=RAii6je2cdnAH6ATCOpc6g==:117 X-Cloudmark-Score: 0 X-Originating-IP: 74.73.87.196 Received: from [74.73.87.196] ([74.73.87.196:53492] helo=[192.168.1.103]) by hrndva-oedge01.mail.rr.com (envelope-from ) (ecelerity 2.2.3.46 r()) with ESMTP id 46/AE-14011-6B0CF2D4; Fri, 14 Jan 2011 03:19:18 +0000 Message-ID: <4D2FC0B7.30705@vmsinfo.com> Date: Thu, 13 Jan 2011 22:19:19 -0500 From: Mladen Gogala User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.13) Gecko/20101207 Lightning/1.0b2 Thunderbird/3.1.7 MIME-Version: 1.0 To: pgsql-performance@postgresql.org Subject: Re: queries with lots of UNIONed relations References: <19302.1294938802@sss.pgh.pa.us> <4879.1294949144@sss.pgh.pa.us> <11563.1294957582@sss.pgh.pa.us> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.9 tagged_above=-10 required=5 tests=BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001 X-Spam-Level: X-Archive-Number: 201101/95 X-Sequence-Number: 41882 On 1/13/2011 5:41 PM, Robert Haas wrote: > You might be right, but I'm not sure. Suppose that there are 100 > inheritance children, and each has 10,000 distinct values, but none of > them are common between the tables. In that situation, de-duplicating > each individual table requires a hash table that can hold 10,000 > entries. But deduplicating everything at once requires a hash table > that can hold 1,000,000 entries. > > Or am I all wet? > Have you considered using Google's map-reduce framework for things like that? Union and group functions look like ideal candidates for such a thing. I am not sure whether map-reduce can be married to a relational database, but I must say that I was impressed with the speed of MongoDB. I am not suggesting that PostgreSQL should sacrifice its ACID compliance for speed, but Mongo sure does look like a speeding bullet. On the other hand, the algorithms that have been paralleled for a long time are precisely sort/merge and hash algorithms used for union and group by functions. This is what I have in mind: http://labs.google.com/papers/mapreduce.html -- Mladen Gogala Sr. Oracle DBA 1500 Broadway New York, NY 10036 (212) 329-5251 www.vmsinfo.com