pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Mladen Gogala <mladen.gogala@vmsinfo.com>
To: pgsql-performance@postgresql.org
Subject: Re: queries with lots of UNIONed relations
Date: Thu, 13 Jan 2011 22:19:19 -0500
Message-ID: <4D2FC0B7.30705@vmsinfo.com> (raw)
In-Reply-To: <AANLkTikH4eyp5JwJpUEkfOX4u2hOemGDCoXPEf01dkAv@mail.gmail.com>
References: <AANLkTiktjfaR9_sDU+F3UZhAQn7qrBktXE1EiOX=inzQ@mail.gmail.com>
	<19302.1294938802@sss.pgh.pa.us>
	<AANLkTikM9=JXPwYpAbfQ5KZgenWBCn5_J2=W6MRhq0U0@mail.gmail.com>
	<4879.1294949144@sss.pgh.pa.us>
	<AANLkTimx00ADoFDbOyFixLOSy4ypKjhQrHnTzanepbJD@mail.gmail.com>
	<AANLkTikYEX25b4E_AoZk=WWspv2OC7ZKct2VeZb1NCMj@mail.gmail.com>
	<11563.1294957582@sss.pgh.pa.us>
	<AANLkTikH4eyp5JwJpUEkfOX4u2hOemGDCoXPEf01dkAv@mail.gmail.com>

On 1/13/2011 5:41 PM, Robert Haas wrote:
> You might be right, but I'm not sure.  Suppose that there are 100
> inheritance children, and each has 10,000 distinct values, but none of
> them are common between the tables.  In that situation, de-duplicating
> each individual table requires a hash table that can hold 10,000
> entries.  But deduplicating everything at once requires a hash table
> that can hold 1,000,000 entries.
>
> Or am I all wet?
>

Have you considered using Google's map-reduce framework for things like 
that? Union and group functions look like  ideal candidates for such a 
thing.  I am not sure whether map-reduce can be married to a relational 
database, but I must say that I was impressed with the speed of MongoDB. 
I am not suggesting that PostgreSQL should sacrifice its ACID compliance 
for speed, but Mongo sure does look like a speeding bullet.
On the other hand, the algorithms that have been paralleled for a long 
time are precisely sort/merge and hash algorithms used for union and 
group by functions. This is what I have in mind:
http://labs.google.com/papers/mapreduce.html

-- 
Mladen Gogala
Sr. Oracle DBA
1500 Broadway
New York, NY 10036
(212) 329-5251
www.vmsinfo.com




view thread (22+ messages)  latest in thread

Message-ID: <4D2FC0B7.30705@vmsinfo.com>
Permalink:  ../4D2FC0B7.30705@vmsinfo.com/
Also on:    postgresql.org/message-id/4D2FC0B7.30705@vmsinfo.com

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-performance@postgresql.org
  Cc: mladen.gogala@vmsinfo.com
  Subject: Re: queries with lots of UNIONed relations
  In-Reply-To: <4D2FC0B7.30705@vmsinfo.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox