MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8;
 format=flowed
Content-Transfer-Encoding: 7bit
Date: Thu, 08 Nov 2012 12:55:12 +0100
From: Oliver Seidel <postgresql@os10000.net>
To: <pgsql-performance@postgresql.org>
Subject: parallel query evaluation
Reply-To: <postgresql@os10000.net>
Mail-Reply-To: <postgresql@os10000.net>
Message-ID: <214a89820818a3da44bcadb4ab829c14@os10000.net>
User-Agent: Webmail/0.6

Hi,

I have

             create table x ( att bigint, val bigint, hash varchar(30) 
);

with 693million rows.  The query

             create table y as select att, val, count(*) as cnt from x 
group by att, val;

ran for more than 2000 minutes and used 14g memory on an 8g physical 
RAM machine -- eventually I stopped it.  Doing

             create table y ( att bigint, val bigint, cnt int );
             and something a bit like: for i in `seq 0 255` | xargs -n 1 
-P 6
                         psql -c "insert into y select att, val, 
count(*) from x where att%256=$1 group by att, val" test

runs 6 out of 256 in 10 minutes -- meaning the whole problem can be 
done in just under 3 hours.

Question 1: do you see any reason why the second method would yield a 
different result from the first method?
Question 2: is that method generalisabl so that it could be included in 
the base system without manual shell glue?

Thanks,

Oliver