From: Oliver Seidel <postgresql@os10000.net>
To: pgsql-performance@postgresql.org
Subject: parallel query evaluation
Date: Thu, 08 Nov 2012 12:55:12 +0100
Message-ID: <214a89820818a3da44bcadb4ab829c14@os10000.net> (raw)
Hi,
I have
create table x ( att bigint, val bigint, hash varchar(30)
);
with 693million rows. The query
create table y as select att, val, count(*) as cnt from x
group by att, val;
ran for more than 2000 minutes and used 14g memory on an 8g physical
RAM machine -- eventually I stopped it. Doing
create table y ( att bigint, val bigint, cnt int );
and something a bit like: for i in `seq 0 255` | xargs -n 1
-P 6
psql -c "insert into y select att, val,
count(*) from x where att%256=$1 group by att, val" test
runs 6 out of 256 in 10 minutes -- meaning the whole problem can be
done in just under 3 hours.
Question 1: do you see any reason why the second method would yield a
different result from the first method?
Question 2: is that method generalisabl so that it could be included in
the base system without manual shell glue?
Thanks,
Oliver
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: postgresql@os10000.net
Subject: Re: parallel query evaluation
In-Reply-To: <214a89820818a3da44bcadb4ab829c14@os10000.net>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox