Received: from maia.hub.org (maia-2.hub.org [200.46.204.251])
	by mail.postgresql.org (Postfix) with ESMTP id B5748BAE0B9
	for <pgsql-performance-postgresql.org@mail.postgresql.org>;
	Thu, 17 Nov 2011 13:55:40 -0400 (AST)
Received: from mail.postgresql.org ([200.46.204.86])
	by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia,
	port 10024) with ESMTP id 44163-04
	for <pgsql-performance-postgresql.org@mail.postgresql.org>;
	Thu, 17 Nov 2011 17:55:34 +0000 (UTC)
Received: from magus.postgresql.org (magus.postgresql.org [87.238.57.229])
	by mail.postgresql.org (Postfix) with ESMTP id BACD9BAE0B8
	for <pgsql-performance@postgresql.org>;
	Thu, 17 Nov 2011 13:55:34 -0400 (AST)
Received: from sss.pgh.pa.us ([66.207.139.130])
	by magus.postgresql.org with esmtp (Exim 4.72)
	(envelope-from <tgl@sss.pgh.pa.us>) id 1RR6BR-0006L0-P8
	for pgsql-performance@postgresql.org; Thu, 17 Nov 2011 17:55:34 +0000
Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1])
	by sss.pgh.pa.us (8.14.2/8.14.2) with ESMTP id pAHHtKVo027244;
	Thu, 17 Nov 2011 12:55:20 -0500 (EST)
To: Jon Nelson <jnelson+pgsql@jamponi.net>
cc: pgsql-performance@postgresql.org
Subject: Re: external sort performance 
In-reply-to: 
 <CAKuK5J3taH85P6zc1j63mkwiNXH2nfJ5ER6R1+Vk5iG_Cq-cFw@mail.gmail.com>
References: 
 <CAKuK5J3taH85P6zc1j63mkwiNXH2nfJ5ER6R1+Vk5iG_Cq-cFw@mail.gmail.com>
Comments: In-reply-to Jon Nelson <jnelson+pgsql@jamponi.net>
	message dated "Thu, 17 Nov 2011 11:10:56 -0600"
Date: Thu, 17 Nov 2011 12:55:20 -0500
Message-ID: <27243.1321552520@sss.pgh.pa.us>
From: Tom Lane <tgl@sss.pgh.pa.us>
X-Virus-Scanned: Maia Mailguard 1.0.1
X-Spam-Status: No, hits=-1.9 tagged_above=-10 required=5 tests=BAYES_00=-1.9
X-Spam-Level: 
X-Archive-Number: 201111/223
X-Sequence-Number: 45734

Jon Nelson <jnelson+pgsql@jamponi.net> writes:
> This is on PG 8.4.8 on Linux, 16GB of "real" RAM.
> Most recently, I enabled trace_sort, disabled hash aggregation[1], and
> set a large work_mem (normally very small, in this case I tried
> anything from 8MB to 256MB. I even tried 1GB and 2GB).

FWIW, I think hash aggregation is your best shot at getting reasonable
performance.  Sorting 175GB of data is going to hurt no matter what.

If the grouped table amounts to 5GB, I wouldn't have expected the hash
table to be more than maybe 2-3X that size (although this does depend on
what aggregates you're running...).  Letting the hash aggregation have
all your RAM might be the best answer.

			regards, tom lane