Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id D0826B5DBE8 for ; Mon, 23 May 2011 02:23:01 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 47780-08 for ; Mon, 23 May 2011 05:22:55 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from www.postnewspapers.com.au (www.postnewspapers.com.au [173.230.139.50]) by mail.postgresql.org (Postfix) with ESMTP id 0D8ABB5DBE3 for ; Mon, 23 May 2011 02:22:54 -0300 (ADT) Received: from localhost (localhost [127.0.0.1]) by www.postnewspapers.com.au (Postfix) with ESMTP id 60D222D; Mon, 23 May 2011 01:22:53 -0400 (EDT) X-Virus-Scanned: amavisd-new at postnewspapers.com.au.com Received: from www.postnewspapers.com.au ([127.0.0.1]) by localhost (www.postnewspapers.com.au [127.0.0.1]) (amavisd-new, port 10024) with LMTP id wUgqN0-Aak-8; Mon, 23 May 2011 01:22:53 -0400 (EDT) Received: from mail.postnewspapers.com.au (ppp171-128.static.internode.on.net [150.101.171.128]) by www.postnewspapers.com.au (Postfix) with ESMTP id ADAB327; Mon, 23 May 2011 01:22:52 -0400 (EDT) Received: from [10.0.0.53] (wallace.localnet [10.0.0.53]) by mail.postnewspapers.com.au (Postfix) with ESMTP id 69419AF9EE; Mon, 23 May 2011 13:22:51 +0800 (WST) Message-ID: <4DD9EF2B.9030104@postnewspapers.com.au> Date: Mon, 23 May 2011 13:22:51 +0800 From: Craig Ringer User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.15) Gecko/20110307 Thunderbird/3.1.9 MIME-Version: 1.0 To: Aren Cambre CC: pgsql-performance@postgresql.org Subject: Re: Postgres refusing to use >1 core References: In-Reply-To: X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.91 tagged_above=-10 required=5 tests=BAYES_00=-1.9, T_RP_MATCHES_RCVD=-0.01 X-Spam-Level: X-Archive-Number: 201105/295 X-Sequence-Number: 43804 On 23/05/11 12:09, Aren Cambre wrote: > Also, thanks for the advice on batching my queries. I am now using a > very efficient bulk data read and write methods for Postgres. > > My program bulk reads 100,000 rows, processes those rows (during which > it does a few SELECTs), and then writes 100,000 rows at a time. > > It cycles through this until it has processed all 12,000,000 rows. > > This, plus the parallelism fix, will probably convert this 30 hour > program to a <2 hour program. It's always good to hear when these things work out. Thanks for reporting back. Using the set-based nature of relational databases to your advantage, writing smarter queries that do more work server-side with fewer round-trips, and effective batching can make a huge difference. -- Craig Ringer