Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes
To: =?utf-8?Q?Torsten_Z=C3=BChlsdorff?= <foo@meisterderspiele.de>,
	pgsql-performance@postgresql.org
Subject: Re: How to insert a bulk of data with unique-violations
 very fast
References: <hu37gm$qrl$1@news.eternal-september.org>
	<AANLkTimwLi6PkFYltlyXecsDQaI6yD3y22WLmWdgwe_R@mail.gmail.com>
	<4C07E418.80103@meisterderspiele.de>
	<AANLkTilj2xHAZMnQ-To3lod3mMPw1DBU_8EkRwTC29qb@mail.gmail.com>
	<hug2od$nu4$1@news.eternal-september.org>
	<AANLkTim36aTPinMgAQ9bUcf0AD4S1YKRrXfApHERRd7U@mail.gmail.com>
	<op.vdwg04j0eorkce@apollo13> <huiroa$sn6$1@news.eternal-september.org>
	<op.vdxvs5l9eorkce@apollo13> <hungra$vlt$2@news.eternal-september.org>
Date: Wed, 09 Jun 2010 12:51:08 +0200
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
From: "Pierre C" <lists@peufeu.com>
Message-ID: <op.vd04fifueorkce@apollo13>
In-Reply-To: <hungra$vlt$2@news.eternal-september.org>
User-Agent: Opera Mail/10.10 (Linux)

>>> Within the data to import most rows have 20 till 50 duplicates.  
>>> Sometime much more, sometimes less.
>>  In that case (source data has lots of redundancy), after importing the  
>> data chunks in parallel, you can run a first pass of de-duplication on  
>> the chunks, also in parallel, something like :
>>  CREATE TEMP TABLE foo_1_dedup AS SELECT DISTINCT * FROM foo_1;
>>  or you could compute some aggregates, counts, etc. Same as before, no  
>> WAL needed, and you can use all your cores in parallel.
>>   From what you say this should reduce the size of your imported data  
>> by a lot (and hence the time spent in the non-parallel operation).
>
> Thank you very much for this advice. I've tried it inanother project  
> with similar import-problems. This really speed the import up.

Glad it was useful ;)