From: =?UTF-8?B?VG9yc3RlbiBaw7xobHNkb3JmZg==?= <foo@meisterderspiele.de>
Subject: Re: How to insert a bulk of data with unique-violations
 very fast
Date: Wed, 09 Jun 2010 09:45:46 +0200
Organization: A noiseless patient Spider
Lines: 29
Message-ID: <hungra$vlt$2@news.eternal-september.org>
References: <hu37gm$qrl$1@news.eternal-september.org>
	<AANLkTimwLi6PkFYltlyXecsDQaI6yD3y22WLmWdgwe_R@mail.gmail.com>
	<4C07E418.80103@meisterderspiele.de>
	<AANLkTilj2xHAZMnQ-To3lod3mMPw1DBU_8EkRwTC29qb@mail.gmail.com>
	<hug2od$nu4$1@news.eternal-september.org>
	<AANLkTim36aTPinMgAQ9bUcf0AD4S1YKRrXfApHERRd7U@mail.gmail.com>
	<op.vdwg04j0eorkce@apollo13>
	<huiroa$sn6$1@news.eternal-september.org>
	<op.vdxvs5l9eorkce@apollo13>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Wed, 9 Jun 2010 07:45:46 +0000 (UTC)
Injection-Info: mx03.eternal-september.org;
	posting-host="AnkPUomeHXvuN7UdHvIkog";
	logging-data="32445"; mail-complaints-to="abuse@eternal-september.org";
	posting-account="U2FsdGVkX1/3dpIrOQjrCYhnWap5dKGw"
User-Agent: Mozilla/5.0 (X11; U; Linux i686; de;
	rv:1.8.1.24) Gecko/20100228 Thunderbird/2.0.0.24 Mnenhy/0.7.5.0
In-Reply-To: <op.vdxvs5l9eorkce@apollo13>
Cancel-Lock: sha1:qyiKLDkkw1xrRzJCZ3kvVFuzEyM=
To: pgsql-performance@postgresql.org

Pierre C schrieb:
> 
>> Within the data to import most rows have 20 till 50 duplicates. 
>> Sometime much more, sometimes less.
> 
> In that case (source data has lots of redundancy), after importing the 
> data chunks in parallel, you can run a first pass of de-duplication on 
> the chunks, also in parallel, something like :
> 
> CREATE TEMP TABLE foo_1_dedup AS SELECT DISTINCT * FROM foo_1;
> 
> or you could compute some aggregates, counts, etc. Same as before, no 
> WAL needed, and you can use all your cores in parallel.
> 
>  From what you say this should reduce the size of your imported data by 
> a lot (and hence the time spent in the non-parallel operation).

Thank you very much for this advice. I've tried it inanother project 
with similar import-problems. This really speed the import up.

Thank everyone for your time and help!

Greetings,
Torsten
-- 
http://www.dddbl.de - ein Datenbank-Layer, der die Arbeit mit 8 
verschiedenen Datenbanksystemen abstrahiert,
Queries von Applikationen trennt und automatisch die Query-Ergebnisse 
auswerten kann.