From: Torsten Zühlsdorff <foo@meisterderspiele.de>
To: pgsql-performance@postgresql.org
Subject: Re: How to insert a bulk of data with unique-violations very fast
Date: Wed, 09 Jun 2010 09:45:46 +0200
Message-ID: <hungra$vlt$2@news.eternal-september.org> (raw)
In-Reply-To: <op.vdxvs5l9eorkce@apollo13>
References: <hu37gm$qrl$1@news.eternal-september.org>
<AANLkTimwLi6PkFYltlyXecsDQaI6yD3y22WLmWdgwe_R@mail.gmail.com>
<4C07E418.80103@meisterderspiele.de>
<AANLkTilj2xHAZMnQ-To3lod3mMPw1DBU_8EkRwTC29qb@mail.gmail.com>
<hug2od$nu4$1@news.eternal-september.org>
<AANLkTim36aTPinMgAQ9bUcf0AD4S1YKRrXfApHERRd7U@mail.gmail.com>
<op.vdwg04j0eorkce@apollo13>
<huiroa$sn6$1@news.eternal-september.org>
<op.vdxvs5l9eorkce@apollo13>
Pierre C schrieb:
>
>> Within the data to import most rows have 20 till 50 duplicates.
>> Sometime much more, sometimes less.
>
> In that case (source data has lots of redundancy), after importing the
> data chunks in parallel, you can run a first pass of de-duplication on
> the chunks, also in parallel, something like :
>
> CREATE TEMP TABLE foo_1_dedup AS SELECT DISTINCT * FROM foo_1;
>
> or you could compute some aggregates, counts, etc. Same as before, no
> WAL needed, and you can use all your cores in parallel.
>
> From what you say this should reduce the size of your imported data by
> a lot (and hence the time spent in the non-parallel operation).
Thank you very much for this advice. I've tried it inanother project
with similar import-problems. This really speed the import up.
Thank everyone for your time and help!
Greetings,
Torsten
--
http://www.dddbl.de - ein Datenbank-Layer, der die Arbeit mit 8
verschiedenen Datenbanksystemen abstrahiert,
Queries von Applikationen trennt und automatisch die Query-Ergebnisse
auswerten kann.
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: foo@meisterderspiele.de
Subject: Re: How to insert a bulk of data with unique-violations very fast
In-Reply-To: <hungra$vlt$2@news.eternal-september.org>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox