pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Pierre C <lists@peufeu.com>
To: Torsten Zühlsdorff <foo@meisterderspiele.de>
To: pgsql-performance@postgresql.org
Subject: Re: How to insert a bulk of data with unique-violations very fast
Date: Wed, 09 Jun 2010 12:51:08 +0200
Message-ID: <op.vd04fifueorkce@apollo13> (raw)
In-Reply-To: <hungra$vlt$2@news.eternal-september.org>
References: <hu37gm$qrl$1@news.eternal-september.org>
	<AANLkTimwLi6PkFYltlyXecsDQaI6yD3y22WLmWdgwe_R@mail.gmail.com>
	<4C07E418.80103@meisterderspiele.de>
	<AANLkTilj2xHAZMnQ-To3lod3mMPw1DBU_8EkRwTC29qb@mail.gmail.com>
	<hug2od$nu4$1@news.eternal-september.org>
	<AANLkTim36aTPinMgAQ9bUcf0AD4S1YKRrXfApHERRd7U@mail.gmail.com>
	<op.vdwg04j0eorkce@apollo13>
	<huiroa$sn6$1@news.eternal-september.org>
	<op.vdxvs5l9eorkce@apollo13>
	<hungra$vlt$2@news.eternal-september.org>

>>> Within the data to import most rows have 20 till 50 duplicates.  
>>> Sometime much more, sometimes less.
>>  In that case (source data has lots of redundancy), after importing the  
>> data chunks in parallel, you can run a first pass of de-duplication on  
>> the chunks, also in parallel, something like :
>>  CREATE TEMP TABLE foo_1_dedup AS SELECT DISTINCT * FROM foo_1;
>>  or you could compute some aggregates, counts, etc. Same as before, no  
>> WAL needed, and you can use all your cores in parallel.
>>   From what you say this should reduce the size of your imported data  
>> by a lot (and hence the time spent in the non-parallel operation).
>
> Thank you very much for this advice. I've tried it inanother project  
> with similar import-problems. This really speed the import up.

Glad it was useful ;)



view thread (13+ messages)

Message-ID: <op.vd04fifueorkce@apollo13>
Permalink:  ../op.vd04fifueorkce@apollo13/
Also on:    postgresql.org/message-id/op.vd04fifueorkce@apollo13

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-performance@postgresql.org
  Cc: lists@peufeu.com, foo@meisterderspiele.de
  Subject: Re: How to insert a bulk of data with unique-violations very fast
  In-Reply-To: <op.vd04fifueorkce@apollo13>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox