pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Pierre C <lists@peufeu.com>
To: Torsten Zühlsdorff <foo@meisterderspiele.de>
To: pgsql-performance@postgresql.org
Subject: Re: How to insert a bulk of data with unique-violations very fast
Date: Mon, 07 Jun 2010 18:52:07 +0200
Message-ID: <op.vdxvs5l9eorkce@apollo13> (raw)
In-Reply-To: <huiroa$sn6$1@news.eternal-september.org>
References: <hu37gm$qrl$1@news.eternal-september.org>
	<AANLkTimwLi6PkFYltlyXecsDQaI6yD3y22WLmWdgwe_R@mail.gmail.com>
	<4C07E418.80103@meisterderspiele.de>
	<AANLkTilj2xHAZMnQ-To3lod3mMPw1DBU_8EkRwTC29qb@mail.gmail.com>
	<hug2od$nu4$1@news.eternal-september.org>
	<AANLkTim36aTPinMgAQ9bUcf0AD4S1YKRrXfApHERRd7U@mail.gmail.com>
	<op.vdwg04j0eorkce@apollo13>
	<huiroa$sn6$1@news.eternal-september.org>


> Within the data to import most rows have 20 till 50 duplicates. Sometime  
> much more, sometimes less.

In that case (source data has lots of redundancy), after importing the  
data chunks in parallel, you can run a first pass of de-duplication on the  
chunks, also in parallel, something like :

CREATE TEMP TABLE foo_1_dedup AS SELECT DISTINCT * FROM foo_1;

or you could compute some aggregates, counts, etc. Same as before, no WAL  
needed, and you can use all your cores in parallel.

 From what you say this should reduce the size of your imported data by a  
lot (and hence the time spent in the non-parallel operation).

With a different distribution, ie duplicates only between existing and  
imported data, and not within the imported data, this strategy would be  
useless.




view thread (13+ messages)  latest in thread

Message-ID: <op.vdxvs5l9eorkce@apollo13>
Permalink:  ../op.vdxvs5l9eorkce@apollo13/
Also on:    postgresql.org/message-id/op.vdxvs5l9eorkce@apollo13

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-performance@postgresql.org
  Cc: lists@peufeu.com, foo@meisterderspiele.de
  Subject: Re: How to insert a bulk of data with unique-violations very fast
  In-Reply-To: <op.vdxvs5l9eorkce@apollo13>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox