From: =?ISO-8859-15?Q?Torsten_Z=FChlsdorff?= <foo@meisterderspiele.de>
Subject: How to insert a bulk of data with unique-violations very
 fast
Date: Tue, 01 Jun 2010 17:03:48 +0200
Organization: A noiseless patient Spider
Lines: 46
Message-ID: <hu37gm$qrl$1@news.eternal-september.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Tue, 1 Jun 2010 15:03:50 +0000 (UTC)
Injection-Info: mx01.eternal-september.org;
	posting-host="AnkPUomeHXvuN7UdHvIkog";
	logging-data="27509"; mail-complaints-to="abuse@eternal-september.org";
	posting-account="U2FsdGVkX19Zt1QO2qcN6zXNTTZ2rSiO"
User-Agent: Mozilla/5.0 (X11; U; Linux i686; de;
	rv:1.8.1.24) Gecko/20100228 Thunderbird/2.0.0.24 Mnenhy/0.7.5.0
Cancel-Lock: sha1:OmyuTYxeyYUNpCkXhmm5/R9xdDo=
To: pgsql-performance@postgresql.org

Hello,

i have a set of unique data which about 150.000.000 rows. Regullary i 
get a list of data, which contains multiple times of rows than the 
already stored one. Often around 2.000.000.000 rows. Within this rows 
are many duplicates and often the set of already stored data.
I want to store just every entry, which is not within the already stored 
one. Also i do not want to store duplicates. Example:

Already stored set:
a,b,c

Given set:
a,b,a,c,d,a,c,d,b

Expected set after import:
a,b,c,d

I now looking for a faster way for the import. At the moment i import 
the new data with copy into an table 'import'. then i remove the 
duplicates and insert every row which is not already known. after that 
import is truncated.

Is there a faster way? Should i just insert every row and ignore it, if 
the unique constrain fails?

Here the simplified table-schema. in real life it's with partitions:
test=# \d urls
                          Tabelle »public.urls«
  Spalte |   Typ   |                       Attribute
--------+---------+-------------------------------------------------------
  url_id | integer | not null default nextval('urls_url_id_seq'::regclass)
  url    | text    | not null
Indexe:
     »urls_url« UNIQUE, btree (url)
     »urls_url_id« btree (url_id)

Thanks for every hint or advice! :)

Greetings from Germany,
Torsten
-- 
http://www.dddbl.de - ein Datenbank-Layer, der die Arbeit mit 8 
verschiedenen Datenbanksystemen abstrahiert,
Queries von Applikationen trennt und automatisch die Query-Ergebnisse 
auswerten kann.