From: ashah <ashah@opuspro.com>
Organization: opus
To: pgsql <pgsql-performance@postgresql.org>
Subject: Massive Inserts Strategies
Date: Wed, 22 Mar 2006 10:32:10 -0500
User-Agent: KMail/1.8.2
MIME-Version: 1.0
Content-Type: text/plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Message-Id: <200603221032.10207.ashah@opuspro.com>

I have a database with foreign keys enabled on the schema. I receive different 
files, some of them are huge. And I need to load these files in the database 
every night. There are several scenerios that I want to design an optimal 
solution for -

1. One of the file has around 80K records and I have to delete everything from 
the table and load this file. The provider never provides a "delta file" so I 
dont have a way to identify which records are already present and which are 
new. If I dont delete everything and insert fresh, I have to make around 80K 
selects to decide if the records exist or not. Now there are lot of tables 
that have foreign keys linked with this table so unless I disable the foreign 
keys, I cannot really delete anything from this table. What would be a good  
practise here?

2. Another file that I receive has around 150K records that I need to load in 
the database. Now one of the fields is logically a "foreign key" to another 
table, and it is linked to the parent table via a database generated unique 
ID instead of the actual value. But the file comes with the actual value. So 
once again, I have to either drop the foreign key, or make 150K selects to 
determine the serial ID so that the foreign key is satisfied. What would be a 
good strategy in this scenerio ?

Please pardon my inexperience with database !

Thanks,
Amit