Date: Wed, 22 Mar 2006 10:09:25 -0600
From: "Jim C. Nasby" <jnasby@pervasive.com>
To: ashah <ashah@opuspro.com>
Cc: pgsql <pgsql-performance@postgresql.org>
Subject: Re: Massive Inserts Strategies
Message-ID: <20060322160924.GJ15742@pervasive.com>
References: <200603221032.10207.ashah@opuspro.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <200603221032.10207.ashah@opuspro.com>
User-Agent: Mutt/1.5.11

Load the files into a temp table and go from there...

COPY ... FROM file;
UPDATE existing_table SET ... WHERE ...;
INSERT INTO existing_table SELECT * FROM temp_table WHERE NOT EXISTS(
SELECT * FROM existing_table WHERE ...)

On Wed, Mar 22, 2006 at 10:32:10AM -0500, ashah wrote:
> I have a database with foreign keys enabled on the schema. I receive different 
> files, some of them are huge. And I need to load these files in the database 
> every night. There are several scenerios that I want to design an optimal 
> solution for -
> 
> 1. One of the file has around 80K records and I have to delete everything from 
> the table and load this file. The provider never provides a "delta file" so I 
> dont have a way to identify which records are already present and which are 
> new. If I dont delete everything and insert fresh, I have to make around 80K 
> selects to decide if the records exist or not. Now there are lot of tables 
> that have foreign keys linked with this table so unless I disable the foreign 
> keys, I cannot really delete anything from this table. What would be a good  
> practise here?
> 
> 2. Another file that I receive has around 150K records that I need to load in 
> the database. Now one of the fields is logically a "foreign key" to another 
> table, and it is linked to the parent table via a database generated unique 
> ID instead of the actual value. But the file comes with the actual value. So 
> once again, I have to either drop the foreign key, or make 150K selects to 
> determine the serial ID so that the foreign key is satisfied. What would be a 
> good strategy in this scenerio ?
> 
> Please pardon my inexperience with database !
> 
> Thanks,
> Amit
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Have you searched our list archives?
> 
>                http://archives.postgresql.org
> 

-- 
Jim C. Nasby, Sr. Engineering Consultant      jnasby@pervasive.com
Pervasive Software      http://pervasive.com    work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf       cell: 512-569-9461