To: pgsql-sql@postgresql.org
From: Harald Fuchs <hf517@protecting.net>
Subject: Re: postgresql multiple insert slow
Date: 06 Jun 2004 13:39:25 +0200
Organization: Linux Private Site
Lines: 43
Message-ID: <pun03g29w2.fsf@srv.protecting.net>
References: <20040219163238.GD10913@mich2.itxmarket.com>
Reply-To: hf517@protecting.net
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
User-Agent: Gnus/5.0808 (Gnus v5.8.8) Emacs/20.7

In article <20040219163238.GD10913@mich2.itxmarket.com>,
"Michael L. Hostbaek" <mich@freebsdcluster.org> writes:

> Hello, 
> I've got a table in an oracle database with approx. 100000 records, that
> I'd like to put into a table in a postgresql database. (This should be
> done a couple of times per week)

> I have written a short perl script, on a server that has remote access
> to both the oracle database as well as the postgresql database. I am
> running postgresql 7.4.1 on FreeBSD.

> My perl script looks something like this:

> [...]
> my $sth2 = $cnx2->prepare('SELECT * FROM oracle_table');
> my $res2 = $sth2->execute();

> while(my($field2,$field5,$field6) = ($sth2->fetchrow_array)) {
> 	if(defined($field2)) {
> 		my $sth = $cnx->prepare('INSERT INTO
> the_pg_table(field1, field2) VALUES(?,?)');
> 		my $result = $sth->execute($field2,$field5);
> 		$sth->finish;

> 	}
> }
> [...]

> I runs fine - and I get no errors - but it takes almost 25 minutes to
> complete.. I tried running the script while just grabbing the rows from
> the oracle database and writing to a text file - and then it only takes
> a couple of minutes .. So it must be the INSERT command that chokes - is
> there a better way to do it ? 

First of all, you should prepare the insert statement only once,
outside of the loop.  Then you could use fetchrow_arrarref instead of
fetchrow_array; this should eliminate a copy operation.

But the biggest win would be not to use INSERT at all.  Instruct
Oracle to dump the rows into a CSV file, and then do just

$cnx->do ("COPY the_pg_table FROM 'csv.file'")