From: Harald Fuchs <hf517@protecting.net>
To: pgsql-sql@postgresql.org
Subject: Re: postgresql multiple insert slow
Date: 06 Jun 2004 13:39:25 +0200
Message-ID: <pun03g29w2.fsf@srv.protecting.net> (raw)
References: <20040219163238.GD10913@mich2.itxmarket.com>
In article <20040219163238.GD10913@mich2.itxmarket.com>,
"Michael L. Hostbaek" <mich@freebsdcluster.org> writes:
> Hello,
> I've got a table in an oracle database with approx. 100000 records, that
> I'd like to put into a table in a postgresql database. (This should be
> done a couple of times per week)
> I have written a short perl script, on a server that has remote access
> to both the oracle database as well as the postgresql database. I am
> running postgresql 7.4.1 on FreeBSD.
> My perl script looks something like this:
> [...]
> my $sth2 = $cnx2->prepare('SELECT * FROM oracle_table');
> my $res2 = $sth2->execute();
> while(my($field2,$field5,$field6) = ($sth2->fetchrow_array)) {
> if(defined($field2)) {
> my $sth = $cnx->prepare('INSERT INTO
> the_pg_table(field1, field2) VALUES(?,?)');
> my $result = $sth->execute($field2,$field5);
> $sth->finish;
> }
> }
> [...]
> I runs fine - and I get no errors - but it takes almost 25 minutes to
> complete.. I tried running the script while just grabbing the rows from
> the oracle database and writing to a text file - and then it only takes
> a couple of minutes .. So it must be the INSERT command that chokes - is
> there a better way to do it ?
First of all, you should prepare the insert statement only once,
outside of the loop. Then you could use fetchrow_arrarref instead of
fetchrow_array; this should eliminate a copy operation.
But the biggest win would be not to use INSERT at all. Instruct
Oracle to dump the rows into a CSV file, and then do just
$cnx->do ("COPY the_pg_table FROM 'csv.file'")
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-sql@postgresql.org
Cc: hf517@protecting.net
Subject: Re: postgresql multiple insert slow
In-Reply-To: <pun03g29w2.fsf@srv.protecting.net>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox