From: Dimitri Fontaine <dfontaine@hi-media.com>
Organization: Hi-Media
To: pgsql-performance@postgresql.org
Subject: Re: Benchmark Data requested
Date: Wed, 6 Feb 2008 11:29:42 +0100
User-Agent: KMail/1.9.7
Cc: Greg Smith <gsmith@gregsmith.com>
References: <1202150236.4252.521.camel@ebony.site>
	<1202248803.4252.895.camel@ebony.site>
	<Pine.GSO.4.64.0802052211020.6216@westnet.com>
In-Reply-To: <Pine.GSO.4.64.0802052211020.6216@westnet.com>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="nextPart2124380.TmxnqdOtnT";
	protocol="application/pgp-signature"; micalg=pgp-sha1
Content-Transfer-Encoding: 7bit
Message-Id: <200802061129.44482.dfontaine@hi-media.com>

--nextPart2124380.TmxnqdOtnT
Content-Type: text/plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Le mercredi 06 f=E9vrier 2008, Greg Smith a =E9crit=A0:
> pgloader is a great tool for a lot of things, particularly if there's any
> chance that some of your rows will get rejected.  But the way things pass
> through the Python/psycopg layer made it uncompetative (more than 50%
> slowdown) against the straight COPY path from a rows/second perspective
> the last time (V2.1.0?)=20

I've yet to add in the psycopg wrapper Marko wrote for skytools: at the mom=
ent=20
I'm using the psycopg1 interface even when psycopg2 is used, and it seems t=
he=20
new version has some great performance improvements. I just didn't bother=20
until now thinking this wouldn't affect COPY.

> I did what I thought was a fair test of it (usual=20
> caveat of "with the type of data I was loading").  Maybe there's been some
> gigantic improvement since then, but it's hard to beat COPY when you've
> got an API layer or two in the middle.

Did you compare to COPY or \copy? I'd expect psycopg COPY api not to be tha=
t=20
more costly than psql one, after all.
Where pgloader is really left behind (in term of tuples inserted per second=
)=20
compared to COPY is when it has to jiggle a lot with the data, I'd say=20
(reformat, reorder, add constants, etc). But I've tried to design it so tha=
t=20
when not configured to arrange (massage?) the data, the code path is the=20
simplest possible.

Do you want to test pgloader again with Marko psycopgwrapper code to see if=
=20
this helps? If yes I'll arrange to push it to CVS ASAP.

Maybe at the end of this PostgreSQL backend code will be smarter than pgloa=
der=20
(wrt error handling and data massaging) and we'll be able to drop the=20
project, but in the meantime I'll try my best to have pgloader as fast as=20
possible :)
=2D-=20
dim

--nextPart2124380.TmxnqdOtnT
Content-Type: application/pgp-signature; name=signature.asc 
Content-Description: This is a digitally signed message part.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQBHqYwYlBXRlnbh1bkRAuXEAJ4yxQQYNH1yjlB+XxXs0H9VTMx7bwCeM+Yy
U7XkV8Amn5GbATmvHgHDleg=
=ue/P
-----END PGP SIGNATURE-----

--nextPart2124380.TmxnqdOtnT--