Received: from localhost (unknown [200.46.204.183]) by postgresql.org (Postfix) with ESMTP id 114D92E0041 for ; Tue, 26 Feb 2008 08:31:35 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 81783-01 for ; Tue, 26 Feb 2008 08:31:31 -0400 (AST) X-Greylist: from auto-whitelisted by SQLgrey-1.7.5 Received: from hi-media.com (9.140-14-84.ripe.coltfrance.com [84.14.140.9]) by postgresql.org (Postfix) with ESMTP id C22CC2E003D for ; Tue, 26 Feb 2008 08:31:31 -0400 (AST) Received: from dim (gut75-1-81-57-27-121.fbx.proxad.net [81.57.27.121]) by hi-media.com (Postfix) with ESMTP id 53E54188072 for ; Tue, 26 Feb 2008 13:05:53 +0100 (CET) From: Dimitri Fontaine Organization: Hi-Media To: pgsql-performance@postgresql.org Subject: multi-threaded pgloader needs your tests Date: Tue, 26 Feb 2008 13:08:53 +0100 User-Agent: KMail/1.9.7 MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart21914495.ApjislzSql"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200802261308.53459.dfontaine@hi-media.com> X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200802/338 X-Sequence-Number: 28880 --nextPart21914495.ApjislzSql Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hi, You may remember some thread about data loading performances and=20 multi-threading support in pgloader: http://archives.postgresql.org/pgsql-performance/2008-02/msg00081.php The pgloader code to handle this is now ready to get tested, a more structu= red=20 project could talk about a Release Candidate status. http://pgloader.projects.postgresql.org/dev/TODO.html http://pgloader.projects.postgresql.org/dev/pgloader.1.html#_parallel_load= ing http://packages.debian.org/pgloader --- experimental has the next version As for the performances benefit of this new version (2.3.0~dev2), all the w= ork=20 could be reduced to zilch because of the python Global Interpreter Lock,=20 which I've been aware of late in the development effort. http://docs.python.org/api/threads.html This documentation states that (a) using generators you're not that=20 concerned, and (b) the global lock still allows for IO and processing at th= e=20 same time. As pgloader uses generators, I'm still not sure how much a probl= em=20 this will be. I'd like to have some feedback about the new version, in term of bugs=20 encountered and performance limitations (is pgloader up to what you would=20 expect a multi-threaded loader to be at?) Regards, =2D-=20 dim --nextPart21914495.ApjislzSql Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQBHxAFVlBXRlnbh1bkRAiVZAJ91bRBP1xXG0BTia+vNk1bi1+RaegCfdiml o9BQ0aNi73NyJnVAHiTVr5Q= =hlhG -----END PGP SIGNATURE----- --nextPart21914495.ApjislzSql--