From: Dimitri Fontaine <dfontaine@hi-media.com>
To: pgsql-performance@postgresql.org
Subject: multi-threaded pgloader needs your tests
Date: Tue, 26 Feb 2008 13:08:53 +0100
Message-ID: <200802261308.53459.dfontaine@hi-media.com> (raw)
Hi,
You may remember some thread about data loading performances and
multi-threading support in pgloader:
http://archives.postgresql.org/pgsql-performance/2008-02/msg00081.php
The pgloader code to handle this is now ready to get tested, a more structured
project could talk about a Release Candidate status.
http://pgloader.projects.postgresql.org/dev/TODO.htmlhttp://pgloader.projects.postgresql.org/dev/pgloader.1.html#_parallel_loadinghttp://packages.debian.org/pgloader --- experimental has the next version
As for the performances benefit of this new version (2.3.0~dev2), all the work
could be reduced to zilch because of the python Global Interpreter Lock,
which I've been aware of late in the development effort.
http://docs.python.org/api/threads.html
This documentation states that (a) using generators you're not that
concerned, and (b) the global lock still allows for IO and processing at the
same time. As pgloader uses generators, I'm still not sure how much a problem
this will be.
I'd like to have some feedback about the new version, in term of bugs
encountered and performance limitations (is pgloader up to what you would
expect a multi-threaded loader to be at?)
Regards,
--
dim
Attachments:
signature.asc (application/pgp-signature, 188B) download
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-performance@postgresql.org
Cc: dfontaine@hi-media.com
Subject: Re: multi-threaded pgloader needs your tests
In-Reply-To: <200802261308.53459.dfontaine@hi-media.com>
References: <200802261308.53459.dfontaine@hi-media.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox