pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Dimitri Fontaine <dfontaine@hi-media.com>
To: pgsql-performance@postgresql.org
Subject: multi-threaded pgloader needs your tests
Date: Tue, 26 Feb 2008 13:08:53 +0100
Message-ID: <200802261308.53459.dfontaine@hi-media.com> (raw)

Hi,

You may remember some thread about data loading performances and 
multi-threading support in pgloader:
  http://archives.postgresql.org/pgsql-performance/2008-02/msg00081.php

The pgloader code to handle this is now ready to get tested, a more structured 
project could talk about a Release Candidate status.
 http://pgloader.projects.postgresql.org/dev/TODO.html
 http://pgloader.projects.postgresql.org/dev/pgloader.1.html#_parallel_loading
 http://packages.debian.org/pgloader --- experimental has the next version

As for the performances benefit of this new version (2.3.0~dev2), all the work 
could be reduced to zilch because of the python Global Interpreter Lock, 
which I've been aware of late in the development effort.
  http://docs.python.org/api/threads.html

This documentation states that (a) using generators you're not that 
concerned, and (b) the global lock still allows for IO and processing at the 
same time. As pgloader uses generators, I'm still not sure how much a problem 
this will be.

I'd like to have some feedback about the new version, in term of bugs 
encountered and performance limitations (is pgloader up to what you would 
expect a multi-threaded loader to be at?)

Regards,
-- 
dim

Attachments:

  signature.asc (application/pgp-signature, 188B) download

view thread (3+ messages)  latest in thread

Message-ID: <200802261308.53459.dfontaine@hi-media.com>
Permalink:  ../200802261308.53459.dfontaine@hi-media.com/
Also on:    postgresql.org/message-id/200802261308.53459.dfontaine@hi-media.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-performance@postgresql.org
  Cc: dfontaine@hi-media.com
  Subject: Re: multi-threaded pgloader needs your tests
  In-Reply-To: <200802261308.53459.dfontaine@hi-media.com>
  References: <200802261308.53459.dfontaine@hi-media.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox