Subject: Re: [PERFORM] multi-threaded pgloader makes it in version 2.3.0
From: Simon Riggs <simon@2ndquadrant.com>
To: Dimitri Fontaine <dfontaine@hi-media.com>
Cc: pgsql-general@postgresql.org
In-Reply-To: <200803101718.16866.dfontaine@hi-media.com>
References: <200802261308.53459.dfontaine@hi-media.com>
	<1204395453.8258.117.camel@ebony.site>
	<200803101718.16866.dfontaine@hi-media.com>
Content-Type: text/plain; charset=utf-8
Date: Mon, 10 Mar 2008 17:14:23 +0000
Message-Id: <1205169263.4269.169.camel@ebony.site>
Mime-Version: 1.0
Content-Transfer-Encoding: 8bit

On Mon, 2008-03-10 at 17:18 +0100, Dimitri Fontaine wrote:

> Le samedi 01 mars 2008, Simon Riggs a écrit :
> > On Tue, 2008-02-26 at 13:08 +0100, Dimitri Fontaine wrote:
> > > I'd like to have some feedback about the new version, in term of bugs
> > > encountered and performance limitations (is pgloader up to what you would
> > > expect a multi-threaded loader to be at?)
> >
> > Maybe post to general as well if you don't get any replies here.
> > New feature is very important for us.
> 
> So, here's yet another mail about pgloader new 2.3.0 version, please forgive 
> me for being over zealous here if that's how I appear to be to you...
> 
> Those links will give you detailed information about the new release.
>   http://pgfoundry.org/projects/pgloader  
>   http://pgfoundry.org/forum/forum.php?forum_id=1283
>   http://pgloader.projects.postgresql.org/#_parallel_loading

Sounds good.

Not sure when or why I would want an rrqueue_size larger than
copy_every, and less sounds very strange. Can we get away with it being
the same thing in all cases?

Do you have some basic performance numbers? It would be good to
understand the overhead of the parallelism on a large file with 1, 2 and
4 threads. Would be good to see if synchronous_commit = off helped speed
things up as well.

Presumably -V and -T still work when we go parallel, but just issue one
query?

-- 
  Simon Riggs
  2ndQuadrant  http://www.2ndQuadrant.com 

  PostgreSQL UK 2008 Conference: http://www.postgresql.org.uk