From: Dimitri Fontaine <dfontaine@hi-media.com>
Organization: Hi-Media
To: Simon Riggs <simon@2ndquadrant.com>
Subject: Re: [PERFORM] multi-threaded pgloader makes it in version 2.3.0
Date: Mon, 10 Mar 2008 18:57:38 +0100
User-Agent: KMail/1.9.7
Cc: pgsql-general@postgresql.org
References: <200802261308.53459.dfontaine@hi-media.com>
	<200803101718.16866.dfontaine@hi-media.com>
	<1205169263.4269.169.camel@ebony.site>
In-Reply-To: <1205169263.4269.169.camel@ebony.site>
MIME-Version: 1.0
Content-Type: multipart/signed; boundary="nextPart1571186.5ZVD1chTkH";
	protocol="application/pgp-signature"; micalg=pgp-sha1
Content-Transfer-Encoding: 7bit
Message-Id: <200803101857.38489.dfontaine@hi-media.com>

--nextPart1571186.5ZVD1chTkH
Content-Type: text/plain;
  charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Le lundi 10 mars 2008, Simon Riggs a =C3=A9crit=C2=A0:
> Not sure when or why I would want an rrqueue_size larger than
> copy_every, and less sounds very strange. Can we get away with it being
> the same thing in all cases?

In fact, that's just that you asked for a reader which reads one line at a=
=20
time and feed the workers in a round robin fashion, and I wanted to feed th=
em=20
more than 1 line at a time, hence this parameter. Of course it could well b=
e=20
it's not needed, and I'll then deprecate it in next version.
Please note it defaults to what you want it to be, so you can just forget=20
about it...

I'm beginning to think you asked 1 line at a time for the first version to =
be=20
easier to implement... :)

> Do you have some basic performance numbers? It would be good to
> understand the overhead of the parallelism on a large file with 1, 2 and
> 4 threads. Would be good to see if synchronous_commit =3D off helped speed
> things up as well.

Didn't have the time to test this performance wise, that's why I asked for=
=20
testing last time. I've planned some perf tests if only to have the=20
opportunity to write up some presentation article, but didn't find the time=
=20
to run them yet.

> Presumably -V and -T still work when we go parallel, but just issue one
> query?

Still work, of course, the 'controller' thread will issue them before to=20
parallelize the work or begin to read the input file. Rejecting still works=
=20
the same too, threads share a reject object which is protected by a lock=20
(mutex), so the file don't get mixed line.=20
I've tried not to compromise any existing feature by adding the parallel on=
es,=20
and didn't have to at the end of it.

Regards,
=2D-=20
dim

--nextPart1571186.5ZVD1chTkH
Content-Type: application/pgp-signature; name=signature.asc 
Content-Description: This is a digitally signed message part.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQBH1XaSlBXRlnbh1bkRAk74AKC0G0KWzBPOueHRC/PqY6FLBX0fuACfXkz+
5T8oYKiJxmkI1DnWpeh3Wfg=
=xQTs
-----END PGP SIGNATURE-----

--nextPart1571186.5ZVD1chTkH--