pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedmulti-threaded pgloader needs your tests
3+ messages / 2 participants
[nested] [flat]
* multi-threaded pgloader needs your tests
@ 2008-02-26 12:08 Dimitri Fontaine <dfontaine@hi-media.com>
0 siblings, 2 replies; 3+ messages in thread
From: Dimitri Fontaine @ 2008-02-26 12:08 UTC (permalink / raw)
To: pgsql-performance
Hi,
You may remember some thread about data loading performances and
multi-threading support in pgloader:
http://archives.postgresql.org/pgsql-performance/2008-02/msg00081.php
The pgloader code to handle this is now ready to get tested, a more structured
project could talk about a Release Candidate status.
http://pgloader.projects.postgresql.org/dev/TODO.html
http://pgloader.projects.postgresql.org/dev/pgloader.1.html#_parallel_loading
http://packages.debian.org/pgloader --- experimental has the next version
As for the performances benefit of this new version (2.3.0~dev2), all the work
could be reduced to zilch because of the python Global Interpreter Lock,
which I've been aware of late in the development effort.
http://docs.python.org/api/threads.html
This documentation states that (a) using generators you're not that
concerned, and (b) the global lock still allows for IO and processing at the
same time. As pgloader uses generators, I'm still not sure how much a problem
this will be.
I'd like to have some feedback about the new version, in term of bugs
encountered and performance limitations (is pgloader up to what you would
expect a multi-threaded loader to be at?)
Regards,
--
dim
Attachments:
signature.asc (application/pgp-signature, 188B) download
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: multi-threaded pgloader needs your tests
@ 2008-03-01 18:17 Simon Riggs <simon@2ndquadrant.com>
parent: Dimitri Fontaine <dfontaine@hi-media.com>
1 sibling, 0 replies; 3+ messages in thread
From: Simon Riggs @ 2008-03-01 18:17 UTC (permalink / raw)
To: Dimitri Fontaine <dfontaine@hi-media.com>; +Cc: pgsql-performance
On Tue, 2008-02-26 at 13:08 +0100, Dimitri Fontaine wrote:
> I'd like to have some feedback about the new version, in term of bugs
> encountered and performance limitations (is pgloader up to what you would
> expect a multi-threaded loader to be at?)
Maybe post to general as well if you don't get any replies here.
New feature is very important for us.
--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com
PostgreSQL UK 2008 Conference: http://www.postgresql.org.uk
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: multi-threaded pgloader needs your tests
@ 2008-03-10 16:08 Dimitri Fontaine <dfontaine@hi-media.com>
parent: Dimitri Fontaine <dfontaine@hi-media.com>
1 sibling, 0 replies; 3+ messages in thread
From: Dimitri Fontaine @ 2008-03-10 16:08 UTC (permalink / raw)
To: pgsql-performance
Hi,
Le mardi 26 février 2008, Dimitri Fontaine a écrit :
> You may remember some thread about data loading performances and
> multi-threading support in pgloader:
> http://archives.postgresql.org/pgsql-performance/2008-02/msg00081.php
As people here have asked for the new features implemented into pgloader
2.3.0, I'm happy to post here about the availability of the new version!
http://pgfoundry.org/projects/pgloader
http://pgfoundry.org/forum/forum.php?forum_id=1283
Please consider this as a testbed related to the parallel COPY and pg_restore
improvements which have been discussed here and on -hackers, as that's how
those new features came to life.
Regards,
--
dim
Attachments:
signature.asc (application/pgp-signature, 188B) download
^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2008-03-10 16:08 UTC | newest]
Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2008-02-26 12:08 multi-threaded pgloader needs your tests Dimitri Fontaine <dfontaine@hi-media.com>
2008-03-01 18:17 ` Simon Riggs <simon@2ndquadrant.com>
2008-03-10 16:08 ` Dimitri Fontaine <dfontaine@hi-media.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox