Received: from localhost (unknown [200.46.204.183]) by postgresql.org (Postfix) with ESMTP id B27242E032A for ; Wed, 6 Feb 2008 07:45:12 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 39484-01 for ; Wed, 6 Feb 2008 07:45:03 -0400 (AST) Received: from outmail137094.authsmtp.co.uk (outmail137094.authsmtp.co.uk [62.13.137.94]) by postgresql.org (Postfix) with ESMTP id A223C2E2F95 for ; Wed, 6 Feb 2008 07:45:03 -0400 (AST) Received: from outmail128178.authsmtp.net (outmail128178.authsmtp.net [62.13.128.178]) by punt4.authsmtp.com (8.14.1/8.14.1/Kp) with ESMTP id m16Bhpj3087535; Wed, 6 Feb 2008 11:43:51 GMT Received: from [192.168.0.3] (85-211-31-42.dyn.gotadsl.co.uk [85.211.31.42]) (authenticated bits=0) by mail.authsmtp.com (8.13.8/8.13.8/Kp) with ESMTP id m16Biaq6083386; Wed, 6 Feb 2008 11:44:37 GMT Subject: Re: Benchmark Data requested --- pgloader CE design ideas From: Simon Riggs To: Dimitri Fontaine Cc: pgsql-performance@postgresql.org, "Jignesh K. Shah" , Greg Smith In-Reply-To: <200802061227.58668.dfontaine@hi-media.com> References: <1202150236.4252.521.camel@ebony.site> <1202221496.4252.680.camel@ebony.site> <200802051815.25800.dfontaine@hi-media.com> <200802061227.58668.dfontaine@hi-media.com> Content-Type: text/plain Date: Wed, 06 Feb 2008 11:45:24 +0000 Message-Id: <1202298324.4252.965.camel@ebony.site> Mime-Version: 1.0 X-Mailer: Evolution 2.8.2 Content-Transfer-Encoding: 7bit X-Server-Quench: e4da994c-d4a8-11dc-9554-001185d377ca X-AuthRoute: OCdxZQATClZOTQEd DAteCiNZVAwpPBRK HVkIKg5MJUcNSQVJ NksachtFaANba1xT HGQLW1xEUVx7XWd/ aA0fZQBDYEtPQQZr TkxLXVBXCwUmXB0H dEdPEx9yfwZFcHg+ Z0VgWj4OWBArfE95 EVMAETwOeGZhPDMC WUVdcx5UcAJLdhoR aVd4ByJDAzANdhEA MjUdGAoUCH11JSJe QgQEN1FabU8XF3Ym SgkfEC4zEEFNAmB2 ZwQpLlkVEEsLem4f eVcsQlMXNVcTBgBX BCl0 X-Authentic-SMTP: 61633235383639.squirrel.dmpriest.net.uk:787/Kp X-Report-SPAM: If SPAM / abuse - report it at: http://www.authsmtp.com/abuse X-Virus-Status: No virus detected - but ensure you scan with your own anti-virus system! X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200802/64 X-Sequence-Number: 28606 On Wed, 2008-02-06 at 12:27 +0100, Dimitri Fontaine wrote: > Multi-Threading behavior and CE support > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- > > Now, pgloader will be able to run N threads, each one loading some > data to a > partitionned child-table target. N will certainly be configured > depending on > the number of server cores and not depending on the partition > numbers... > > So what do we do when reading a tuple we want to store in a partition > which > has no dedicated Thread started yet, and we already have N Threads > running? > I'm thinking about some LRU(Thread) to choose a Thread to terminate > (launch > COPY with current buffer and quit) and start a new one for the > current > partition target. > Hopefully there won't be such high values of N that the LRU is a bad > choice > per see, and the input data won't be so messy to have to stop/start > Threads > at each new line. For me, it would be good to see a --parallel=n parameter that would allow pg_loader to distribute rows in "round-robin" manner to "n" different concurrent COPY statements. i.e. a non-routing version. Making that work well, whilst continuing to do error-handling seems like a challenge, but a very useful goal. Adding intelligence to the row distribution may be technically hard but may also simply move the bottleneck onto pg_loader. We may need multiple threads in pg_loader, or we may just need multiple sessions from pg_loader. Experience from doing the non-routing parallel version may help in deciding whether to go for the routing version. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com