Received: from localhost (unknown [200.46.204.184]) by postgresql.org (Postfix) with ESMTP id 3E1422E4136 for ; Thu, 7 Feb 2008 13:15:54 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.184]) (amavisd-maia, port 10024) with ESMTP id 58689-03 for ; Thu, 7 Feb 2008 13:15:46 -0400 (AST) Received: from is.rice.edu (is.rice.edu [128.42.42.24]) by postgresql.org (Postfix) with ESMTP id 8FC0A2E43FA for ; Thu, 7 Feb 2008 13:15:47 -0400 (AST) Received: from scan1.mail.rice.edu (scan1.mail.rice.edu [128.42.58.160]) by is.rice.edu (Postfix) with ESMTP id 1658541825; Thu, 7 Feb 2008 11:15:46 -0600 (CST) Received: from is.rice.edu ([128.42.42.24]) by scan1.mail.rice.edu (scan1.mail.rice.edu [128.42.58.160]) (amavisd-new, port 10024) with ESMTP id 23330-08; Thu, 7 Feb 2008 11:15:45 -0600 (CST) Received: by is.rice.edu (Postfix, from userid 18612) id 520D541820; Thu, 7 Feb 2008 11:15:45 -0600 (CST) Date: Thu, 7 Feb 2008 11:15:44 -0600 From: Kenneth Marshall To: Greg Smith Cc: pgsql-performance@postgresql.org Subject: Re: Benchmark Data requested --- pgloader CE design ideas Message-ID: <20080207171544.GU4201@it.is.rice.edu> References: <1202150236.4252.521.camel@ebony.site> <200802062059.07174.dfontaine@hi-media.com> <200802071031.50121.dfontaine@hi-media.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.16 (2007-06-09) X-Virus-Scanned: by amavis-2.2.1 at scan1.mail.rice.edu X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200802/86 X-Sequence-Number: 28628 On Thu, Feb 07, 2008 at 12:06:42PM -0500, Greg Smith wrote: > On Thu, 7 Feb 2008, Dimitri Fontaine wrote: > >> I was thinking of not even reading the file content from the controller >> thread, just decide splitting points in bytes (0..ST_SIZE/4 - >> ST_SIZE/4+1..2*ST_SIZE/4 etc) and let the reading thread fine-tune by >> beginning to process input after having read first newline, etc. > > The problem I was pointing out is that if chunk#2 moved foward a few bytes > before it started reading in search of a newline, how will chunk#1 know > that it's supposed to read up to that further point? You have to stop #1 > from reading further when it catches up with where #2 started. Since the > start of #2 is fuzzy until some reading is done, what you're describing > will need #2 to send some feedback to #1 after they've both started, and > that sounds bad to me. I like designs where the boundaries between threads > are clearly defined before any of them start and none of them ever talk to > the others. > As long as both processes understand the start condition, there is not a problem. p1 starts at beginning and processes through chunk2 offset until it reaches the start condition. p2 starts loading from chunk2 offset plus the amount needed to reach the start condition, ... DBfile|---------------|--x--------------|x----------------|-x--| x chunk1-----------> x chunk2--------> x chunk3----------->... As long as both pieces use the same test, they will each process non-overlapping segments of the file and still process 100% of the file. Ken >> In both cases, maybe it would also be needed for pgloader to be able to >> have a >> separate thread for COPYing the buffer to the server, allowing it to >> continue >> preparing next buffer in the meantime? > > That sounds like a V2.0 design to me. I'd only chase after that level of > complexity if profiling suggests that's where the bottleneck really is. > > -- > * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD > > ---------------------------(end of broadcast)--------------------------- > TIP 6: explain analyze is your friend >