Received: from localhost (unknown [200.46.204.183]) by postgresql.org (Postfix) with ESMTP id B7CC12E01D2 for ; Tue, 5 Feb 2008 23:35:14 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 73385-06 for ; Tue, 5 Feb 2008 23:35:12 -0400 (AST) Received: from westnet.com (westnet.com [216.187.52.2]) by postgresql.org (Postfix) with ESMTP id 88B222E019C for ; Tue, 5 Feb 2008 23:35:11 -0400 (AST) Received: from westnet.com (localhost [127.0.0.1]) by westnet.com (8.14.0/8.14.0) with ESMTP id m163Z9QU010103 for ; Tue, 5 Feb 2008 22:35:09 -0500 (EST) Received: from localhost (gsmith@localhost) by westnet.com (8.14.0/8.13.2/Submit) with ESMTP id m163Z9xY010100 for ; Tue, 5 Feb 2008 22:35:09 -0500 (EST) X-Authentication-Warning: westnet.com: gsmith owned process doing -bs Date: Tue, 5 Feb 2008 22:35:09 -0500 (EST) From: Greg Smith X-X-Sender: gsmith@westnet.com To: pgsql-performance@postgresql.org Subject: Re: Benchmark Data requested In-Reply-To: <1202248803.4252.895.camel@ebony.site> Message-ID: References: <1202150236.4252.521.camel@ebony.site> <200802051506.51581.dfontaine@hi-media.com> <1202221496.4252.680.camel@ebony.site> <200802051815.25800.dfontaine@hi-media.com> <47A8C1B9.10200@enterprisedb.com> <47A8CC1B.1050000@sun.com> <1202248803.4252.895.camel@ebony.site> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200802/56 X-Sequence-Number: 28598 On Tue, 5 Feb 2008, Simon Riggs wrote: > On Tue, 2008-02-05 at 15:50 -0500, Jignesh K. Shah wrote: >> >> Even if it is a single core, the mere fact that the loading process will >> eventually wait for a read from the input file which cannot be >> non-blocking, the OS can timeslice it well for the second process to use >> those wait times for the index population work. > > If Dimitri is working on parallel load, why bother? pgloader is a great tool for a lot of things, particularly if there's any chance that some of your rows will get rejected. But the way things pass through the Python/psycopg layer made it uncompetative (more than 50% slowdown) against the straight COPY path from a rows/second perspective the last time (V2.1.0?) I did what I thought was a fair test of it (usual caveat of "with the type of data I was loading"). Maybe there's been some gigantic improvement since then, but it's hard to beat COPY when you've got an API layer or two in the middle. I suspect what will end up happening is that a parallel loading pgloader will scale something like this: 1 CPU: Considerably slower than COPY 2-3 CPUs: Close to even with COPY 4+ CPUs: Faster than COPY Maybe I'm wrong, but I wouldn't abandon looking into another approach until that territory is mapped out a bit better. Given the very large number of dual-core systems out there now relative to those with more, optimizing the straight COPY path with any way to take advantage of even one more core to things like index building is well worth doing. Heikki's idea sounded good to me regardless, and if that can be separated out enough to get a second core into the index building at the same time so much the better. -- * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD