From: Greg Smith <gsmith@gregsmith.com>
To: pgsql-performance@postgresql.org
Subject: Re: Benchmark Data requested
Date: Tue, 5 Feb 2008 22:35:09 -0500 (EST)
Message-ID: <Pine.GSO.4.64.0802052211020.6216@westnet.com> (raw)
In-Reply-To: <1202248803.4252.895.camel@ebony.site>
References: <1202150236.4252.521.camel@ebony.site>
<200802051506.51581.dfontaine@hi-media.com>
<1202221496.4252.680.camel@ebony.site>
<200802051815.25800.dfontaine@hi-media.com>
<47A8C1B9.10200@enterprisedb.com>
<47A8CC1B.1050000@sun.com>
<1202248803.4252.895.camel@ebony.site>
On Tue, 5 Feb 2008, Simon Riggs wrote:
> On Tue, 2008-02-05 at 15:50 -0500, Jignesh K. Shah wrote:
>>
>> Even if it is a single core, the mere fact that the loading process will
>> eventually wait for a read from the input file which cannot be
>> non-blocking, the OS can timeslice it well for the second process to use
>> those wait times for the index population work.
>
> If Dimitri is working on parallel load, why bother?
pgloader is a great tool for a lot of things, particularly if there's any
chance that some of your rows will get rejected. But the way things pass
through the Python/psycopg layer made it uncompetative (more than 50%
slowdown) against the straight COPY path from a rows/second perspective
the last time (V2.1.0?) I did what I thought was a fair test of it (usual
caveat of "with the type of data I was loading"). Maybe there's been some
gigantic improvement since then, but it's hard to beat COPY when you've
got an API layer or two in the middle.
I suspect what will end up happening is that a parallel loading pgloader
will scale something like this:
1 CPU: Considerably slower than COPY
2-3 CPUs: Close to even with COPY
4+ CPUs: Faster than COPY
Maybe I'm wrong, but I wouldn't abandon looking into another approach
until that territory is mapped out a bit better.
Given the very large number of dual-core systems out there now relative to
those with more, optimizing the straight COPY path with any way to take
advantage of even one more core to things like index building is well
worth doing. Heikki's idea sounded good to me regardless, and if that can
be separated out enough to get a second core into the index building at
the same time so much the better.
--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: gsmith@gregsmith.com
Subject: Re: Benchmark Data requested
In-Reply-To: <Pine.GSO.4.64.0802052211020.6216@westnet.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox