From: Greg Smith <gsmith@gregsmith.com>
To: pgsql-performance@postgresql.org
Subject: Re: Benchmark Data requested --- pgloader CE design ideas
Date: Wed, 6 Feb 2008 10:56:03 -0500 (EST)
Message-ID: <Pine.GSO.4.64.0802061041230.15780@westnet.com> (raw)
In-Reply-To: <1202298324.4252.965.camel@ebony.site>
References: <1202150236.4252.521.camel@ebony.site>
<1202221496.4252.680.camel@ebony.site>
<200802051815.25800.dfontaine@hi-media.com>
<200802061227.58668.dfontaine@hi-media.com>
<1202298324.4252.965.camel@ebony.site>
On Wed, 6 Feb 2008, Simon Riggs wrote:
> For me, it would be good to see a --parallel=n parameter that would
> allow pg_loader to distribute rows in "round-robin" manner to "n"
> different concurrent COPY statements. i.e. a non-routing version.
Let me expand on this. In many of these giant COPY situations the
bottleneck is plain old sequential I/O to a single process. You can
almost predict how fast the rows will load using dd. Having a process
that pulls rows in and distributes them round-robin is good, but it won't
crack that bottleneck. The useful approaches I've seen for other
databases all presume that the data files involved are large enough that
on big hardware, you can start multiple processes running at different
points in the file and beat anything possible with a single reader.
If I'm loading a TB file, odds are good I can split that into 4 or more
vertical pieces (say rows 1-25%, 25-50%, 50-75%, 75-100%), start 4 loaders
at once, and get way more than 1 disk worth of throughput reading. You
have to play with the exact number because if you push the split too far
you introduce seek slowdown instead of improvements, but that's the basic
design I'd like to see one day. It's not parallel loading that's useful
for the cases I'm thinking about until something like this comes around.
--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: gsmith@gregsmith.com
Subject: Re: Benchmark Data requested --- pgloader CE design ideas
In-Reply-To: <Pine.GSO.4.64.0802061041230.15780@westnet.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox