From: Greg Smith <gsmith@gregsmith.com>
To: Dimitri Fontaine <dfontaine@hi-media.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: Benchmark Data requested --- pgloader CE design ideas
Date: Wed, 6 Feb 2008 18:36:13 -0500 (EST)
Message-ID: <Pine.GSO.4.64.0802061801540.11463@westnet.com> (raw)
In-Reply-To: <200802062059.07174.dfontaine@hi-media.com>
References: <1202150236.4252.521.camel@ebony.site>
<Pine.GSO.4.64.0802061041230.15780@westnet.com>
<200802061837.41911.dfontaine@hi-media.com>
<200802062059.07174.dfontaine@hi-media.com>
On Wed, 6 Feb 2008, Dimitri Fontaine wrote:
> In fact, the -F option works by having pgloader read the given number of lines
> but skip processing them, which is not at all what Greg is talking about here
> I think.
Yeah, that's not useful.
> Greg, what would you think of a pgloader which will separate file reading
> based on file size as given by stat (os.stat(file)[ST_SIZE]) and number of
> threads: we split into as many pieces as section_threads section config
> value.
Now you're talking. Find a couple of split points that way, fine-tune the
boundaries a bit so they rest on line termination points, and off you go.
Don't forget that the basic principle here implies you'll never know until
you're done just how many lines were really in the file. When thread#1 is
running against chunk#1, it will never have any idea what line chunk#2
really started at until it reaches there, at which point it's done and
that information isn't helpful anymore.
You have to stop thinking in terms of lines for this splitting; all you
can do is split the file into useful byte sections and then count the
lines within them as you go. Anything else requires a counting scan of
the file and such a sequential read is exactly what can't happen
(especially not more than once), it just takes too long.
--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: gsmith@gregsmith.com, dfontaine@hi-media.com
Subject: Re: Benchmark Data requested --- pgloader CE design ideas
In-Reply-To: <Pine.GSO.4.64.0802061801540.11463@westnet.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox