From: Jim C. Nasby <jim@nasby.net>
To: Craig A. James <cjames@modgraph-usa.com>
To: g@nasby.net
Cc: Worky Workerson <worky.workerson@gmail.com>
Cc: Merlin Moncure <mmoncure@gmail.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: Best COPY Performance
Date: Tue, 24 Oct 2006 19:14:49 -0500
Message-ID: <20061025001448.GZ26892@nasby.net> (raw)
In-Reply-To: <453D443B.20302@modgraph-usa.com>
References: <ce4072df0610201305u7836dc7bn437ec0aaf1c3e25f@mail.gmail.com>
<b42b73150610201811m84369f7kc7d58b39aed3ad73@mail.gmail.com>
<ce4072df0610230810t5d66bbb8xf1e9c0ed71b6b6c0@mail.gmail.com>
<20061023215913.GW26892@nasby.net>
<453D443B.20302@modgraph-usa.com>
On Mon, Oct 23, 2006 at 03:37:47PM -0700, Craig A. James wrote:
> Jim C. Nasby wrote:
> >http://stats.distributed.net used to use a perl script to do some
> >transformations before loading data into the database. IIRC, when we
> >switched to using C we saw 100x improvement in speed, so I suspect that
> >if you want performance perl isn't the way to go. I think you can
> >compile perl into C, so maybe that would help some.
>
> I use Perl extensively, and have never seen a performance problem. I
> suspect the perl-to-C "100x improvement" was due to some other factor, like
> a slight change in the schema, indexes, or the fundamental way the client
> (C vs Perl) handled the data during the transformation, or just plain bad
> Perl code.
>
> Modern scripting languages like Perl and Python make programmers far, far
> more productive than the bad old days of C/C++. Don't shoot yourself in
> the foot by reverting to low-level languages like C/C++ until you've
> exhausted all other possibilities. I only use C/C++ for intricate
> scientific algorithms.
>
> In many cases, Perl is *faster* than C/C++ code that I write, because I
> can't take the time (for example) to write the high-performance string
> manipulation that have been fine-tuned and extensively optimized in Perl.
Well, the code is all at
http://cvs.distributed.net/viewcvs.cgi/stats-proc/hourly/ (see logmod
directory and logmod_*.pl). There have been changes made to the C code
since we changed over, but you can find the appropriate older versions
in there. IIRC, nothing in the database changed when we went from perl
to C (it's likely that was the *only* change that happened anywhere
around that time).
--
Jim Nasby jim@nasby.net
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: jim@nasby.net, cjames@modgraph-usa.com, g@nasby.net, worky.workerson@gmail.com, mmoncure@gmail.com
Subject: Re: Best COPY Performance
In-Reply-To: <20061025001448.GZ26892@nasby.net>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox