Date: Tue, 24 Oct 2006 19:14:49 -0500
From: "Jim C. Nasby" <jim@nasby.net>
To: "Craig A. James" <cjames@modgraph-usa.com>, g@nasby.net
Cc: Worky Workerson <worky.workerson@gmail.com>,
	Merlin Moncure <mmoncure@gmail.com>, pgsql-performance@postgresql.org
Subject: Re: Best COPY Performance
Message-ID: <20061025001448.GZ26892@nasby.net>
References: <ce4072df0610201305u7836dc7bn437ec0aaf1c3e25f@mail.gmail.com>
	<b42b73150610201811m84369f7kc7d58b39aed3ad73@mail.gmail.com>
	<ce4072df0610230810t5d66bbb8xf1e9c0ed71b6b6c0@mail.gmail.com>
	<20061023215913.GW26892@nasby.net>
	<453D443B.20302@modgraph-usa.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <453D443B.20302@modgraph-usa.com>
User-Agent: Mutt/1.5.13 (2006-08-11)

On Mon, Oct 23, 2006 at 03:37:47PM -0700, Craig A. James wrote:
> Jim C. Nasby wrote:
> >http://stats.distributed.net used to use a perl script to do some
> >transformations before loading data into the database. IIRC, when we
> >switched to using C we saw 100x improvement in speed, so I suspect that
> >if you want performance perl isn't the way to go. I think you can
> >compile perl into C, so maybe that would help some.
> 
> I use Perl extensively, and have never seen a performance problem.  I 
> suspect the perl-to-C "100x improvement" was due to some other factor, like 
> a slight change in the schema, indexes, or the fundamental way the client 
> (C vs Perl) handled the data during the transformation, or just plain bad 
> Perl code.
> 
> Modern scripting languages like Perl and Python make programmers far, far 
> more productive than the bad old days of C/C++.  Don't shoot yourself in 
> the foot by reverting to low-level languages like C/C++ until you've 
> exhausted all other possibilities.  I only use C/C++ for intricate 
> scientific algorithms.
> 
> In many cases, Perl is *faster* than C/C++ code that I write, because I 
> can't take the time (for example) to write the high-performance string 
> manipulation that have been fine-tuned and extensively optimized in Perl.

Well, the code is all at
http://cvs.distributed.net/viewcvs.cgi/stats-proc/hourly/ (see logmod
directory and logmod_*.pl). There have been changes made to the C code
since we changed over, but you can find the appropriate older versions
in there. IIRC, nothing in the database changed when we went from perl
to C (it's likely that was the *only* change that happened anywhere
around that time).
-- 
Jim Nasby                                            jim@nasby.net
EnterpriseDB      http://enterprisedb.com      512.569.9461 (cell)