From: Worky Workerson <worky.workerson@gmail.com>
To: Jim C. Nasby <jim@nasby.net>
Cc: Merlin Moncure <mmoncure@gmail.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: Best COPY Performance
Date: Tue, 24 Oct 2006 09:17:08 -0400
Message-ID: <ce4072df0610240617s72f35540odd3d4e0903313b63@mail.gmail.com> (raw)
In-Reply-To: <20061023215913.GW26892@nasby.net>
References: <ce4072df0610201305u7836dc7bn437ec0aaf1c3e25f@mail.gmail.com>
<b42b73150610201811m84369f7kc7d58b39aed3ad73@mail.gmail.com>
<ce4072df0610230810t5d66bbb8xf1e9c0ed71b6b6c0@mail.gmail.com>
<20061023215913.GW26892@nasby.net>
> http://stats.distributed.net used to use a perl script to do some
> transformations before loading data into the database. IIRC, when we
> switched to using C we saw 100x improvement in speed, so I suspect that
> if you want performance perl isn't the way to go. I think you can
> compile perl into C, so maybe that would help some.
Like Craig mentioned, I have never seen those sorts of improvements
going from perl->C, and developer efficiency is primo for me. I've
profiled most of the stuff, and have used XS modules and Inline::C on
the appropriate, often used functions, but I still think that it comes
down to my using CSV and Text::CSV_XS. Even though its XS, CSV is
still a pain in the ass.
> Ultimately, you might be best of using triggers instead of rules for the
> partitioning since then you could use copy. Or go to raw insert commands
> that are wrapped in a transaction.
Eh, I've put the partition loading logic in the loader, which seems to
work out pretty well, especially since I keep things sorted and am the
only one inserting into the DB and do so with bulk loads. But I'll
keep this in mind for later use.
Thanks!
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: worky.workerson@gmail.com, jim@nasby.net, mmoncure@gmail.com
Subject: Re: Best COPY Performance
In-Reply-To: <ce4072df0610240617s72f35540odd3d4e0903313b63@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox