pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Jim C. Nasby <jim@nasby.net>
To: Worky Workerson <worky.workerson@gmail.com>
Cc: Craig A. James <cjames@modgraph-usa.com>
Cc: Merlin Moncure <mmoncure@gmail.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: Best COPY Performance
Date: Wed, 25 Oct 2006 10:28:13 -0500
Message-ID: <20061025152812.GQ26892@nasby.net> (raw)
In-Reply-To: <ce4072df0610250503g26f32cb7r4e668613c8665774@mail.gmail.com>
References: <ce4072df0610201305u7836dc7bn437ec0aaf1c3e25f@mail.gmail.com>
	<b42b73150610201811m84369f7kc7d58b39aed3ad73@mail.gmail.com>
	<ce4072df0610230810t5d66bbb8xf1e9c0ed71b6b6c0@mail.gmail.com>
	<20061023215913.GW26892@nasby.net>
	<ce4072df0610240617s72f35540odd3d4e0903313b63@mail.gmail.com>
	<20061025001709.GA26892@nasby.net>
	<453EF7C4.2040105@modgraph-usa.com>
	<ce4072df0610250503g26f32cb7r4e668613c8665774@mail.gmail.com>

On Wed, Oct 25, 2006 at 08:03:38AM -0400, Worky Workerson wrote:
> I'm just doing CSV style transformations (and calling a lot of
> functions along the way), but the end result is a straight bulk load
> of data into a blank database.  And we've established that Postgres
> can do *way* better than what I am seeing, so its not suprising that
> perl is using 100% of a CPU.

If you're loading into an empty database, there's a number of tricks
that will help you:

Turn off fsync
Add constraints and indexes *after* you've loaded the data (best to add
as much of them as possible on a per-table basis right after the table
is loaded so that it's hopefully still in cache)
Crank up maintenance_work_mem, especially for tables that won't fit into
cache anyway
Bump up checkpoint segments and wal_buffers.
Disable PITR
Create a table and load it's data in a single transaction (8.2 will
avoid writing any WAL data if you do this and PITR is turned off)
-- 
Jim Nasby                                            jim@nasby.net
EnterpriseDB      http://enterprisedb.com      512.569.9461 (cell)



view thread (52+ messages)  latest in thread

Message-ID: <20061025152812.GQ26892@nasby.net>
Permalink:  ../20061025152812.GQ26892@nasby.net/
Also on:    postgresql.org/message-id/20061025152812.GQ26892@nasby.net

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-performance@postgresql.org
  Cc: jim@nasby.net, worky.workerson@gmail.com, cjames@modgraph-usa.com, mmoncure@gmail.com
  Subject: Re: Best COPY Performance
  In-Reply-To: <20061025152812.GQ26892@nasby.net>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox