Message-ID: <48C8006F.50706@brightbuilders.com>
Date: Wed, 10 Sep 2008 11:14:23 -0600
From: Ryan Hansen <ryan.hansen@brightbuilders.com>
User-Agent: Thunderbird 2.0.0.16 (X11/20080724)
MIME-Version: 1.0
To: pgsql-performance@postgresql.org
Subject: Re: Improve COPY performance for large data sets
Content-Type: multipart/mixed; boundary="------------040704080001080404010401"

This is a multi-part message in MIME format.
--------------040704080001080404010401
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

NEVERMIND!!

I found it.  Turns out there was still a constraint on the table.  Once 
I dropped that, the time went down to 44 minutes.

Maybe I am an idiot after all. :)

-Ryan

--------------040704080001080404010401
Content-Type: message/rfc822;
	name="Improve COPY performance for large data sets.eml"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
	filename="Improve COPY performance for large data sets.eml"

X-Mozilla-Keys: 
Message-ID: <48C7FA69.1080507@brightbuilders.com>
Date: Wed, 10 Sep 2008 10:48:41 -0600
From: Ryan Hansen <ryan.hansen@brightbuilders.com>
User-Agent: Thunderbird 2.0.0.16 (X11/20080724)
MIME-Version: 1.0
To: pgsql-performance@postgresql.org
Subject: Improve COPY performance for large data sets
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Greetings,

I'm relatively new to PostgreSQL but I've been in the IT applications 
industry for a long time, mostly in the LAMP world.

One thing I'm experiencing some trouble with is running a COPY of a 
large file (20+ million records) into a table in a reasonable amount of 
time.  Currently it's taking about 12 hours to complete on a 64 bit 
server with 3 GB memory allocated (shared_buffer), single SATA 320 GB 
drive.  I don't seem to get any improvement running the same operation 
on a dual opteron dual-core, 16 GB server.

I'm not asking for someone to solve my problem, just some direction in 
the best ways to tune for faster bulk loading, since this will be a 
fairly regular operation for our application (assuming it can work this 
way).  I've toyed with the maintenance_work_mem and some of the other 
params, but it's still way slower than it seems like it should be.
So any contributions are much appreciated.

Thanks!

P.S. Assume I've done a ton of reading and research into PG tuning, 
which I have.  I just can't seem to find anything beyond the basics that 
talks about really speeding up bulk loads.


--------------040704080001080404010401--