User-Agent: Microsoft-Entourage/11.2.5.060620
Date: Mon, 30 Oct 2006 08:03:41 -0700
Subject: Re: Best COPY Performance
From: "Luke Lonergan" <llonergan@greenplum.com>
To: "Stefan Kaltenbrunner" <stefan@kaltenbrunner.cc>
cc: "Spiegelberg, Greg" <gspiegelberg@cranel.com>,
	"Worky Workerson" <worky.workerson@gmail.com>,
	"Merlin Moncure" <mmoncure@gmail.com>, pgsql-performance@postgresql.org
Message-ID: <C16B625D.5708%llonergan@greenplum.com>
Thread-Topic: [PERFORM] Best COPY Performance
Thread-Index: Acb8PPeqNfqbKGgwEduGtgAWy4o9DA==
In-Reply-To: <454620DF.1000003@kaltenbrunner.cc>
MIME-Version: 1.0
Content-Type: text/plain;
 charset=us-ascii
Content-Transfer-Encoding: 7bit

Stefan,

On 10/30/06 8:57 AM, "Stefan Kaltenbrunner" <stefan@kaltenbrunner.cc> wrote:

>> We've found that there is an ultimate bottleneck at about 12-14MB/s despite
>> having sequential write to disk speeds of 100s of MB/s.  I forget what the
>> latest bottleneck was.
> 
> I have personally managed to load a bit less then 400k/s (5 int columns
> no indexes) - on very fast disk hardware - at that point postgresql is
> completely CPU bottlenecked (2,6Ghz Opteron).

400,000 rows/s x 4 bytes/column x 5 columns/row = 8MB/s

> Using multiple processes to load the data will help to scale up to about
>   900k/s (4 processes on 4 cores).

18MB/s?  Have you done this?  I've not seen this much of an improvement
before by using multiple COPY processes to the same table.

Another question: how to measure MB/s - based on the input text file?  On
the DBMS storage size?  We usually consider the input text file in the
calculation of COPY rate.

- Luke