From: Luke Lonergan <llonergan@greenplum.com>
To: Stefan Kaltenbrunner <stefan@kaltenbrunner.cc>
Cc: Spiegelberg, Greg <gspiegelberg@cranel.com>
Cc: Worky Workerson <worky.workerson@gmail.com>
Cc: Merlin Moncure <mmoncure@gmail.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: Best COPY Performance
Date: Mon, 30 Oct 2006 08:03:41 -0700
Message-ID: <C16B625D.5708%llonergan@greenplum.com> (raw)
In-Reply-To: <454620DF.1000003@kaltenbrunner.cc>
Stefan,
On 10/30/06 8:57 AM, "Stefan Kaltenbrunner" <stefan@kaltenbrunner.cc> wrote:
>> We've found that there is an ultimate bottleneck at about 12-14MB/s despite
>> having sequential write to disk speeds of 100s of MB/s. I forget what the
>> latest bottleneck was.
>
> I have personally managed to load a bit less then 400k/s (5 int columns
> no indexes) - on very fast disk hardware - at that point postgresql is
> completely CPU bottlenecked (2,6Ghz Opteron).
400,000 rows/s x 4 bytes/column x 5 columns/row = 8MB/s
> Using multiple processes to load the data will help to scale up to about
> 900k/s (4 processes on 4 cores).
18MB/s? Have you done this? I've not seen this much of an improvement
before by using multiple COPY processes to the same table.
Another question: how to measure MB/s - based on the input text file? On
the DBMS storage size? We usually consider the input text file in the
calculation of COPY rate.
- Luke
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: llonergan@greenplum.com, stefan@kaltenbrunner.cc, gspiegelberg@cranel.com, worky.workerson@gmail.com, mmoncure@gmail.com
Subject: Re: Best COPY Performance
In-Reply-To: <C16B625D.5708%llonergan@greenplum.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox