Message-ID: <42E038E9.1080107@commandprompt.com>
Date: Thu, 21 Jul 2005 17:08:09 -0700
From: "Joshua D. Drake" <jd@commandprompt.com>
Organization: Command Prompt, Inc.
User-Agent: Mozilla Thunderbird 1.0.2 (X11/20050404)
MIME-Version: 1.0
To: Luke Lonergan <llonergan@greenplum.com>
Cc: Mark Wong <markw@osdl.org>, Andrew Dunstan <andrew@dunslane.net>,
	Alvaro Herrera <alvherre@surnet.cl>,
	Bruce Momjian <pgman@candle.pha.pa.us>,
	Alon Goldshuv <agoldshuv@greenplum.com>,
	pgsql-patches@postgresql.org, maryedie@osdl.org
Subject: Re: COPY FROM performance improvements
References: <BF057A77.96D7%llonergan@greenplum.com>
In-Reply-To: <BF057A77.96D7%llonergan@greenplum.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Luke Lonergan wrote:
> Cool!
> 
> At what rate does your disk setup write sequential data, e.g.:
>   time dd if=/dev/zero of=bigfile bs=8k count=500000
> 
> (sized for 2x RAM on a system with 2GB)
> 
> BTW - the Compaq smartarray controllers are pretty broken on Linux from a
> performance standpoint in our experience.  We've had disastrously bad
> results from the SmartArray 5i and 6 controllers on kernels from 2.4 ->
> 2.6.10, on the order of 20MB/s.

O.k. this strikes me as interesting, now we know that Compaq and Dell 
are borked for Linux. Is there a name brand server (read Enterprise) 
that actually does provide reasonable performance?

> 
> For comparison, the results on our dual opteron with a single LSI SCSI
> controller with software RAID0 on a 2.6.10 kernel:
> 
> [llonergan@stinger4 dbfast]$ time dd if=/dev/zero of=bigfile bs=8k
> count=500000
> 500000+0 records in
> 500000+0 records out
> 
> real    0m24.702s
> user    0m0.077s
> sys     0m8.794s
> 
> Which calculates out to about 161MB/s.
> 
> - Luke
> 
> 
> On 7/21/05 2:55 PM, "Mark Wong" <markw@osdl.org> wrote:
> 
> 
>>I just ran through a few tests with the v14 patch against 100GB of data
>>from dbt3 and found a 30% improvement; 3.6 hours vs 5.3 hours.  Just to
>>give a few details, I only loaded data and started a COPY in parallel
>>for each the data files:
>>http://www.testing.osdl.org/projects/dbt3testing/results/fast_copy/
>>
>>Here's a visual of my disk layout, for those familiar with the database
>>schema:
>>http://www.testing.osdl.org/projects/dbt3testing/results/fast_copy/layout-dev4
>>-010-dbt3.html
>>
>>I have 6 arrays of fourteen 15k rpm drives in a split-bus configuration
>>attached to a 4-way itanium2 via 6 compaq smartarray pci-x controllers.
>>
>>Let me know if you have any questions.
>>
>>Mark
>>
> 
> 
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 9: In versions below 8.0, the planner will ignore your desire to
>        choose an index scan if your joining column's datatypes do not
>        match


-- 
Your PostgreSQL solutions provider, Command Prompt, Inc.
24x7 support - 1.800.492.2240, programming, and consulting
Home of PostgreSQL Replicator, plPHP, plPerlNG and pgPHPToolkit
http://www.commandprompt.com / http://www.postgresql.org