X-Original-To: pgsql-patches-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id A5CAA528DE for ; Thu, 21 Jul 2005 20:15:03 -0300 (ADT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 16472-01 for ; Thu, 21 Jul 2005 23:14:57 +0000 (GMT) Received: from mail.Mi8.com (d01gw01.mi8.com [63.240.6.47]) by svr1.postgresql.org (Postfix) with ESMTP id D9BEA52836 for ; Thu, 21 Jul 2005 20:14:55 -0300 (ADT) Received: from 172.16.1.118 by mail.Mi8.com with ESMTP (- Welcome to Mi8 Corporation www.Mi8.com (D1)); Thu, 21 Jul 2005 19:14:49 -0400 X-Server-Uuid: 241911D6-425B-44B9-A073-E3FE0F8FC774 Received: from MI8NYCMAIL06.Mi8.com ([172.16.1.175]) by d01smtp03.Mi8.com with Microsoft SMTPSVC(6.0.3790.1830); Thu, 21 Jul 2005 19:14:48 -0400 Received: from 67.103.45.218 ([67.103.45.218]) by MI8NYCMAIL06.Mi8.com ( [172.16.1.219]) via Exchange Front-End Server mi8owa.mi8.com ( [172.16.1.104]) with Microsoft Exchange Server HTTP-DAV ; Thu, 21 Jul 2005 19:14:48 -0500 User-Agent: Microsoft-Entourage/11.1.0.040913 Date: Thu, 21 Jul 2005 16:14:47 -0700 Subject: Re: COPY FROM performance improvements From: "Luke Lonergan" To: "Mark Wong" Cc: "Andrew Dunstan" , "Alvaro Herrera" , "Bruce Momjian" , "Alon Goldshuv" , pgsql-patches@postgresql.org, maryedie@osdl.org Message-ID: In-Reply-To: <200507212154.j6LLsdjA002847@smtp.osdl.org> MIME-Version: 1.0 X-OriginalArrivalTime: 21 Jul 2005 23:14:48.0751 (UTC) FILETIME=[FD8C3BF0:01C58E49] X-WSS-ID: 6EFEF3E32B44412511-01-01 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=1.574 tagged_above=0 required=5 tests=AWL, FORGED_RCVD_HELO, RCVD_NUMERIC_HELO X-Spam-Level: * X-Archive-Number: 200507/425 X-Sequence-Number: 16765 Cool! At what rate does your disk setup write sequential data, e.g.: time dd if=/dev/zero of=bigfile bs=8k count=500000 (sized for 2x RAM on a system with 2GB) BTW - the Compaq smartarray controllers are pretty broken on Linux from a performance standpoint in our experience. We've had disastrously bad results from the SmartArray 5i and 6 controllers on kernels from 2.4 -> 2.6.10, on the order of 20MB/s. For comparison, the results on our dual opteron with a single LSI SCSI controller with software RAID0 on a 2.6.10 kernel: [llonergan@stinger4 dbfast]$ time dd if=/dev/zero of=bigfile bs=8k count=500000 500000+0 records in 500000+0 records out real 0m24.702s user 0m0.077s sys 0m8.794s Which calculates out to about 161MB/s. - Luke On 7/21/05 2:55 PM, "Mark Wong" wrote: > I just ran through a few tests with the v14 patch against 100GB of data > from dbt3 and found a 30% improvement; 3.6 hours vs 5.3 hours. Just to > give a few details, I only loaded data and started a COPY in parallel > for each the data files: > http://www.testing.osdl.org/projects/dbt3testing/results/fast_copy/ > > Here's a visual of my disk layout, for those familiar with the database > schema: > http://www.testing.osdl.org/projects/dbt3testing/results/fast_copy/layout-dev4 > -010-dbt3.html > > I have 6 arrays of fourteen 15k rpm drives in a split-bus configuration > attached to a 4-way itanium2 via 6 compaq smartarray pci-x controllers. > > Let me know if you have any questions. > > Mark >