From: Mark Wong <markw@osdl.org>
To: Luke Lonergan <llonergan@greenplum.com>
Cc: Andrew Dunstan <andrew@dunslane.net>
Cc: Alvaro Herrera <alvherre@surnet.cl>
Cc: Bruce Momjian <pgman@candle.pha.pa.us>
Cc: Alon Goldshuv <agoldshuv@greenplum.com>
Cc: pgsql-performance@postgresql.org
Cc: maryedie@osdl.org
Subject: Re: [PATCHES] COPY FROM performance improvements
Date: Fri, 22 Jul 2005 12:47:18 -0700
Message-ID: <200507221946.j6MJkljA024095@smtp.osdl.org> (raw)
In-Reply-To: <BF057A77.96D7%llonergan@greenplum.com>
References: <200507212154.j6LLsdjA002847@smtp.osdl.org>
<BF057A77.96D7%llonergan@greenplum.com>
On a single spindle:
$ time dd if=/dev/zero of=bigfile bs=8k count=2000000
2000000+0 records in
2000000+0 records out
real 2m8.569s
user 0m0.725s
sys 0m19.633s
None of my drives are partitioned big enough for me to create 2x RAM
sized files on a single disk. I have 16MB RAM and only 36GB drives.
But here are some number for my 12-disk lvm2 striped volume.
$ time dd if=/dev/zero of=bigfile3 bs=8k count=4000000
4000000+0 records in
4000000+0 records out
real 1m17.059s
user 0m1.479s
sys 0m41.293s
Mark
On Thu, 21 Jul 2005 16:14:47 -0700
"Luke Lonergan" <llonergan@greenplum.com> wrote:
> Cool!
>
> At what rate does your disk setup write sequential data, e.g.:
> time dd if=/dev/zero of=bigfile bs=8k count=500000
>
> (sized for 2x RAM on a system with 2GB)
>
> BTW - the Compaq smartarray controllers are pretty broken on Linux from a
> performance standpoint in our experience. We've had disastrously bad
> results from the SmartArray 5i and 6 controllers on kernels from 2.4 ->
> 2.6.10, on the order of 20MB/s.
>
> For comparison, the results on our dual opteron with a single LSI SCSI
> controller with software RAID0 on a 2.6.10 kernel:
>
> [llonergan@stinger4 dbfast]$ time dd if=/dev/zero of=bigfile bs=8k
> count=500000
> 500000+0 records in
> 500000+0 records out
>
> real 0m24.702s
> user 0m0.077s
> sys 0m8.794s
>
> Which calculates out to about 161MB/s.
>
> - Luke
>
>
> On 7/21/05 2:55 PM, "Mark Wong" <markw@osdl.org> wrote:
>
> > I just ran through a few tests with the v14 patch against 100GB of data
> > from dbt3 and found a 30% improvement; 3.6 hours vs 5.3 hours. Just to
> > give a few details, I only loaded data and started a COPY in parallel
> > for each the data files:
> > http://www.testing.osdl.org/projects/dbt3testing/results/fast_copy/
> >
> > Here's a visual of my disk layout, for those familiar with the database
> > schema:
> > http://www.testing.osdl.org/projects/dbt3testing/results/fast_copy/layout-dev4
> > -010-dbt3.html
> >
> > I have 6 arrays of fourteen 15k rpm drives in a split-bus configuration
> > attached to a 4-way itanium2 via 6 compaq smartarray pci-x controllers.
> >
> > Let me know if you have any questions.
> >
> > Mark
> >
>
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: markw@osdl.org, llonergan@greenplum.com, andrew@dunslane.net, alvherre@surnet.cl, pgman@candle.pha.pa.us, agoldshuv@greenplum.com, maryedie@osdl.org
Subject: Re: [PATCHES] COPY FROM performance improvements
In-Reply-To: <200507221946.j6MJkljA024095@smtp.osdl.org>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox