X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id C3C501FEE591
	for <pgsql-performance-postgresql.org@localhost.postgresql.org>;
	Sun, 21 Aug 2005 11:54:38 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 98444-10
	for <pgsql-performance-postgresql.org@localhost.postgresql.org>;
	Sun, 21 Aug 2005 14:54:29 +0000 (GMT)
Received: from penguin.goodinassociates.com (unknown [206.80.71.229])
	by svr1.postgresql.org (Postfix) with ESMTP id 463361FEE588
	for <pgsql-performance@postgresql.org>;
	Sun, 21 Aug 2005 11:54:29 -0300 (ADT)
Received: from [206.80.71.242] ([206.80.71.242])
	by penguin.goodinassociates.com (8.13.1/8.12.11) with ESMTP id
	j7LEsQbd020366; Sun, 21 Aug 2005 09:54:27 -0500
Subject: Re: extremly low memory usage
From: Jeremiah Jahn <jeremiah@cs.earlham.edu>
To: John A Meinel <john@arbash-meinel.com>
Cc: Ron <rjpeace@earthlink.net>,
	postgres performance <pgsql-performance@postgresql.org>
In-Reply-To: <4307E7A4.7090300@arbash-meinel.com>
References: <1124331108.27881.96.camel@bluejay.goodinassociates.com>
	<EA4219B9-E2B0-4756-ABF2-D84F245FDFB5@torgo.978.org>
	<1124386761.27881.119.camel@bluejay.goodinassociates.com>
	<4304CB77.4070403@arbash-meinel.com>
	<1124470109.27881.152.camel@bluejay.goodinassociates.com>
	<43061472.2080201@arbash-meinel.com>
	<1124481386.27881.167.camel@bluejay.goodinassociates.com>
	<43064908.7010505@arbash-meinel.com>
	<1124564003.27881.229.camel@bluejay.goodinassociates.com>
	<6.2.3.4.0.20050820154422.06021150@pop.earthlink.net>
	<4307E7A4.7090300@arbash-meinel.com>
Content-Type: text/plain
Date: Sun, 21 Aug 2005 09:54:26 -0500
Message-Id: <1124636066.27881.238.camel@bluejay.goodinassociates.com>
Mime-Version: 1.0
X-Mailer: Evolution 2.2.2 (2.2.2-5) 
Content-Transfer-Encoding: 7bit
Received-SPF: none (penguin.goodinassociates.com: 206.80.71.242 is neither
	permitted nor denied by domain of cs.earlham.edu)
	client-ip=206.80.71.242; envelope-from=jeremiah@cs.earlham.edu;
	helo=[206.80.71.242];
X-Virus-Scanned: ClamAV 0.86.1/1034/Thu Aug 18 15:07:58 2005 on
	penguin.goodinassociates.com
X-Virus-Status: Clean
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.01 required=5 tests=[AWL=0.010]
X-Spam-Level: 
X-Archive-Number: 200508/307
X-Sequence-Number: 14054

On Sat, 2005-08-20 at 21:32 -0500, John A Meinel wrote:
> Ron wrote:
> > At 02:53 PM 8/20/2005, Jeremiah Jahn wrote:
> > 
> >> On Fri, 2005-08-19 at 16:03 -0500, John A Meinel wrote:
> >> > Jeremiah Jahn wrote:
> >> > > On Fri, 2005-08-19 at 12:18 -0500, John A Meinel wrote:
> >> > >
> >> <snip>
> >> >
> >> > > it's cached alright. I'm getting a read rate of about 150MB/sec. I 
> >> would
> >> > > have thought is would be faster with my raid setup. I think I'm 
> >> going to
> >> > > scrap the whole thing and get rid of LVM. I'll just do a straight 
> >> ext3
> >> > > system. Maybe that will help. Still trying to get suggestions for a
> >> > > stripe size.
> >> > >
> 
> Well, since you can get a read of the RAID at 150MB/s, that means that 
> it is actual I/O speed. It may not be cached in RAM. Perhaps you could 
> try the same test, only using say 1G, which should be cached.

[root@io pgsql]# time dd if=/dev/zero of=testfile bs=1024 count=1000000
1000000+0 records in
1000000+0 records out

real    0m8.885s
user    0m0.299s
sys     0m6.998s
[root@io pgsql]# time dd of=/dev/null if=testfile bs=1024 count=1000000
1000000+0 records in
1000000+0 records out

real    0m1.654s
user    0m0.232s
sys     0m1.415s


> 
> >> >
> >> > I don't think 150MB/s is out of the realm for a 14 drive array.
> >> > How fast is time dd if=/dev/zero of=testfile bs=8192 count=1000000
> >> >
> >> time dd if=/dev/zero of=testfile bs=8192 count=1000000
> >> 1000000+0 records in
> >> 1000000+0 records out
> >>
> >> real    1m24.248s
> >> user    0m0.381s
> >> sys     0m33.028s
> >>
> >>
> >> > (That should create a 8GB file, which is too big to cache everything)
> >> > And then how fast is:
> >> > time dd if=testfile of=/dev/null bs=8192 count=1000000
> >>
> >> time dd if=testfile of=/dev/null bs=8192 count=1000000
> >> 1000000+0 records in
> >> 1000000+0 records out
> >>
> >> real    0m54.139s
> >> user    0m0.326s
> >> sys     0m8.916s
> >>
> >>
> >> and on a second run:
> >>
> >> real    0m55.667s
> >> user    0m0.341s
> >> sys     0m9.013s
> >>
> >>
> >> >
> >> > That should give you a semi-decent way of measuring how fast the RAID
> >> > system is, since it should be too big to cache in ram.
> >>
> >> about 150MB/Sec. Is there no better way to make this go faster...?
> 
> I'm actually curious about PCI bus saturation at this point. Old 32-bit 
> 33MHz pci could only push 1Gbit = 100MB/s. Now, I'm guessing that this 
> is a higher performance system. But I'm really surprised that your write 
> speed is that close to your read speed. (100MB/s write, 150MB/s read).

The raid array I have is currently set up to use a single channel. But I
have dual controllers In the array. And dual external slots on the card.
The machine is brand new and has pci-e backplane.  



> 
> > 
> > Assuming these are U320 15Krpm 147GB HDs, a RAID 10 array of 14 of them 
> > doing raw sequential IO like this should be capable of at
> >  ~7*75MB/s= 525MB/s using Seagate Cheetah 15K.4's,  ~7*79MB/s= 553MB/s 
BTW I'm using Seagate Cheetah 15K.4's

> > if using Fujitsu MAU's, and ~7*86MB/s= 602MB/s if using Maxtor Atlas 15K 
> > II's to devices external to the RAID array.
> 
> I know I thought these were SATA drives, over 2 controllers. I could be 
> completely wrong, though.
> 
> > 
> > _IF_ the controller setup is high powered enough to keep that kind of IO 
> > rate up.  This will require a controller or controllers providing dual 
> > channel U320 bandwidth externally and quad channel U320 bandwidth 
> > internally.  IOW, it needs a controller or controllers talking 64b 
> > 133MHz PCI-X, reasonably fast DSP/CPU units, and probably a decent sized 
> > IO buffer as well.
> > 
> > AFAICT, the Dell PERC4 controllers use various flavors of the LSI Logic 
> > MegaRAID controllers.  What I don't know is which exact one yours is, 
> > nor do I know if it (or any of the MegaRAID controllers) are high 
> > powered enough.

PERC4eDC-PCI Express, 128MB Cache, 2-External Channels

> > 
> > Talk to your HW supplier to make sure you have controllers adequate to 
> > your HD's.
> > 
> > ...and yes, your average access time will be in the 5.5ms - 6ms range 
> > when doing a physical seek.
> > Even with RAID, you want to minimize seeks and maximize sequential IO 
> > when accessing them.
> > Best to not go to HD at all ;-)
> 
> Well, certainly, if you can get more into RAM, you're always better off. 
> For writing, a battery-backed write cache, and for reading lots of 
> system RAM.

I'm not really worried about the writing, it's the reading the reading
that needs to be faster. 

> 
> > 
> > Hope this helps,
> > Ron Peacetree
> > 
> 
> John
> =:->
-- 
Speak softly and carry a +6 two-handed sword.