From fhevia@ip-tel.com.ar Wed Dec 26 15:14:46 2007 Received: from localhost (unknown [200.46.204.184]) by postgresql.org (Postfix) with ESMTP id D419B2E0077 for ; Wed, 26 Dec 2007 11:14:46 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.184]) (amavisd-maia, port 10024) with ESMTP id 05571-05 for ; Wed, 26 Dec 2007 11:14:31 -0400 (AST) Received: from pyxis.powervt.com.ar (host3.powervt.com.ar [200.43.63.66]) by postgresql.org (Postfix) with ESMTP id C78482E00E6 for ; Wed, 26 Dec 2007 11:14:39 -0400 (AST) Received: from pcfhevia ([201.216.249.98]) by pyxis.powervt.com.ar (IPGroup Mail Server) with ESMTP id 1OT48953 for ; Wed, 26 Dec 2007 12:14:37 -0300 From: "Fernando Hevia" To: "'pgsql-performance'" Subject: With 4 disks should I go for RAID 5 or RAID 10 Date: Wed, 26 Dec 2007 12:15:03 -0300 Organization: IP-TEL Message-ID: <0ea101c847d2$16f94bc0$8f01010a@iptel.com.ar> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0EA2_01C847B8.F1AC13C0" X-Mailer: Microsoft Office Outlook 11 Thread-Index: AchH0hW3pWqaNxHJRb2fxqprlGJkJg== X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3198 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/357 X-Sequence-Number: 28173 This is a multi-part message in MIME format. ------=_NextPart_000_0EA2_01C847B8.F1AC13C0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hi list, I am building kind of a poor man's database server: Pentium D 945 (2 x 3 Ghz cores) 4 GB RAM 4 x 160 GB SATA II 7200 rpm (Intel server motherboard has only 4 SATA ports) Database will be about 30 GB in size initially and growing 10 GB per year. Data is inserted overnight in two big tables and during the day mostly read-only queries are run. Parallelism is rare. I have read about different raid levels with Postgres but the advice found seems to apply on 8+ disks systems. With only four disks and performance in mind should I build a RAID 10 or RAID 5 array? Raid 0 is overruled since redundancy is needed. I am going to use software Raid with Linux (Ubuntu Server 6.06). Thanks for any hindsight. Regards, Fernando. ------=_NextPart_000_0EA2_01C847B8.F1AC13C0 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable With 4 disks should I go for RAID 5 or RAID 10

Hi list,

I am = building kind of a poor man’s database server:

Pentium = D 945 (2 x 3 Ghz cores)

4 GB = RAM

4 x 160 = GB SATA II 7200 rpm (Intel server motherboard has only 4 SATA ports)

Database will be about 30 GB in size = initially = and growing 10 GB per = year. Data is inserted = overnight = in two big tables = and = during the day = mostly = read-only queries are = run. Parallelism is = rare.

I have read about different raid levels with = Postgres but the advice found seems to apply on 8+ disks = systems. = With = only four disks and = performance in mind should I = build a RAID = 10 or RAID 5 array? Raid 0 is overruled since redundancy is = needed.

I am going to use software Raid with Linux (Ubuntu Server = 6.06).

Thanks for any hindsight.

Regards,

Fernando.

------=_NextPart_000_0EA2_01C847B8.F1AC13C0-- From wmoran@collaborativefusion.com Wed Dec 26 15:22:43 2007 Received: from localhost (unknown [200.46.204.187]) by postgresql.org (Postfix) with ESMTP id 60B452E0232 for ; Wed, 26 Dec 2007 11:22:43 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.187]) (amavisd-maia, port 10024) with ESMTP id 65146-04-2 for ; Wed, 26 Dec 2007 11:22:08 -0400 (AST) Received: from mx00.pub.collaborativefusion.com (mx00.pub.collaborativefusion.com [206.210.89.199]) by postgresql.org (Postfix) with ESMTP id 674972E00D4 for ; Wed, 26 Dec 2007 11:21:07 -0400 (AST) Received: from vanquish.pitbpa0.priv.collaborativefusion.com (vanquish.pitbpa0.priv.collaborativefusion.com [192.168.2.61]) (SSL: TLSv1/SSLv3,256bits,AES256-SHA) by wingspan with esmtp; Wed, 26 Dec 2007 10:21:05 -0500 id 00056416.47727161.00010637 Date: Wed, 26 Dec 2007 10:21:05 -0500 From: Bill Moran To: "Fernando Hevia" Cc: "'pgsql-performance'" Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 Message-Id: <20071226102105.57d9b502.wmoran@collaborativefusion.com> In-Reply-To: <0ea101c847d2$16f94bc0$8f01010a@iptel.com.ar> References: <0ea101c847d2$16f94bc0$8f01010a@iptel.com.ar> Organization: Collaborative Fusion X-Mailer: Sylpheed 2.4.7 (GTK+ 2.12.1; i386-portbld-freebsd6.2) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/358 X-Sequence-Number: 28174 RAID 10. I snipped the rest of your message because none of it matters. Never use RAID 5 on a database system. Ever. There is absolutely NO reason to every put yourself through that much suffering. If you hate yourself that much just commit suicide, it's less drastic. -- Bill Moran Collaborative Fusion Inc. http://people.collaborativefusion.com/~wmoran/ wmoran@collaborativefusion.com Phone: 412-422-3463x4023 From mark@mark.mielke.cc Wed Dec 26 16:26:42 2007 Received: from localhost (unknown [200.46.204.187]) by postgresql.org (Postfix) with ESMTP id 319092E0337 for ; Wed, 26 Dec 2007 12:26:42 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.187]) (amavisd-maia, port 10024) with ESMTP id 84019-04 for ; Wed, 26 Dec 2007 12:26:34 -0400 (AST) Received: from ironport2-out.pppoe.ca (ironport2-out.pppoe.ca [206.248.154.182]) by postgresql.org (Postfix) with ESMTP id BA9D92E033E for ; Wed, 26 Dec 2007 12:26:34 -0400 (AST) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ao8CAOMPckfO+Igp/2dsb2JhbACCb6ZU X-IronPort-AV: E=Sophos;i="4.24,208,1196658000"; d="scan'208,217";a="11979816" Received: from smtp.pppoe.ca ([65.39.196.238]) by ironport2-out.pppoe.ca with ESMTP; 26 Dec 2007 11:26:32 -0500 Received: from [206.248.136.41] ([206.248.136.41]) by smtp.pppoe.ca (Internet Mail Server v1.0) with ESMTP id FCA39832; Wed, 26 Dec 2007 11:26:32 -0500 Message-ID: <47728000.4000709@mark.mielke.cc> Date: Wed, 26 Dec 2007 11:23:28 -0500 From: Mark Mielke User-Agent: Thunderbird 2.0.0.9 (X11/20071115) MIME-Version: 1.0 To: Fernando Hevia CC: 'pgsql-performance' Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 References: <0ea101c847d2$16f94bc0$8f01010a@iptel.com.ar> In-Reply-To: <0ea101c847d2$16f94bc0$8f01010a@iptel.com.ar> Content-Type: multipart/alternative; boundary="------------020407020400030204000305" X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/360 X-Sequence-Number: 28176 This is a multi-part message in MIME format. --------------020407020400030204000305 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Fernando Hevia wrote: > > Database will be about 30 GB in size initially and growing 10 GB per > year. Data is inserted overnight in two big tables and during the day > mostly read-only queries are run. Parallelism is rare. > > I have read about different raid levels with Postgres but the advice > found seems to apply on 8+ disks systems. With only four disks and > performance in mind should I build a RAID 10 or RAID 5 array? Raid 0 > is overruled since redundancy is needed. > > I am going to use software Raid with Linux (Ubuntu Server 6.06). > In my experience, software RAID 5 is horrible. Write performance can decrease below the speed of one disk on its own, and read performance will not be significantly more than RAID 1+0 as the number of stripes has only increased from 2 to 3, and if reading while writing, you will not get 3X as RAID 5 write requires at least two disks to be involved. I believe hardware RAID 5 is also horrible, but since the hardware hides it from the application, a hardware RAID 5 user might not care. Software RAID 1+0 works fine on Linux with 4 disks. This is the setup I use for my personal server. Cheers, mark -- Mark Mielke --------------020407020400030204000305 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Fernando Hevia wrote:
With 4 disks should I go for RAID 5 or RAID 10

Database will be about 30 GB in size initially and growing 10 GB per year. Data is inserted overnight in two big tables and during the day mostly read-only queries are run. Parallelism is rare.

I have read about different raid levels with Postgres but the advice found seems to apply on 8+ disks systems. With only four disks and performance in mind should I build a RAID 10 or RAID 5 array? Raid 0 is overruled since redundancy is needed.

I am going to use software Raid with Linux (Ubuntu Server 6.06).


In my experience, software RAID 5 is horrible. Write performance can decrease below the speed of one disk on its own, and read performance will not be significantly more than RAID 1+0 as the number of stripes has only increased from 2 to 3, and if reading while writing, you will not get 3X as RAID 5 write requires at least two disks to be involved. I believe hardware RAID 5 is also horrible, but since the hardware hides it from the application, a hardware RAID 5 user might not care.

Software RAID 1+0 works fine on Linux with 4 disks. This is the setup I use for my personal server.

Cheers,
mark

 
-- 
Mark Mielke <mark@mielke.cc>
--------------020407020400030204000305-- From gsmith@gregsmith.com Wed Dec 26 18:14:12 2007 Received: from localhost (unknown [200.46.204.191]) by postgresql.org (Postfix) with ESMTP id AB2742E162B for ; Wed, 26 Dec 2007 14:14:12 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.191]) (amavisd-maia, port 10024) with ESMTP id 07355-03 for ; Wed, 26 Dec 2007 14:14:08 -0400 (AST) Received: from westnet.com (westnet.com [216.187.52.2]) by postgresql.org (Postfix) with ESMTP id CCB972E032A for ; Wed, 26 Dec 2007 14:14:07 -0400 (AST) Received: from westnet.com (localhost [127.0.0.1]) by westnet.com (8.14.0/8.14.0) with ESMTP id lBQIDvEG015216; Wed, 26 Dec 2007 13:13:57 -0500 (EST) Received: from localhost (gsmith@localhost) by westnet.com (8.14.0/8.13.2/Submit) with ESMTP id lBQIDu3i015211; Wed, 26 Dec 2007 13:13:57 -0500 (EST) X-Authentication-Warning: westnet.com: gsmith owned process doing -bs Date: Wed, 26 Dec 2007 13:13:56 -0500 (EST) From: Greg Smith X-X-Sender: gsmith@westnet.com To: Mark Mielke cc: Fernando Hevia , "'pgsql-performance'" Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 In-Reply-To: <47728000.4000709@mark.mielke.cc> Message-ID: References: <0ea101c847d2$16f94bc0$8f01010a@iptel.com.ar> <47728000.4000709@mark.mielke.cc> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/364 X-Sequence-Number: 28180 On Wed, 26 Dec 2007, Mark Mielke wrote: > I believe hardware RAID 5 is also horrible, but since the hardware hides > it from the application, a hardware RAID 5 user might not care. Typically anything doing hardware RAID 5 also has a reasonable sized write cache on the controller, which softens the problem a bit. As soon as you exceed what it can buffer you're back to suffering again. -- * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD From fhevia@ip-tel.com.ar Wed Dec 26 19:28:10 2007 Received: from localhost (unknown [200.46.204.183]) by postgresql.org (Postfix) with ESMTP id C9C2B2E0082 for ; Wed, 26 Dec 2007 15:28:10 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 32861-01 for ; Wed, 26 Dec 2007 15:28:07 -0400 (AST) Received: from pyxis.powervt.com.ar (host3.powervt.com.ar [200.43.63.66]) by postgresql.org (Postfix) with ESMTP id 81ED42E0096 for ; Wed, 26 Dec 2007 15:28:07 -0400 (AST) Received: from pcfhevia ([201.216.249.98]) by pyxis.powervt.com.ar (IPGroup Mail Server) with ESMTP id 1OT48953; Wed, 26 Dec 2007 16:28:06 -0300 From: "Fernando Hevia" To: "'Bill Moran'" Cc: "'pgsql-performance'" Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 Date: Wed, 26 Dec 2007 16:28:25 -0300 Organization: IP-TEL Message-ID: <0f8101c847f5$7c665520$8f01010a@iptel.com.ar> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 11 In-Reply-To: <20071226102105.57d9b502.wmoran@collaborativefusion.com> Thread-Index: AchH0zrJcx4XrKlqQOW3TLWQALQJygAHuS3Q X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3198 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/366 X-Sequence-Number: 28182 > Bill Moran wrote: > > RAID 10. > > I snipped the rest of your message because none of it matters. Never use > RAID 5 on a database system. Ever. There is absolutely NO reason to > every put yourself through that much suffering. If you hate yourself > that much just commit suicide, it's less drastic. > Well, that's a pretty strong argument. No suicide in my plans, gonna stick to RAID 10. :) Thanks. From fhevia@ip-tel.com.ar Wed Dec 26 19:32:06 2007 Received: from localhost (unknown [200.46.204.182]) by postgresql.org (Postfix) with ESMTP id 37A2E2E0239 for ; Wed, 26 Dec 2007 15:32:06 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.182]) (amavisd-maia, port 10024) with ESMTP id 63887-03 for ; Wed, 26 Dec 2007 15:31:55 -0400 (AST) Received: from pyxis.powervt.com.ar (host6.powervt.com.ar [200.43.63.66]) by postgresql.org (Postfix) with ESMTP id C0C092E2F7B for ; Wed, 26 Dec 2007 15:32:02 -0400 (AST) Received: from pcfhevia ([201.216.249.98]) by pyxis.powervt.com.ar (IPGroup Mail Server) with ESMTP id 1OT48953; Wed, 26 Dec 2007 16:32:01 -0300 From: "Fernando Hevia" To: "'Mark Mielke'" Cc: "'pgsql-performance'" Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 Date: Wed, 26 Dec 2007 16:32:21 -0300 Organization: IP-TEL Message-ID: <0f8201c847f6$08e15090$8f01010a@iptel.com.ar> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 11 In-Reply-To: <47728000.4000709@mark.mielke.cc> Thread-Index: AchH3BdWYUZsOQN3SV2flaSC9a9algAGZrtQ X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3198 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/367 X-Sequence-Number: 28183 Mark Mielke Wrote: >In my experience, software RAID 5 is horrible. Write performance can >decrease below the speed of one disk on its own, and read performance will >not be significantly more than RAID 1+0 as the number of stripes has only >increased from 2 to 3, and if reading while writing, you will not get 3X as >RAID 5 write requires at least two disks to be involved. I believe hardware >RAID 5 is also horrible, but since the hardware hides it from the >application, a hardware RAID 5 user might not care. >Software RAID 1+0 works fine on Linux with 4 disks. This is the setup I use >for my personal server. I will use software RAID so RAID 1+0 seems to be the obvious choice. Thanks for the advice! From fhevia@ip-tel.com.ar Wed Dec 26 20:52:13 2007 Received: from localhost (unknown [200.46.204.184]) by postgresql.org (Postfix) with ESMTP id F02BC2E0189 for ; Wed, 26 Dec 2007 16:52:13 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.184]) (amavisd-maia, port 10024) with ESMTP id 80193-04 for ; Wed, 26 Dec 2007 16:52:05 -0400 (AST) Received: from pyxis.powervt.com.ar (host3.powervt.com.ar [200.43.63.66]) by postgresql.org (Postfix) with ESMTP id 13B3B2E018B for ; Wed, 26 Dec 2007 16:52:08 -0400 (AST) Received: from pcfhevia ([201.216.249.98]) by pyxis.powervt.com.ar (IPGroup Mail Server) with ESMTP id 1OT48953 for ; Wed, 26 Dec 2007 17:52:07 -0300 From: "Fernando Hevia" To: "'pgsql-performance'" Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 Date: Wed, 26 Dec 2007 17:52:27 -0300 Organization: IP-TEL Message-ID: <0f9701c84801$399dcff0$8f01010a@iptel.com.ar> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 11 In-Reply-To: Thread-Index: AchH/DEIgfg1TyClQfihy8y3+RKyKAAAjdLA X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3198 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/370 X-Sequence-Number: 28186 > David Lang Wrote: > > with only four drives the space difference between raid 1+0 and raid 5 > isn't that much, but when you do a write you must write to two drives (the > drive holding the data you are changing, and the drive that holds the > parity data for that stripe, possibly needing to read the old parity data > first, resulting in stalling for seek/read/calculate/seek/write since > the drive moves on after the read), when you read you must read _all_ > drives in the set to check the data integrity. Thanks for the explanation David. It's good to know not only what but also why. Still I wonder why reads do hit all drives. Shouldn't only 2 disks be read: the one with the data and the parity disk? > > for seek heavy workloads (which almost every database application is) the > extra seeks involved can be murder on your performance. if your workload > is large sequential reads/writes, and you can let the OS buffer things for > you, the performance of raid 5 is much better. Well, actually most of my application involves large sequential reads/writes. The memory available for buffering (4GB) isn't bad either, at least for my scenario. On the other hand I have got such strong posts against RAID 5 that I doubt to even consider it. > > Linux software raid can do more then two disks in a mirror, so you may be > able to get the added protection with raid 1 sets (again, probably not > relavent to four drives), although there were bugs in this within the last > six months or so, so you need to be sure your kernel is new enough to have > the fix. > Well, here rises another doubt. Should I go for a single RAID 1+0 storing OS + Data + WAL files or will I be better off with two RAID 1 separating data from OS + Wal files? > now, if you can afford solid-state drives which don't have noticable seek > times, things are completely different ;-) Ha, sadly budget is very tight. :) Regards, Fernando. From fw@deneb.enyo.de Wed Dec 26 20:56:02 2007 Received: from localhost (unknown [200.46.204.187]) by postgresql.org (Postfix) with ESMTP id 24A4A2E01AE for ; Wed, 26 Dec 2007 16:56:02 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.187]) (amavisd-maia, port 10024) with ESMTP id 68600-04 for ; Wed, 26 Dec 2007 16:55:54 -0400 (AST) Received: from mail.enyo.de (mail.enyo.de [212.9.189.167]) by postgresql.org (Postfix) with ESMTP id 57C722E016C for ; Wed, 26 Dec 2007 16:55:57 -0400 (AST) Received: from deneb.vpn.enyo.de ([212.9.189.177] helo=deneb.enyo.de) by mail.enyo.de with esmtp id 1J7dHy-00036g-4J; Wed, 26 Dec 2007 21:55:42 +0100 Received: from fw by deneb.enyo.de with local (Exim 4.68) (envelope-from ) id 1J7dHu-00023K-IG; Wed, 26 Dec 2007 21:55:38 +0100 From: Florian Weimer To: david@lang.hm Cc: Fernando Hevia , "'Mark Mielke'" , "'pgsql-performance'" Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 References: <0f8201c847f6$08e15090$8f01010a@iptel.com.ar> Date: Wed, 26 Dec 2007 21:55:38 +0100 In-Reply-To: (david@lang.hm's message of "Wed, 26 Dec 2007 13:28:12 -0800 (PST)") Message-ID: <87zlvxnpkl.fsf@mid.deneb.enyo.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/372 X-Sequence-Number: 28188 > seek/read/calculate/seek/write since the drive moves on after the > read), when you read you must read _all_ drives in the set to check > the data integrity. I don't know of any RAID implementation that performs consistency checking on each read operation. 8-( From wmoran@collaborativefusion.com Wed Dec 26 21:14:41 2007 Received: from localhost (unknown [200.46.204.187]) by postgresql.org (Postfix) with ESMTP id 74C9E2E018C for ; Wed, 26 Dec 2007 17:14:41 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.187]) (amavisd-maia, port 10024) with ESMTP id 73240-07 for ; Wed, 26 Dec 2007 17:14:29 -0400 (AST) Received: from mx00.pub.collaborativefusion.com (mx00.pub.collaborativefusion.com [206.210.89.199]) by postgresql.org (Postfix) with ESMTP id 41F4A2E00FA for ; Wed, 26 Dec 2007 17:14:32 -0400 (AST) Received: from vanquish.pitbpa0.priv.collaborativefusion.com (vanquish.pitbpa0.priv.collaborativefusion.com [192.168.2.61]) (SSL: TLSv1/SSLv3,256bits,AES256-SHA) by wingspan with esmtp; Wed, 26 Dec 2007 16:14:30 -0500 id 00056410.4772C436.00013258 Date: Wed, 26 Dec 2007 16:14:30 -0500 From: Bill Moran To: "Fernando Hevia" Cc: "'pgsql-performance'" Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 Message-Id: <20071226161430.72ef5d31.wmoran@collaborativefusion.com> In-Reply-To: <0f9701c84801$399dcff0$8f01010a@iptel.com.ar> References: <0f9701c84801$399dcff0$8f01010a@iptel.com.ar> Organization: Collaborative Fusion X-Mailer: Sylpheed 2.4.7 (GTK+ 2.12.1; i386-portbld-freebsd6.2) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/374 X-Sequence-Number: 28190 In response to "Fernando Hevia" : > > > David Lang Wrote: > > > > with only four drives the space difference between raid 1+0 and raid 5 > > isn't that much, but when you do a write you must write to two drives (the > > drive holding the data you are changing, and the drive that holds the > > parity data for that stripe, possibly needing to read the old parity data > > first, resulting in stalling for seek/read/calculate/seek/write since > > the drive moves on after the read), when you read you must read _all_ > > drives in the set to check the data integrity. > > Thanks for the explanation David. It's good to know not only what but also > why. Still I wonder why reads do hit all drives. Shouldn't only 2 disks be > read: the one with the data and the parity disk? In order to recalculate the parity, it has to have data from all disks. Thus, if you have 4 disks, it has to read 2 (the unknown data blocks included in the parity calculation) then write 2 (the new data block and the new parity data) Caching can help some, but if your data ends up being any size at all, the cache misses become more frequent than the hits. Even when caching helps, you max speed is still only the speed of a single disk. > > for seek heavy workloads (which almost every database application is) the > > extra seeks involved can be murder on your performance. if your workload > > is large sequential reads/writes, and you can let the OS buffer things for > > you, the performance of raid 5 is much better. > > Well, actually most of my application involves large sequential > reads/writes. Will it? Will you be deleting or updating data? If so, you'll generate dead tuples, which vacuum will have to clean up, which means seeks, and means you new data isn't liable to be sequentially written. The chance that you actually have a workload that will result in consistently sequential writes at the disk level is very slim, in my experience. When vacuum is taking hours and hours, you'll understand the pain. > The memory available for buffering (4GB) isn't bad either, at > least for my scenario. On the other hand I have got such strong posts > against RAID 5 that I doubt to even consider it. If 4G is enough to buffer all your data, then why do you need the extra space of RAID 5? If you need the extra space of the RAID 5, then 4G isn't enough to buffer all your data, and that buffer will be of limited usefulness. In any event, even if you've got 300G of RAM to buffer data in, sooner or later you've got to write it to disk, and no matter how much RAM you have, your write speed will be limited by how fast your disks can commit. If you had a database multiple petabytes in size, you could worry about needing the extra space that RAID 5 gives you, but then you'd realize that the speed problems associated with RAID 5 will make a petabyte sized database completely unmanageable. There's just no scenario where RAID 5 is a win for database work. Period. Rationalize all you want. For those trying to defend RAID 5, I invite you to try it. When you're on the verge of suicide because you can't get any work done, don't say I didn't say so. > Well, here rises another doubt. Should I go for a single RAID 1+0 storing OS > + Data + WAL files or will I be better off with two RAID 1 separating data > from OS + Wal files? Generally speaking, if you want the absolute best performance, it's generally recommended to keep the WAL logs on one partition/controller and the remaining database files on a second one. However, with only 4 disks, you might get just as much out of a RAID 1+0. > > now, if you can afford solid-state drives which don't have noticable seek > > times, things are completely different ;-) > > Ha, sadly budget is very tight. :) Budget is always tight. That's why you don't want a RAID 5. Do a RAID 5 now thinking you'll save a few bucks, and you'll be spending twice that much later trying to fix your mistake. It's called tripping over a dime to pick up a nickel. -- Bill Moran Collaborative Fusion Inc. http://people.collaborativefusion.com/~wmoran/ wmoran@collaborativefusion.com Phone: 412-422-3463x4023 From david@lang.hm Wed Dec 26 20:15:52 2007 Received: from localhost (unknown [200.46.204.183]) by postgresql.org (Postfix) with ESMTP id AD6C12E2EE2 for ; Wed, 26 Dec 2007 16:15:52 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 46903-06 for ; Wed, 26 Dec 2007 16:15:49 -0400 (AST) Received: from bifrost.lang.hm (dsl081-033-126.lax1.dsl.speakeasy.net [64.81.33.126]) by postgresql.org (Postfix) with ESMTP id C85BC2E2EA4 for ; Wed, 26 Dec 2007 16:15:49 -0400 (AST) Received: from asgard.lang.hm (asgard.lang.hm [10.0.0.100]) by bifrost.lang.hm (8.13.4/8.13.4/Debian-3) with ESMTP id lBQKFepL022802; Wed, 26 Dec 2007 12:15:40 -0800 Date: Wed, 26 Dec 2007 13:28:12 -0800 (PST) From: david@lang.hm X-X-Sender: dlang@asgard.lang.hm To: Fernando Hevia cc: "'Mark Mielke'" , "'pgsql-performance'" Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 In-Reply-To: <0f8201c847f6$08e15090$8f01010a@iptel.com.ar> Message-ID: References: <0f8201c847f6$08e15090$8f01010a@iptel.com.ar> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/368 X-Sequence-Number: 28184 On Wed, 26 Dec 2007, Fernando Hevia wrote: > Mark Mielke Wrote: > >> In my experience, software RAID 5 is horrible. Write performance can >> decrease below the speed of one disk on its own, and read performance will >> not be significantly more than RAID 1+0 as the number of stripes has only >> increased from 2 to 3, and if reading while writing, you will not get 3X as >> RAID 5 write requires at least two disks to be involved. I believe hardware >> RAID 5 is also horrible, but since the hardware hides it from the >> application, a hardware RAID 5 user might not care. > >> Software RAID 1+0 works fine on Linux with 4 disks. This is the setup I use >> for my personal server. > > I will use software RAID so RAID 1+0 seems to be the obvious choice. > Thanks for the advice! to clarify things a bit more. with only four drives the space difference between raid 1+0 and raid 5 isn't that much, but when you do a write you must write to two drives (the drive holding the data you are changing, and the drive that holds the parity data for that stripe, possibly needing to read the old parity data first, resulting in stalling for seek/read/calculate/seek/write since the drive moves on after the read), when you read you must read _all_ drives in the set to check the data integrity. for seek heavy workloads (which almost every database application is) the extra seeks involved can be murder on your performance. if your workload is large sequential reads/writes, and you can let the OS buffer things for you, the performance of raid 5 is much better. on the other hand, doing raid 6 (instead of raid 5) gives you extra data protection in exchange for the performance hit, but with only 4 drives this probably isn't what you are looking for. Linux software raid can do more then two disks in a mirror, so you may be able to get the added protection with raid 1 sets (again, probably not relavent to four drives), although there were bugs in this within the last six months or so, so you need to be sure your kernel is new enough to have the fix. now, if you can afford solid-state drives which don't have noticable seek times, things are completely different ;-) David Lang From mark@mark.mielke.cc Wed Dec 26 21:43:52 2007 Received: from localhost (unknown [200.46.204.191]) by postgresql.org (Postfix) with ESMTP id 589022E2EC2 for ; Wed, 26 Dec 2007 17:43:52 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.191]) (amavisd-maia, port 10024) with ESMTP id 81301-04 for ; Wed, 26 Dec 2007 17:43:48 -0400 (AST) Received: from ironport2-out.pppoe.ca (ironport2-out.pppoe.ca [206.248.154.182]) by postgresql.org (Postfix) with ESMTP id 70A402E2EC0 for ; Wed, 26 Dec 2007 17:43:48 -0400 (AST) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ao8CALNZckfO+Igp/2dsb2JhbACCb6Zd X-IronPort-AV: E=Sophos;i="4.24,209,1196658000"; d="scan'208,217";a="11988430" Received: from smtp.pppoe.ca ([65.39.196.238]) by ironport2-out.pppoe.ca with ESMTP; 26 Dec 2007 16:43:47 -0500 Received: from [206.248.136.41] ([206.248.136.41]) by smtp.pppoe.ca (Internet Mail Server v1.0) with ESMTP id FHR97646; Wed, 26 Dec 2007 16:43:46 -0500 Message-ID: <4772CA58.5050406@mark.mielke.cc> Date: Wed, 26 Dec 2007 16:40:40 -0500 From: Mark Mielke User-Agent: Thunderbird 2.0.0.9 (X11/20071115) MIME-Version: 1.0 To: Florian Weimer CC: david@lang.hm, Fernando Hevia , 'pgsql-performance' Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 References: <0f8201c847f6$08e15090$8f01010a@iptel.com.ar> <87zlvxnpkl.fsf@mid.deneb.enyo.de> In-Reply-To: <87zlvxnpkl.fsf@mid.deneb.enyo.de> Content-Type: multipart/alternative; boundary="------------060408010307090801050105" X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/378 X-Sequence-Number: 28194 This is a multi-part message in MIME format. --------------060408010307090801050105 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Florian Weimer wrote: >> seek/read/calculate/seek/write since the drive moves on after the >> read), when you read you must read _all_ drives in the set to check >> the data integrity. >> > I don't know of any RAID implementation that performs consistency > checking on each read operation. 8-( > Dave had too much egg nog... :-) Yep - checking consistency on read would eliminate the performance benefits of RAID under any redundant configuration. Cheers, mark -- Mark Mielke --------------060408010307090801050105 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Florian Weimer wrote: 
seek/read/calculate/seek/write since the drive moves on after the
read), when you read you must read _all_ drives in the set to check
the data integrity.
    
I don't know of any RAID implementation that performs consistency
checking on each read operation. 8-(

Dave had too much egg nog... :-)

Yep - checking consistency on read would eliminate the performance benefits of RAID under any redundant configuration.

Cheers,
mark

-- 
Mark Mielke <mark@mielke.cc>
--------------060408010307090801050105-- From mark@mark.mielke.cc Wed Dec 26 21:57:12 2007 Received: from localhost (unknown [200.46.204.182]) by postgresql.org (Postfix) with ESMTP id D4B8C2E2EA3 for ; Wed, 26 Dec 2007 17:57:12 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.182]) (amavisd-maia, port 10024) with ESMTP id 63710-08 for ; Wed, 26 Dec 2007 17:57:05 -0400 (AST) Received: from ironport2-out.pppoe.ca (ironport2-out.pppoe.ca [206.248.154.182]) by postgresql.org (Postfix) with ESMTP id 8187A2E2F28 for ; Wed, 26 Dec 2007 17:57:07 -0400 (AST) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ao8CADZdckfO+Igp/2dsb2JhbACpRg X-IronPort-AV: E=Sophos;i="4.24,209,1196658000"; d="scan'208";a="11988861" Received: from smtp.pppoe.ca ([65.39.196.238]) by ironport2-out.pppoe.ca with ESMTP; 26 Dec 2007 16:57:06 -0500 Received: from [206.248.136.41] ([206.248.136.41]) by smtp.pppoe.ca (Internet Mail Server v1.0) with ESMTP id FHF39806; Wed, 26 Dec 2007 16:57:06 -0500 Message-ID: <4772CD78.4010302@mark.mielke.cc> Date: Wed, 26 Dec 2007 16:54:00 -0500 From: Mark Mielke User-Agent: Thunderbird 2.0.0.9 (X11/20071115) MIME-Version: 1.0 To: david@lang.hm CC: Fernando Hevia , 'pgsql-performance' Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 References: <0f9701c84801$399dcff0$8f01010a@iptel.com.ar> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/382 X-Sequence-Number: 28198 david@lang.hm wrote: >> Thanks for the explanation David. It's good to know not only what but >> also >> why. Still I wonder why reads do hit all drives. Shouldn't only 2 >> disks be >> read: the one with the data and the parity disk? > no, becouse the parity is of the sort (A+B+C+P) mod X = 0 > so if X=10 (which means in practice that only the last decimal digit > of anything matters, very convienient for examples) > A=1, B=2, C=3, A+B+C=6, P=4, A+B+C+P=10=0 > if you read B and get 3 and P and get 4 you don't know if this is > right or not unless you also read A and C (at which point you would > get A+B+C+P=11=1=error) I don't think this is correct. RAID 5 is parity which is XOR. The property of XOR is such that it doesn't matter what the other drives are. You can write any block given either: 1) The block you are overwriting and the parity, or 2) all other blocks except for the block we are writing and the parity. Now, it might be possible that option 2) is taken more than option 1) for some complicated reasons, but it is NOT to check consistency. The array is assumed consistent until proven otherwise. > in theory a system could get the same performance with a large > sequential read/write on raid5/6 as on a raid0 array of equivilent > size (i.e. same number of data disks, ignoring the parity disks) > becouse the OS could read the entire stripe in at once, do the > calculation once, and use all the data (or when writing, don't write > anything until you are ready to write the entire stripe, calculate the > parity and write everything once). For the same number of drives, this cannot be possible. With 10 disks, on raid5, 9 disks hold data, and 1 holds parity. The theoretical maximum performance is only 9/10 of the 10/10 performance possible with RAID 0. > Unfortunantly in practice filesystems don't support this, they don't > do enough readahead to want to keep the entire stripe (so after they > read it all in they throw some of it away), they (mostly) don't know > where a stripe starts (and so intermingle different types of data on > one stripe and spread data across multiple stripes unessasarily), and > they tend to do writes in small, scattered chunks (rather then > flushing an entire stripes worth of data at once) In my experience, this theoretical maximum is not attainable without significant write cache, and an intelligent controller, neither of which Linux software RAID seems to have by default. My situation was a bit worse in that I used applications that fsync() or journalled metadata that is ordered, which forces the Linux software RAID to flush far more than it should - but the same system works very well with RAID 1+0. >>> Linux software raid can do more then two disks in a mirror, so you >>> may be >>> able to get the added protection with raid 1 sets (again, probably not >>> relavent to four drives), although there were bugs in this within >>> the last >>> six months or so, so you need to be sure your kernel is new enough >>> to have >>> the fix. >>> >> Well, here rises another doubt. Should I go for a single RAID 1+0 >> storing OS >> + Data + WAL files or will I be better off with two RAID 1 separating >> data >> from OS + Wal files? > if you can afford the space, you are almost certinly better seperating > the WAL from the data (I think I've seen debates about which is better > OS+data/Wal or date/OS+Wal, but very little disagreement that either > is better than combining them all) I don't think there is a good answer for this question. If you can afford my drives, you could also afford to make your RAID 1+0 bigger. Splitting OS/DATA/WAL is only "absolute best" if can arrange your 3 arrays such that there size is relative to their access patterns. For example, in an overly simplified case, if you use OS 1/4 of DATA, and WAL 1/2 of DATA, then perhaps "best" is to have a two-disk RAID 1 for OS, a four-disk RAID 1+0 for WAL, and an eight-disk RAID 1+0 for DATA. This gives a total of 14 disks. :-) In practice, if you have four drives, and you try and it into two plus two, you're going to find that two of the drives are going to be more idle than the other two. I have a fun setup - I use RAID 1 across all four drives for the OS, RAID 1+0 for the database, wal, and other parts, and RAID 0 for a "build" partition. :-) Cheers, mark -- Mark Mielke From mark@mark.mielke.cc Wed Dec 26 22:00:25 2007 Received: from localhost (unknown [200.46.204.187]) by postgresql.org (Postfix) with ESMTP id EC1E72E2F62 for ; Wed, 26 Dec 2007 18:00:25 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.187]) (amavisd-maia, port 10024) with ESMTP id 89320-01 for ; Wed, 26 Dec 2007 18:00:17 -0400 (AST) Received: from ironport2-out.pppoe.ca (ironport2-out.pppoe.ca [206.248.154.182]) by postgresql.org (Postfix) with ESMTP id 951F62E2F65 for ; Wed, 26 Dec 2007 18:00:20 -0400 (AST) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ao8CAGNeckfO+Igp/2dsb2JhbACpSA X-IronPort-AV: E=Sophos;i="4.24,209,1196658000"; d="scan'208";a="11988978" Received: from smtp.pppoe.ca ([65.39.196.238]) by ironport2-out.pppoe.ca with ESMTP; 26 Dec 2007 17:00:15 -0500 Received: from [206.248.136.41] ([206.248.136.41]) by smtp.pppoe.ca (Internet Mail Server v1.0) with ESMTP id FIA71014; Wed, 26 Dec 2007 17:00:14 -0500 Message-ID: <4772CE34.2040608@mark.mielke.cc> Date: Wed, 26 Dec 2007 16:57:08 -0500 From: Mark Mielke User-Agent: Thunderbird 2.0.0.9 (X11/20071115) MIME-Version: 1.0 To: david@lang.hm CC: Florian Weimer , Fernando Hevia , 'pgsql-performance' Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 References: <0f8201c847f6$08e15090$8f01010a@iptel.com.ar> <87zlvxnpkl.fsf@mid.deneb.enyo.de> <4772CA58.5050406@mark.mielke.cc> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/383 X-Sequence-Number: 28199 david@lang.hm wrote: > On Wed, 26 Dec 2007, Mark Mielke wrote: > >> Florian Weimer wrote: >>>> seek/read/calculate/seek/write since the drive moves on after the >>>> read), when you read you must read _all_ drives in the set to check >>>> the data integrity. >>> I don't know of any RAID implementation that performs consistency >>> checking on each read operation. 8-( >> Dave had too much egg nog... :-) >> Yep - checking consistency on read would eliminate the performance >> benefits of RAID under any redundant configuration. > except for raid0, raid is primarily a reliability benifit, any > performance benifit is incidental, not the primary purpose. > that said, I have heard of raid1 setups where it only reads off of one > of the drives, but I have not heard of higher raid levels doing so. What do you mean "heard of"? Which raid system do you know of that reads all drives for RAID 1? Linux dmraid reads off ONLY the first. Linux mdadm reads off the "best" one. Neither read from both. Why should it need to read from both? What will it do if the consistency check fails? It's not like it can tell which disk is the right one. It only knows that the whole array is inconsistent. Until it gets an actual hardware failure (read error, write error), it doesn't know which disk is wrong. Cheers, mark -- Mark Mielke From mark@mark.mielke.cc Wed Dec 26 22:02:59 2007 Received: from localhost (unknown [200.46.204.182]) by postgresql.org (Postfix) with ESMTP id 834BA2E2F8E for ; Wed, 26 Dec 2007 18:02:59 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.182]) (amavisd-maia, port 10024) with ESMTP id 69216-05 for ; Wed, 26 Dec 2007 18:02:50 -0400 (AST) Received: from ironport2-out.pppoe.ca (ironport2-out.pppoe.ca [206.248.154.182]) by postgresql.org (Postfix) with ESMTP id C47D12E2F88 for ; Wed, 26 Dec 2007 18:02:52 -0400 (AST) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ao8CAGNeckfO+Igp/2dsb2JhbACpSA X-IronPort-AV: E=Sophos;i="4.24,209,1196658000"; d="scan'208";a="11989071" Received: from smtp.pppoe.ca ([65.39.196.238]) by ironport2-out.pppoe.ca with ESMTP; 26 Dec 2007 17:02:52 -0500 Received: from [206.248.136.41] ([206.248.136.41]) by smtp.pppoe.ca (Internet Mail Server v1.0) with ESMTP id FIC80451; Wed, 26 Dec 2007 17:02:51 -0500 Message-ID: <4772CED1.2070808@mark.mielke.cc> Date: Wed, 26 Dec 2007 16:59:45 -0500 From: Mark Mielke User-Agent: Thunderbird 2.0.0.9 (X11/20071115) MIME-Version: 1.0 To: david@lang.hm CC: Florian Weimer , Fernando Hevia , 'pgsql-performance' Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 References: <0f8201c847f6$08e15090$8f01010a@iptel.com.ar> <87zlvxnpkl.fsf@mid.deneb.enyo.de> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/384 X-Sequence-Number: 28200 david@lang.hm wrote: > I could see a raid 1 array not doing consistancy checking (after all, > it has no way of knowing what's right if it finds an error), but since > raid 5/6 can repair the data I would expect them to do the checking > each time. Your messages are spread across the thread. :-) RAID 5 cannot repair the data. I don't know much about RAID 6, but I expect it cannot necessarily repair the data either. It still doesn't know which drive is wrong. In any case, there is no implementation I am aware of that performs mandatory consistency checks on read. This would be silliness. Cheers, mark -- Mark Mielke From mark@mark.mielke.cc Wed Dec 26 22:09:05 2007 Received: from localhost (unknown [200.46.204.191]) by postgresql.org (Postfix) with ESMTP id C43D62E3198 for ; Wed, 26 Dec 2007 18:09:05 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.191]) (amavisd-maia, port 10024) with ESMTP id 90585-02 for ; Wed, 26 Dec 2007 18:09:02 -0400 (AST) Received: from ironport2-out.pppoe.ca (ironport2-out.pppoe.ca [206.248.154.182]) by postgresql.org (Postfix) with ESMTP id 2D73E2E3180 for ; Wed, 26 Dec 2007 18:09:02 -0400 (AST) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ao8CAI5fckfO+Igp/2dsb2JhbACpSg X-IronPort-AV: E=Sophos;i="4.24,209,1196658000"; d="scan'208";a="11989254" Received: from smtp.pppoe.ca ([65.39.196.238]) by ironport2-out.pppoe.ca with ESMTP; 26 Dec 2007 17:09:01 -0500 Received: from [206.248.136.41] ([206.248.136.41]) by smtp.pppoe.ca (Internet Mail Server v1.0) with ESMTP id FIJ85100; Wed, 26 Dec 2007 17:09:00 -0500 Message-ID: <4772D042.8090402@mark.mielke.cc> Date: Wed, 26 Dec 2007 17:05:54 -0500 From: Mark Mielke User-Agent: Thunderbird 2.0.0.9 (X11/20071115) MIME-Version: 1.0 To: Bill Moran CC: Fernando Hevia , 'pgsql-performance' Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 References: <0f9701c84801$399dcff0$8f01010a@iptel.com.ar> <20071226161430.72ef5d31.wmoran@collaborativefusion.com> In-Reply-To: <20071226161430.72ef5d31.wmoran@collaborativefusion.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/385 X-Sequence-Number: 28201 Bill Moran wrote: > In order to recalculate the parity, it has to have data from all disks. Thus, > if you have 4 disks, it has to read 2 (the unknown data blocks included in > the parity calculation) then write 2 (the new data block and the new > parity data) Caching can help some, but if your data ends up being any > size at all, the cache misses become more frequent than the hits. Even > when caching helps, you max speed is still only the speed of a single > disk. > If you have 4 disks, it can do either: 1) Read the old block, read the parity block, XOR the old block with the parity block and the new block resulting in the new parity block, write both the new parity block and the new block. 2) Read the two unknown blocks, XOR with the new block resulting in the new parity block, write both the new parity block and the new block. You are emphasizing 2 - but the scenario is also overly simplistic. Imagine you had 10 drives on RAID 5. Would it make more sense to read 8 blocks and then write two (option 2, and the one you describe), or read two blocks and then write two (option 1). Obviously, if option 1 or option 2 can be satisfied from cache, it is better to not read at all. I note that you also disagree with Dave, in that you are not claiming it performs consistency checks on read. No system does this as performance would go to the crapper. Cheers, mark -- Mark Mielke From wmoran@collaborativefusion.com Wed Dec 26 22:11:14 2007 Received: from localhost (unknown [200.46.204.187]) by postgresql.org (Postfix) with ESMTP id 38E7E2E31C2 for ; Wed, 26 Dec 2007 18:11:14 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.187]) (amavisd-maia, port 10024) with ESMTP id 91222-03 for ; Wed, 26 Dec 2007 18:11:03 -0400 (AST) Received: from mx00.pub.collaborativefusion.com (mx00.pub.collaborativefusion.com [206.210.89.199]) by postgresql.org (Postfix) with ESMTP id E06B62E31DB for ; Wed, 26 Dec 2007 18:11:06 -0400 (AST) Received: from vanquish.pitbpa0.priv.collaborativefusion.com (vanquish.pitbpa0.priv.collaborativefusion.com [192.168.2.61]) (SSL: TLSv1/SSLv3,256bits,AES256-SHA) by wingspan with esmtp; Wed, 26 Dec 2007 17:11:05 -0500 id 00056405.4772D179.0001380A Date: Wed, 26 Dec 2007 17:11:05 -0500 From: Bill Moran To: Mark Mielke Cc: david@lang.hm, Florian Weimer , Fernando Hevia , 'pgsql-performance' Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 Message-Id: <20071226171105.889f6146.wmoran@collaborativefusion.com> In-Reply-To: <4772CE34.2040608@mark.mielke.cc> References: <0f8201c847f6$08e15090$8f01010a@iptel.com.ar> <87zlvxnpkl.fsf@mid.deneb.enyo.de> <4772CA58.5050406@mark.mielke.cc> <4772CE34.2040608@mark.mielke.cc> Organization: Collaborative Fusion X-Mailer: Sylpheed 2.4.7 (GTK+ 2.12.1; i386-portbld-freebsd6.2) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/386 X-Sequence-Number: 28202 In response to Mark Mielke : > david@lang.hm wrote: > > On Wed, 26 Dec 2007, Mark Mielke wrote: > > > >> Florian Weimer wrote: > >>>> seek/read/calculate/seek/write since the drive moves on after the > >>>> read), when you read you must read _all_ drives in the set to check > >>>> the data integrity. > >>> I don't know of any RAID implementation that performs consistency > >>> checking on each read operation. 8-( > >> Dave had too much egg nog... :-) > >> Yep - checking consistency on read would eliminate the performance > >> benefits of RAID under any redundant configuration. > > except for raid0, raid is primarily a reliability benifit, any > > performance benifit is incidental, not the primary purpose. > > that said, I have heard of raid1 setups where it only reads off of one > > of the drives, but I have not heard of higher raid levels doing so. > What do you mean "heard of"? Which raid system do you know of that reads > all drives for RAID 1? I'm fairly sure that FreeBSD's GEOM does. Of course, it couldn't be doing consistency checking at that point. -- Bill Moran Collaborative Fusion Inc. http://people.collaborativefusion.com/~wmoran/ wmoran@collaborativefusion.com Phone: 412-422-3463x4023 From wmoran@collaborativefusion.com Wed Dec 26 22:16:17 2007 Received: from localhost (unknown [200.46.204.182]) by postgresql.org (Postfix) with ESMTP id 7B84A2E31E9 for ; Wed, 26 Dec 2007 18:16:17 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.182]) (amavisd-maia, port 10024) with ESMTP id 11772-04-4 for ; Wed, 26 Dec 2007 18:16:09 -0400 (AST) Received: from mx00.pub.collaborativefusion.com (mx00.pub.collaborativefusion.com [206.210.89.199]) by postgresql.org (Postfix) with ESMTP id B20532E327E for ; Wed, 26 Dec 2007 18:16:09 -0400 (AST) Received: from vanquish.pitbpa0.priv.collaborativefusion.com (vanquish.pitbpa0.priv.collaborativefusion.com [192.168.2.61]) (SSL: TLSv1/SSLv3,256bits,AES256-SHA) by wingspan with esmtp; Wed, 26 Dec 2007 17:16:08 -0500 id 00056422.4772D2A8.00013942 Date: Wed, 26 Dec 2007 17:16:08 -0500 From: Bill Moran To: Mark Mielke Cc: Fernando Hevia , 'pgsql-performance' Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 Message-Id: <20071226171608.7fd6e625.wmoran@collaborativefusion.com> In-Reply-To: <4772D042.8090402@mark.mielke.cc> References: <0f9701c84801$399dcff0$8f01010a@iptel.com.ar> <20071226161430.72ef5d31.wmoran@collaborativefusion.com> <4772D042.8090402@mark.mielke.cc> Organization: Collaborative Fusion X-Mailer: Sylpheed 2.4.7 (GTK+ 2.12.1; i386-portbld-freebsd6.2) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/387 X-Sequence-Number: 28203 In response to Mark Mielke : > Bill Moran wrote: > > In order to recalculate the parity, it has to have data from all disks. Thus, > > if you have 4 disks, it has to read 2 (the unknown data blocks included in > > the parity calculation) then write 2 (the new data block and the new > > parity data) Caching can help some, but if your data ends up being any > > size at all, the cache misses become more frequent than the hits. Even > > when caching helps, you max speed is still only the speed of a single > > disk. > > > If you have 4 disks, it can do either: > > 1) Read the old block, read the parity block, XOR the old block with > the parity block and the new block resulting in the new parity block, > write both the new parity block and the new block. > 2) Read the two unknown blocks, XOR with the new block resulting in > the new parity block, write both the new parity block and the new block. > > You are emphasizing 2 - but the scenario is also overly simplistic. > Imagine you had 10 drives on RAID 5. Would it make more sense to read 8 > blocks and then write two (option 2, and the one you describe), or read > two blocks and then write two (option 1). Obviously, if option 1 or > option 2 can be satisfied from cache, it is better to not read at all. Good point that I wasn't aware of. > I note that you also disagree with Dave, in that you are not claiming it > performs consistency checks on read. No system does this as performance > would go to the crapper. I call straw man :) I don't disagree. I simply don't know. There's no reason why it _couldn't_ do consistency checking as it ran ... of course, performance would suck. Generally what you expect out of RAID 5|6 is that it can rebuild a drive in the event of a failure, so I doubt if anyone does consistency checking by default, and I wouldn't be surprised if a lot of systems don't have the option to do it at all. -- Bill Moran Collaborative Fusion Inc. http://people.collaborativefusion.com/~wmoran/ wmoran@collaborativefusion.com Phone: 412-422-3463x4023 From mark@mark.mielke.cc Wed Dec 26 22:24:27 2007 Received: from localhost (unknown [200.46.204.184]) by postgresql.org (Postfix) with ESMTP id 07DA82E037A for ; Wed, 26 Dec 2007 18:24:27 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.184]) (amavisd-maia, port 10024) with ESMTP id 98657-05-2 for ; Wed, 26 Dec 2007 18:24:17 -0400 (AST) Received: from ironport2-out.pppoe.ca (ironport2-out.pppoe.ca [206.248.154.182]) by postgresql.org (Postfix) with ESMTP id 3C9C32E00BD for ; Wed, 26 Dec 2007 18:24:22 -0400 (AST) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ao8CABRjckfO+Igp/2dsb2JhbACCb6Zq X-IronPort-AV: E=Sophos;i="4.24,209,1196658000"; d="scan'208,217";a="11989712" Received: from smtp.pppoe.ca ([65.39.196.238]) by ironport2-out.pppoe.ca with ESMTP; 26 Dec 2007 17:24:21 -0500 Received: from [206.248.136.41] ([206.248.136.41]) by smtp.pppoe.ca (Internet Mail Server v1.0) with ESMTP id FIY75720; Wed, 26 Dec 2007 17:24:20 -0500 Message-ID: <4772D3DA.10302@mark.mielke.cc> Date: Wed, 26 Dec 2007 17:21:14 -0500 From: Mark Mielke User-Agent: Thunderbird 2.0.0.9 (X11/20071115) MIME-Version: 1.0 To: Bill Moran CC: david@lang.hm, Florian Weimer , Fernando Hevia , 'pgsql-performance' Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 References: <0f8201c847f6$08e15090$8f01010a@iptel.com.ar> <87zlvxnpkl.fsf@mid.deneb.enyo.de> <4772CA58.5050406@mark.mielke.cc> <4772CE34.2040608@mark.mielke.cc> <20071226171105.889f6146.wmoran@collaborativefusion.com> In-Reply-To: <20071226171105.889f6146.wmoran@collaborativefusion.com> Content-Type: multipart/alternative; boundary="------------060301010805000101030407" X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/390 X-Sequence-Number: 28206 This is a multi-part message in MIME format. --------------060301010805000101030407 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Bill Moran wrote: > >> What do you mean "heard of"? Which raid system do you know of that reads >> all drives for RAID 1? >> > > I'm fairly sure that FreeBSD's GEOM does. Of course, it couldn't be doing > consistency checking at that point. > According to this: http://www.freebsd.org/cgi/man.cgi?query=gmirror&apropos=0&sektion=8&manpath=FreeBSD+6-current&format=html There is a -b (balance) option that seems pretty clear that it does not read from all drives if it does not have to: Create a mirror. The order of components is important, because a component's priority is based on its position (starting from 0). The component with the biggest priority is used by the prefer balance algorithm and is also used as a master component when resynchronization is needed, e.g. after a power failure when the device was open for writing. Additional options include: *-b* /balance/ Specifies balance algorithm to use, one of: *load* Read from the component with the lowest load. *prefer* Read from the component with the biggest priority. *round-robin* Use round-robin algorithm when choosing component to read. *split* Split read requests, which are big- ger than or equal to slice size on N pieces, where N is the number of active components. This is the default balance algorithm. Cheers, mark -- Mark Mielke --------------060301010805000101030407 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Bill Moran wrote: 

What do you mean "heard of"? Which raid system do you know of that reads 
all drives for RAID 1?
    

I'm fairly sure that FreeBSD's GEOM does.  Of course, it couldn't be doing
consistency checking at that point.
According to this:

http://www.freebsd.org/cgi/man.cgi?query=gmirror&apropos=0&sektion=8&manpath=FreeBSD+6-current&format=html

There is a -b (balance) option that seems pretty clear that it does not read from all drives if it does not have to:

    Create a mirror. 
                 The order of components is important,
                 because a component's priority is based on its position
                 (starting from 0).  The component with the biggest priority
                 is used by the prefer balance algorithm and is also used as a
                 master component when resynchronization is needed, e.g. after
                 a power failure when the device was open for writing.
    Additional options include:

                 -b balance  Specifies balance algorithm to use, one of:

                             load         Read from the component with the
                                          lowest load.

                             prefer       Read from the component with the
                                          biggest priority.

                             round-robin  Use round-robin algorithm when
                                          choosing component to read.

                             split        Split read requests, which are big-
                                          ger than or equal to slice size on N
                                          pieces, where N is the number of
                                          active components.  This is the
                                          default balance algorithm.


Cheers,
mark

-- 
Mark Mielke <mark@mielke.cc>
--------------060301010805000101030407-- From mark@mark.mielke.cc Wed Dec 26 22:36:24 2007 Received: from localhost (unknown [200.46.204.184]) by postgresql.org (Postfix) with ESMTP id 974B92E2DB8 for ; Wed, 26 Dec 2007 18:36:24 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.184]) (amavisd-maia, port 10024) with ESMTP id 00866-09-2 for ; Wed, 26 Dec 2007 18:36:09 -0400 (AST) Received: from ironport2-out.pppoe.ca (ironport2-out.pppoe.ca [206.248.154.182]) by postgresql.org (Postfix) with ESMTP id 0EA192E2DD7 for ; Wed, 26 Dec 2007 18:36:14 -0400 (AST) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ao8CAJZmckfO+Igp/2dsb2JhbACpRQ X-IronPort-AV: E=Sophos;i="4.24,209,1196658000"; d="scan'208";a="11990060" Received: from smtp.pppoe.ca ([65.39.196.238]) by ironport2-out.pppoe.ca with ESMTP; 26 Dec 2007 17:36:13 -0500 Received: from [206.248.136.41] ([206.248.136.41]) by smtp.pppoe.ca (Internet Mail Server v1.0) with ESMTP id FIK75713; Wed, 26 Dec 2007 17:36:13 -0500 Message-ID: <4772D6A3.2040208@mark.mielke.cc> Date: Wed, 26 Dec 2007 17:33:07 -0500 From: Mark Mielke User-Agent: Thunderbird 2.0.0.9 (X11/20071115) MIME-Version: 1.0 To: david@lang.hm CC: Fernando Hevia , 'pgsql-performance' Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 References: <0f9701c84801$399dcff0$8f01010a@iptel.com.ar> <4772CD78.4010302@mark.mielke.cc> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/392 X-Sequence-Number: 28208 david@lang.hm wrote: > however I was addressing the point that for reads you can't do any > checking until you have read in all the blocks. > if you never check the consistency, how will it ever be proven otherwise. A scheme often used is to mark the disk/slice as "clean" during clean system shutdown (or RAID device shutdown). When it comes back up, it is assumed clean. Why wouldn't it be clean? However, if it comes up "unclean", this does indeed require an EXPENSIVE resynchronization process. Note, however, that resynchronization usually reads or writes all disks, whether RAID 1, RAID 5, RAID 6, or RAID 1+0. My RAID 1+0 does a full resynchronization if shut down uncleanly. There is nothing specific about RAID 5 here. Now, technically - none of these RAID levels requires a full resynchronization, even though it is almost always recommended and performed by default. There is an option in Linux software RAID (mdadm) to "skip" the resynchronization process. The danger here is that you could read one of the blocks this minute and get one block, and read the same block a different minute, and get a different block. This would occur in RAID 1 if it did round-robin or disk with the nearest head to the desired block, or whatever, and it made a different decision before and after the minute. What is the worst that can happen though? Any system that does careful journalling / synchronization should usually be fine. The "risk" is similar to write caching without battery backing, in that if the drive tells the system "write complete", and the system goes on to perform other work, but the write is not complete, then corruption becomes a possibility. Anyways - point is again that RAID 5 is not special here. > but for your application, the fact that you are doing lots of fsyncs > is what's killing you, becouse the fsync forces a lot of data to be > written out, swamping the caches involved, and requiring that you wait > for seeks. nothing other then a battery backed disk cache of some sort > (either on the controller or a solid-state drive on a journaled > filesystem would work) Yep. :-) Cheers, mark -- Mark Mielke From gsmith@gregsmith.com Wed Dec 26 22:40:15 2007 Received: from localhost (unknown [200.46.204.191]) by postgresql.org (Postfix) with ESMTP id 570DF2E0114 for ; Wed, 26 Dec 2007 18:40:15 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.191]) (amavisd-maia, port 10024) with ESMTP id 00446-05 for ; Wed, 26 Dec 2007 18:40:08 -0400 (AST) Received: from westnet.com (westnet.com [216.187.52.2]) by postgresql.org (Postfix) with ESMTP id 5DF602E2D97 for ; Wed, 26 Dec 2007 18:40:08 -0400 (AST) Received: from westnet.com (localhost [127.0.0.1]) by westnet.com (8.14.0/8.14.0) with ESMTP id lBQMe7E0012281 for ; Wed, 26 Dec 2007 17:40:07 -0500 (EST) Received: from localhost (gsmith@localhost) by westnet.com (8.14.0/8.13.2/Submit) with ESMTP id lBQMe7ui012278 for ; Wed, 26 Dec 2007 17:40:07 -0500 (EST) X-Authentication-Warning: westnet.com: gsmith owned process doing -bs Date: Wed, 26 Dec 2007 17:40:06 -0500 (EST) From: Greg Smith X-X-Sender: gsmith@westnet.com To: "'pgsql-performance'" Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 In-Reply-To: Message-ID: References: <0f8201c847f6$08e15090$8f01010a@iptel.com.ar> <87zlvxnpkl.fsf@mid.deneb.enyo.de> <4772CA58.5050406@mark.mielke.cc> <4772CE34.2040608@mark.mielke.cc> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/393 X-Sequence-Number: 28209 On Wed, 26 Dec 2007, david@lang.hm wrote: > yes, the two linux software implementations only read from one disk, but I > have seen hardware implementations where it reads from both drives, and if > they disagree it returns a read error rather then possibly invalid data (it's > up to the admin to figure out which drive is bad at that point). Right, many of the old implementations did that; even the Wikipedia article on this subject mentions it in the "RAID 1 performance" section: http://en.wikipedia.org/wiki/Standard_RAID_levels The thing that changed is on modern drives, the internal error detection and correction is good enough that if you lose a sector, the drive will normally figure that out at the firmware level and return a read error rather than bad data. That lowers of the odds of one drive becoming corrupted and returning a bad sector as a result enough that the overhead of reading from both drives isn't considered as important. I'm not aware of a current card that does that but I wouldn't be surprised to discover one existed. -- * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD From david@lang.hm Wed Dec 26 21:40:08 2007 Received: from localhost (unknown [200.46.204.187]) by postgresql.org (Postfix) with ESMTP id 2BF202E023F for ; Wed, 26 Dec 2007 17:40:08 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.187]) (amavisd-maia, port 10024) with ESMTP id 81989-01 for ; Wed, 26 Dec 2007 17:39:56 -0400 (AST) Received: from bifrost.lang.hm (dsl081-033-126.lax1.dsl.speakeasy.net [64.81.33.126]) by postgresql.org (Postfix) with ESMTP id 50C652E01B5 for ; Wed, 26 Dec 2007 17:39:59 -0400 (AST) Received: from asgard.lang.hm (asgard.lang.hm [10.0.0.100]) by bifrost.lang.hm (8.13.4/8.13.4/Debian-3) with ESMTP id lBQLdleb023105; Wed, 26 Dec 2007 13:39:47 -0800 Date: Wed, 26 Dec 2007 14:52:20 -0800 (PST) From: david@lang.hm X-X-Sender: dlang@asgard.lang.hm To: Fernando Hevia cc: "'pgsql-performance'" Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 In-Reply-To: <0f9701c84801$399dcff0$8f01010a@iptel.com.ar> Message-ID: References: <0f9701c84801$399dcff0$8f01010a@iptel.com.ar> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/376 X-Sequence-Number: 28192 On Wed, 26 Dec 2007, Fernando Hevia wrote: >> David Lang Wrote: >> >> with only four drives the space difference between raid 1+0 and raid 5 >> isn't that much, but when you do a write you must write to two drives (the >> drive holding the data you are changing, and the drive that holds the >> parity data for that stripe, possibly needing to read the old parity data >> first, resulting in stalling for seek/read/calculate/seek/write since >> the drive moves on after the read), when you read you must read _all_ >> drives in the set to check the data integrity. > > Thanks for the explanation David. It's good to know not only what but also > why. Still I wonder why reads do hit all drives. Shouldn't only 2 disks be > read: the one with the data and the parity disk? no, becouse the parity is of the sort (A+B+C+P) mod X = 0 so if X=10 (which means in practice that only the last decimal digit of anything matters, very convienient for examples) A=1, B=2, C=3, A+B+C=6, P=4, A+B+C+P=10=0 if you read B and get 3 and P and get 4 you don't know if this is right or not unless you also read A and C (at which point you would get A+B+C+P=11=1=error) >> for seek heavy workloads (which almost every database application is) the >> extra seeks involved can be murder on your performance. if your workload >> is large sequential reads/writes, and you can let the OS buffer things for >> you, the performance of raid 5 is much better. > > Well, actually most of my application involves large sequential > reads/writes. The memory available for buffering (4GB) isn't bad either, at > least for my scenario. On the other hand I have got such strong posts > against RAID 5 that I doubt to even consider it. in theory a system could get the same performance with a large sequential read/write on raid5/6 as on a raid0 array of equivilent size (i.e. same number of data disks, ignoring the parity disks) becouse the OS could read the entire stripe in at once, do the calculation once, and use all the data (or when writing, don't write anything until you are ready to write the entire stripe, calculate the parity and write everything once). Unfortunantly in practice filesystems don't support this, they don't do enough readahead to want to keep the entire stripe (so after they read it all in they throw some of it away), they (mostly) don't know where a stripe starts (and so intermingle different types of data on one stripe and spread data across multiple stripes unessasarily), and they tend to do writes in small, scattered chunks (rather then flushing an entire stripes worth of data at once) those who have been around long enough to remember the days of MFM/RLL (when you could still find the real layout of the drives) may remember optmizing things to work a track at a time instead of a sector at a time. this is the exact same logic, just needing to be applied to drive stripes instead of sectors and tracks on a single drive. the issue has been raised with the kernel developers, but there's a lot of work to be done (especially in figuring out how to get all the layers the info they need in a reasonable way) >> Linux software raid can do more then two disks in a mirror, so you may be >> able to get the added protection with raid 1 sets (again, probably not >> relavent to four drives), although there were bugs in this within the last >> six months or so, so you need to be sure your kernel is new enough to have >> the fix. >> > > Well, here rises another doubt. Should I go for a single RAID 1+0 storing OS > + Data + WAL files or will I be better off with two RAID 1 separating data > from OS + Wal files? if you can afford the space, you are almost certinly better seperating the WAL from the data (I think I've seen debates about which is better OS+data/Wal or date/OS+Wal, but very little disagreement that either is better than combining them all) David Lang >> now, if you can afford solid-state drives which don't have noticable seek >> times, things are completely different ;-) > > Ha, sadly budget is very tight. :) > > Regards, > Fernando. > > > ---------------------------(end of broadcast)--------------------------- > TIP 5: don't forget to increase your free space map settings > From david@lang.hm Wed Dec 26 21:42:03 2007 Received: from localhost (unknown [200.46.204.182]) by postgresql.org (Postfix) with ESMTP id 929232E2ECA for ; Wed, 26 Dec 2007 17:42:03 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.182]) (amavisd-maia, port 10024) with ESMTP id 58319-02-2 for ; Wed, 26 Dec 2007 17:41:55 -0400 (AST) Received: from bifrost.lang.hm (dsl081-033-126.lax1.dsl.speakeasy.net [64.81.33.126]) by postgresql.org (Postfix) with ESMTP id 0FF002E2EC0 for ; Wed, 26 Dec 2007 17:41:56 -0400 (AST) Received: from asgard.lang.hm (asgard.lang.hm [10.0.0.100]) by bifrost.lang.hm (8.13.4/8.13.4/Debian-3) with ESMTP id lBQLfgkD023126; Wed, 26 Dec 2007 13:41:42 -0800 Date: Wed, 26 Dec 2007 14:54:15 -0800 (PST) From: david@lang.hm X-X-Sender: dlang@asgard.lang.hm To: Florian Weimer cc: Fernando Hevia , "'Mark Mielke'" , "'pgsql-performance'" Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 In-Reply-To: <87zlvxnpkl.fsf@mid.deneb.enyo.de> Message-ID: References: <0f8201c847f6$08e15090$8f01010a@iptel.com.ar> <87zlvxnpkl.fsf@mid.deneb.enyo.de> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/377 X-Sequence-Number: 28193 On Wed, 26 Dec 2007, Florian Weimer wrote: >> seek/read/calculate/seek/write since the drive moves on after the >> read), when you read you must read _all_ drives in the set to check >> the data integrity. > > I don't know of any RAID implementation that performs consistency > checking on each read operation. 8-( I could see a raid 1 array not doing consistancy checking (after all, it has no way of knowing what's right if it finds an error), but since raid 5/6 can repair the data I would expect them to do the checking each time. David Lang From david@lang.hm Wed Dec 26 21:53:32 2007 Received: from localhost (unknown [200.46.204.184]) by postgresql.org (Postfix) with ESMTP id 13FDD2E2EBB for ; Wed, 26 Dec 2007 17:53:32 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.184]) (amavisd-maia, port 10024) with ESMTP id 92471-01 for ; Wed, 26 Dec 2007 17:53:23 -0400 (AST) Received: from bifrost.lang.hm (dsl081-033-126.lax1.dsl.speakeasy.net [64.81.33.126]) by postgresql.org (Postfix) with ESMTP id E095C2E2EA3 for ; Wed, 26 Dec 2007 17:53:27 -0400 (AST) Received: from asgard.lang.hm (asgard.lang.hm [10.0.0.100]) by bifrost.lang.hm (8.13.4/8.13.4/Debian-3) with ESMTP id lBQLrH19023172; Wed, 26 Dec 2007 13:53:17 -0800 Date: Wed, 26 Dec 2007 15:05:50 -0800 (PST) From: david@lang.hm X-X-Sender: dlang@asgard.lang.hm To: Mark Mielke cc: Florian Weimer , Fernando Hevia , "'pgsql-performance'" Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 In-Reply-To: <4772CA58.5050406@mark.mielke.cc> Message-ID: References: <0f8201c847f6$08e15090$8f01010a@iptel.com.ar> <87zlvxnpkl.fsf@mid.deneb.enyo.de> <4772CA58.5050406@mark.mielke.cc> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/381 X-Sequence-Number: 28197 On Wed, 26 Dec 2007, Mark Mielke wrote: > Florian Weimer wrote: >>> seek/read/calculate/seek/write since the drive moves on after the >>> read), when you read you must read _all_ drives in the set to check >>> the data integrity. >>> >> I don't know of any RAID implementation that performs consistency >> checking on each read operation. 8-( >> > > Dave had too much egg nog... :-) > > Yep - checking consistency on read would eliminate the performance benefits > of RAID under any redundant configuration. except for raid0, raid is primarily a reliability benifit, any performance benifit is incidental, not the primary purpose. that said, I have heard of raid1 setups where it only reads off of one of the drives, but I have not heard of higher raid levels doing so. David Lang From david@lang.hm Wed Dec 26 22:22:16 2007 Received: from localhost (unknown [200.46.204.184]) by postgresql.org (Postfix) with ESMTP id 1B4922E326C for ; Wed, 26 Dec 2007 18:22:16 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.184]) (amavisd-maia, port 10024) with ESMTP id 98657-03 for ; Wed, 26 Dec 2007 18:22:05 -0400 (AST) Received: from bifrost.lang.hm (dsl081-033-126.lax1.dsl.speakeasy.net [64.81.33.126]) by postgresql.org (Postfix) with ESMTP id 30F7C2E3248 for ; Wed, 26 Dec 2007 18:22:10 -0400 (AST) Received: from asgard.lang.hm (asgard.lang.hm [10.0.0.100]) by bifrost.lang.hm (8.13.4/8.13.4/Debian-3) with ESMTP id lBQMM1BH023358; Wed, 26 Dec 2007 14:22:01 -0800 Date: Wed, 26 Dec 2007 15:34:35 -0800 (PST) From: david@lang.hm X-X-Sender: dlang@asgard.lang.hm To: Mark Mielke cc: Fernando Hevia , "'pgsql-performance'" Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 In-Reply-To: <4772CD78.4010302@mark.mielke.cc> Message-ID: References: <0f9701c84801$399dcff0$8f01010a@iptel.com.ar> <4772CD78.4010302@mark.mielke.cc> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/388 X-Sequence-Number: 28204 On Wed, 26 Dec 2007, Mark Mielke wrote: > david@lang.hm wrote: >>> Thanks for the explanation David. It's good to know not only what but also >>> why. Still I wonder why reads do hit all drives. Shouldn't only 2 disks be >>> read: the one with the data and the parity disk? >> no, becouse the parity is of the sort (A+B+C+P) mod X = 0 >> so if X=10 (which means in practice that only the last decimal digit of >> anything matters, very convienient for examples) >> A=1, B=2, C=3, A+B+C=6, P=4, A+B+C+P=10=0 >> if you read B and get 3 and P and get 4 you don't know if this is right or >> not unless you also read A and C (at which point you would get >> A+B+C+P=11=1=error) > I don't think this is correct. RAID 5 is parity which is XOR. The property of > XOR is such that it doesn't matter what the other drives are. You can write > any block given either: 1) The block you are overwriting and the parity, or > 2) all other blocks except for the block we are writing and the parity. Now, > it might be possible that option 2) is taken more than option 1) for some > complicated reasons, but it is NOT to check consistency. The array is assumed > consistent until proven otherwise. I was being sloppy in explaining the reason, you are correct that for writes you don't need to read all the data, you just need the current parity block, the old data you are going to replace, and the new data to be able to calculate the new parity block (and note that even with my checksum example this would be the case). however I was addressing the point that for reads you can't do any checking until you have read in all the blocks. if you never check the consistency, how will it ever be proven otherwise. >> in theory a system could get the same performance with a large sequential >> read/write on raid5/6 as on a raid0 array of equivilent size (i.e. same >> number of data disks, ignoring the parity disks) becouse the OS could read >> the entire stripe in at once, do the calculation once, and use all the data >> (or when writing, don't write anything until you are ready to write the >> entire stripe, calculate the parity and write everything once). > For the same number of drives, this cannot be possible. With 10 disks, on > raid5, 9 disks hold data, and 1 holds parity. The theoretical maximum > performance is only 9/10 of the 10/10 performance possible with RAID 0. I was saying that a 10 drive raid0 could be the same performance as a 10+1 drive raid 5 or a 10+2 drive raid 6 array. this is why I said 'same number of data disks, ignoring the parity disks'. in practice you would probably not do quite this good anyway (you have the parity calculation to make and the extra drive or two's worth of data passing over your busses), but it could be a lot closer then any implementation currently is. >> Unfortunantly in practice filesystems don't support this, they don't do >> enough readahead to want to keep the entire stripe (so after they read it >> all in they throw some of it away), they (mostly) don't know where a stripe >> starts (and so intermingle different types of data on one stripe and spread >> data across multiple stripes unessasarily), and they tend to do writes in >> small, scattered chunks (rather then flushing an entire stripes worth of >> data at once) > In my experience, this theoretical maximum is not attainable without > significant write cache, and an intelligent controller, neither of which > Linux software RAID seems to have by default. My situation was a bit worse in > that I used applications that fsync() or journalled metadata that is ordered, > which forces the Linux software RAID to flush far more than it should - but > the same system works very well with RAID 1+0. my statements above apply to any type of raid implementation, hardware or software. the thing that saves the hardware implementation is that the data is written to a battery-backed cache and the controller lies to the system, telling it that the write is complete, and then it does the write later. on a journaling filesystem you could get very similar results if you put the journal on a solid-state drive. but for your application, the fact that you are doing lots of fsyncs is what's killing you, becouse the fsync forces a lot of data to be written out, swamping the caches involved, and requiring that you wait for seeks. nothing other then a battery backed disk cache of some sort (either on the controller or a solid-state drive on a journaled filesystem would work) David Lang From david@lang.hm Wed Dec 26 22:24:21 2007 Received: from localhost (unknown [200.46.204.191]) by postgresql.org (Postfix) with ESMTP id B28052E0068 for ; Wed, 26 Dec 2007 18:24:21 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.191]) (amavisd-maia, port 10024) with ESMTP id 96011-04 for ; Wed, 26 Dec 2007 18:24:17 -0400 (AST) Received: from bifrost.lang.hm (dsl081-033-126.lax1.dsl.speakeasy.net [64.81.33.126]) by postgresql.org (Postfix) with ESMTP id D31B52E017E for ; Wed, 26 Dec 2007 18:24:16 -0400 (AST) Received: from asgard.lang.hm (asgard.lang.hm [10.0.0.100]) by bifrost.lang.hm (8.13.4/8.13.4/Debian-3) with ESMTP id lBQMO5cX023361; Wed, 26 Dec 2007 14:24:05 -0800 Date: Wed, 26 Dec 2007 15:36:38 -0800 (PST) From: david@lang.hm X-X-Sender: dlang@asgard.lang.hm To: Mark Mielke cc: Florian Weimer , Fernando Hevia , "'pgsql-performance'" Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 In-Reply-To: <4772CE34.2040608@mark.mielke.cc> Message-ID: References: <0f8201c847f6$08e15090$8f01010a@iptel.com.ar> <87zlvxnpkl.fsf@mid.deneb.enyo.de> <4772CA58.5050406@mark.mielke.cc> <4772CE34.2040608@mark.mielke.cc> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/389 X-Sequence-Number: 28205 On Wed, 26 Dec 2007, Mark Mielke wrote: > david@lang.hm wrote: >> On Wed, 26 Dec 2007, Mark Mielke wrote: >> >>> Florian Weimer wrote: >>>>> seek/read/calculate/seek/write since the drive moves on after the >>>>> read), when you read you must read _all_ drives in the set to check >>>>> the data integrity. >>>> I don't know of any RAID implementation that performs consistency >>>> checking on each read operation. 8-( >>> Dave had too much egg nog... :-) >>> Yep - checking consistency on read would eliminate the performance >>> benefits of RAID under any redundant configuration. >> except for raid0, raid is primarily a reliability benifit, any performance >> benifit is incidental, not the primary purpose. >> that said, I have heard of raid1 setups where it only reads off of one of >> the drives, but I have not heard of higher raid levels doing so. > What do you mean "heard of"? Which raid system do you know of that reads all > drives for RAID 1? > > Linux dmraid reads off ONLY the first. Linux mdadm reads off the "best" one. > Neither read from both. Why should it need to read from both? What will it do > if the consistency check fails? It's not like it can tell which disk is the > right one. It only knows that the whole array is inconsistent. Until it gets > an actual hardware failure (read error, write error), it doesn't know which > disk is wrong. yes, the two linux software implementations only read from one disk, but I have seen hardware implementations where it reads from both drives, and if they disagree it returns a read error rather then possibly invalid data (it's up to the admin to figure out which drive is bad at that point). no, I don't remember which card this was. I've been playing around with things in this space for quite a while. David Lang From david@lang.hm Wed Dec 26 22:26:24 2007 Received: from localhost (unknown [200.46.204.187]) by postgresql.org (Postfix) with ESMTP id 4169F2E095C for ; Wed, 26 Dec 2007 18:26:24 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.187]) (amavisd-maia, port 10024) with ESMTP id 96147-01 for ; Wed, 26 Dec 2007 18:26:09 -0400 (AST) Received: from bifrost.lang.hm (dsl081-033-126.lax1.dsl.speakeasy.net [64.81.33.126]) by postgresql.org (Postfix) with ESMTP id 1AB5A2E259B for ; Wed, 26 Dec 2007 18:26:11 -0400 (AST) Received: from asgard.lang.hm (asgard.lang.hm [10.0.0.100]) by bifrost.lang.hm (8.13.4/8.13.4/Debian-3) with ESMTP id lBQMQ1ZI023383; Wed, 26 Dec 2007 14:26:01 -0800 Date: Wed, 26 Dec 2007 15:38:34 -0800 (PST) From: david@lang.hm X-X-Sender: dlang@asgard.lang.hm To: Mark Mielke cc: Florian Weimer , Fernando Hevia , "'pgsql-performance'" Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 In-Reply-To: <4772CED1.2070808@mark.mielke.cc> Message-ID: References: <0f8201c847f6$08e15090$8f01010a@iptel.com.ar> <87zlvxnpkl.fsf@mid.deneb.enyo.de> <4772CED1.2070808@mark.mielke.cc> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/391 X-Sequence-Number: 28207 On Wed, 26 Dec 2007, Mark Mielke wrote: > david@lang.hm wrote: >> I could see a raid 1 array not doing consistancy checking (after all, it >> has no way of knowing what's right if it finds an error), but since raid >> 5/6 can repair the data I would expect them to do the checking each time. > Your messages are spread across the thread. :-) > > RAID 5 cannot repair the data. I don't know much about RAID 6, but I expect > it cannot necessarily repair the data either. It still doesn't know which > drive is wrong. In any case, there is no implementation I am aware of that > performs mandatory consistency checks on read. This would be silliness. sorry, raid 5 can repair data if it knows which chunk is bad (the same way it can rebuild a drive). Raid 6 does something slightly different for it's parity, I know it can recover from two drives going bad, but I haven't looked into the question of it detecting bad data. David Lang From pgsql@Sheeky.Biz Thu Dec 27 03:34:21 2007 Received: from localhost (unknown [200.46.204.191]) by postgresql.org (Postfix) with ESMTP id E1D982E0071 for ; Wed, 26 Dec 2007 23:34:21 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.191]) (amavisd-maia, port 10024) with ESMTP id 29587-06 for ; Wed, 26 Dec 2007 23:34:17 -0400 (AST) Received: from ipmail01.adl6.internode.on.net (ipmail01.adl6.internode.on.net [203.16.214.146]) by postgresql.org (Postfix) with ESMTP id 4D6472E003A for ; Wed, 26 Dec 2007 23:34:16 -0400 (AST) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Aj0KAEOrckfLevdH/2dsb2JhbAAsgSunew X-IronPort-AV: E=Sophos;i="4.24,209,1196602200"; d="scan'208";a="13498389" Received: from ppp247-71.static.internode.on.net (HELO DevBox.local) ([203.122.247.71]) by ipmail01.adl6.internode.on.net with ESMTP; 27 Dec 2007 14:04:12 +1030 Message-ID: <47731D32.3040403@Sheeky.Biz> Date: Thu, 27 Dec 2007 14:04:10 +1030 From: Shane Ambler User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: Fernando Hevia CC: 'pgsql-performance' Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 References: <0f9701c84801$399dcff0$8f01010a@iptel.com.ar> In-Reply-To: <0f9701c84801$399dcff0$8f01010a@iptel.com.ar> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/397 X-Sequence-Number: 28213 Fernando Hevia wrote: I'll start a little ways back first - > Well, here rises another doubt. Should I go for a single RAID 1+0 storing OS > + Data + WAL files or will I be better off with two RAID 1 separating data > from OS + Wal files? earlier you wrote - > Database will be about 30 GB in size initially and growing 10 GB per year. > Data is inserted overnight in two big tables and during the day mostly > read-only queries are run. Parallelism is rare. Now if the data is added overnight while no-one is using the server then reading is where you want performance, provided any degradation in writing doesn't slow down the overnight data loading enough to make it too long to finish while no-one else is using it. So in theory the only time you will have an advantage of having WAL on a separate disk from data is at night when the data is loading itself (I am assuming this is an automated step) But *some*? gains can be made from having the OS separate from the data. (This is for a theoretical discussion challenging the info/rumors that abounds about RAID setups) not to start a bitch fight or flame war. So for the guys who know the intricacies of RAID implementation - I don't have any real world performance measures here. For a setup that is only reading from disk (Santa sprinkles the data down the air vent while we are all snug in our bed) It has been mentioned that raid drivers/controllers can balance the workload across the different disks - as Mark mentioned from the FreeBSD 6 man pages - the balance option can be set to load|prefer|round-robin|split So in theory a modern RAID 1 setup can be configured to get similar read speeds as RAID 0 but would still drop to single disk speeds (or similar) when writing, but RAID 0 can get the faster write performance. So in a perfect setup (probably 1+0) 4x 300MB/s SATA drives could deliver 1200MB/s of data to RAM, which is also assuming that all 4 channels have their own data path to RAM and aren't sharing. (anyone know how segregated the on board controllers such as these are?) (do some pci controllers offer better throughput?) We all know that doesn't happen in the real world ;-) Let's say we are restricted to 80% - 1000MB/s - and some of that (10%) gets used by the system - so we end up with 900MB/s delivered off disk to postgres - that would still be more than the perfect rate at which 2x 300MB/s drives can deliver. So in this situation - if configured correctly with a good controller (driver for software RAID etc) a single 4 disk RAID 1+0 could outperform two 2 disk RAID 1 setups with data/OS+WAL split between the two. Is the real world speeds so different that this theory is real fantasy or has hardware reached a point performance wise where this is close to fact?? -- Shane Ambler pgSQL (at) Sheeky (dot) Biz Get Sheeky @ http://Sheeky.Biz From gsmith@gregsmith.com Thu Dec 27 04:10:53 2007 Received: from localhost (unknown [200.46.204.191]) by postgresql.org (Postfix) with ESMTP id 6ABFF2E0216 for ; Thu, 27 Dec 2007 00:10:53 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.191]) (amavisd-maia, port 10024) with ESMTP id 43988-03 for ; Thu, 27 Dec 2007 00:10:47 -0400 (AST) Received: from westnet.com (westnet.com [216.187.52.2]) by postgresql.org (Postfix) with ESMTP id 481A42E01CC for ; Thu, 27 Dec 2007 00:10:47 -0400 (AST) Received: from westnet.com (localhost [127.0.0.1]) by westnet.com (8.14.0/8.14.0) with ESMTP id lBR4AkvE019345; Wed, 26 Dec 2007 23:10:46 -0500 (EST) Received: from localhost (gsmith@localhost) by westnet.com (8.14.0/8.13.2/Submit) with ESMTP id lBR4AkFP019341; Wed, 26 Dec 2007 23:10:46 -0500 (EST) X-Authentication-Warning: westnet.com: gsmith owned process doing -bs Date: Wed, 26 Dec 2007 23:10:46 -0500 (EST) From: Greg Smith X-X-Sender: gsmith@westnet.com To: Shane Ambler cc: "'pgsql-performance'" Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 In-Reply-To: <47731D32.3040403@Sheeky.Biz> Message-ID: References: <0f9701c84801$399dcff0$8f01010a@iptel.com.ar> <47731D32.3040403@Sheeky.Biz> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/398 X-Sequence-Number: 28214 On Thu, 27 Dec 2007, Shane Ambler wrote: > So in theory a modern RAID 1 setup can be configured to get similar read > speeds as RAID 0 but would still drop to single disk speeds (or similar) when > writing, but RAID 0 can get the faster write performance. The trick is, you need a perfect controller that scatters individual reads evenly across the two disks as sequential reads move along the disk to pull this off, bouncing between a RAID 1 pair to use all the bandwidth available. There are caches inside the disk, read-ahead strategies as well, and that all has to line up just right for a single client to get all the bandwidth. Real-world disks and controllers don't quite behave well enough for that to predictably deliver what you might expect from theory. With RAID 0, getting the full read speed of 2Xsingle drive is much more likely to actually happen than in RAID 1. > So in a perfect setup (probably 1+0) 4x 300MB/s SATA drives could > deliver 1200MB/s of data to RAM, which is also assuming that all 4 > channels have their own data path to RAM and aren't sharing. (anyone > know how segregated the on board controllers such as these are?) (do > some pci controllers offer better throughput?) OK, first off, beyond the occasional trivial burst you'll be hard pressed to ever sustain over 60MB/s out of any single SATA drive. So the theoretical max 4-channel speed is closer to 240MB/s. A regular PCI bus tops out at a theoretical 133MB/s, and you sure can saturate one with 4 disks and a good controller. This is why server configurations have controller cards that use PCI-X (1024MB/s) or lately PCI-e aka PCI/Express (250MB/s for each channel with up to 16 being common). If your SATA cards are on a motherboard, that's probably using some integrated controller via the Southbridge AKA the ICH. That's probably got 250MB/s or more and in current products can easily outrun most sets of disks you'll ever connect. Even on motherboards that support 8 SATA channels it will be difficult for anything else on the system to go higher than 250MB/s even if the drives could potentially do more, and once you're dealing with real-world workloads. If you have multiple SATA controllers each with their own set of disk, then you're back to having to worry about the bus limits. So, yes, there are bus throughput considerations here, but unless you're building a giant array or using some older bus technology you're unlikely to hit them with spinning SATA disks. > We all know that doesn't happen in the real world ;-) Let's say we are > restricted to 80% - 1000MB/s Yeah, as mentioned above it's actually closer to 20%. While your numbers are off by a bunch, the reality for database use means these computations don't matter much anyway. The seek related behavior drives a lot of this more than sequential throughput, and decisions like whether to split out the OS or WAL or whatever need to factor all that, rather than just the theoretical I/O. For example, one reason it's popular to split the WAL onto another disk is that under normal operation the disk never does a seek. So if there's a dedicated disk for that, the disk just writes but never moves much. Where if the WAL is shared, the disk has to jump between writing that data and whatever else is going on, and peak possible WAL throughput is waaaay slower because of those seeks. (Note that unless you have a bunch of disks, your WAL is unlikely to be a limiter anyway so you still may not want to make it separate). (This topic so badly needs a PostgreSQL specific FAQ) -- * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD From mark@mark.mielke.cc Thu Dec 27 04:39:49 2007 Received: from localhost (unknown [200.46.204.187]) by postgresql.org (Postfix) with ESMTP id 2A5502E01AC for ; Thu, 27 Dec 2007 00:39:49 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.187]) (amavisd-maia, port 10024) with ESMTP id 47599-01 for ; Thu, 27 Dec 2007 00:39:38 -0400 (AST) Received: from ironport2-out.pppoe.ca (ironport2-out.pppoe.ca [206.248.154.182]) by postgresql.org (Postfix) with ESMTP id E41602E018D for ; Thu, 27 Dec 2007 00:39:38 -0400 (AST) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ao8CAPe6ckfO+Igp/2dsb2JhbACqMg X-IronPort-AV: E=Sophos;i="4.24,209,1196658000"; d="scan'208";a="11997706" Received: from smtp.pppoe.ca ([65.39.196.238]) by ironport2-out.pppoe.ca with ESMTP; 26 Dec 2007 23:39:37 -0500 Received: from [206.248.136.41] ([206.248.136.41]) by smtp.pppoe.ca (Internet Mail Server v1.0) with ESMTP id GQO08537; Wed, 26 Dec 2007 23:39:37 -0500 Message-ID: <47732BCB.2090302@mark.mielke.cc> Date: Wed, 26 Dec 2007 23:36:27 -0500 From: Mark Mielke User-Agent: Thunderbird 2.0.0.9 (X11/20071115) MIME-Version: 1.0 To: Shane Ambler CC: Fernando Hevia , 'pgsql-performance' Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 References: <0f9701c84801$399dcff0$8f01010a@iptel.com.ar> <47731D32.3040403@Sheeky.Biz> In-Reply-To: <47731D32.3040403@Sheeky.Biz> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/399 X-Sequence-Number: 28215 Shane Ambler wrote: > So in theory a modern RAID 1 setup can be configured to get similar > read speeds as RAID 0 but would still drop to single disk speeds (or > similar) when writing, but RAID 0 can get the faster write performance. Unfortunately, it's a bit more complicated than that. RAID 1 has a sequential read problem, as read-ahead is wasted, and you may as well read from one disk and ignore the others. RAID 1 does, however, allows for much greater concurrency. 4 processes on a 4 disk RAID 1 system can, theoretically, each do whatever they want, without impacting each other. Database loads involving a single active read user will see greater performance with RAID 0. Database loads involving multiple concurrent active read users will see greater performance with RAID 1. All of these assume writes are not being performed to any great significance. Even single writes cause all disks in a RAID 1 system to synchronize temporarily eliminating the read benefit. RAID 0 allows some degree of concurrent reads and writes occurring at the same time (assuming even distribution of the data across the devices). Of course, RAID 0 systems have an expected life that decreases as the number of disks in the system increase. So, this is where we get to RAID 1+0. Redundancy, good read performance, good write performance, relatively simple implementation. For a mere cost of double the number of disk storage, you can get around the problems of RAID 1 and the problems of RAID 0. :-) > So in a perfect setup (probably 1+0) 4x 300MB/s SATA drives could > deliver 1200MB/s of data to RAM, which is also assuming that all 4 > channels have their own data path to RAM and aren't sharing. > (anyone know how segregated the on board controllers such as these are?) > (do some pci controllers offer better throughput?) > We all know that doesn't happen in the real world ;-) Let's say we are > restricted to 80% - 1000MB/s - and some of that (10%) gets used by the > system - so we end up with 900MB/s delivered off disk to postgres - > that would still be more than the perfect rate at which 2x 300MB/s > drives can deliver. I expect you would have to have good hardware, and a well tuned system to see 80%+ theoretical for common work loads. But then, this isn't unique to RAID. Even in a single disk system, one has trouble achieving 80%+ theoretical. :-) I achieve something closer to +20% - +60% over the theoretical performance of a single disk with my four disk RAID 1+0 partitions. Lots of compromises in my system though that I won't get into. For me, I value the redundancy, allowing for a single disk to fail and giving me time to easily recover, but for the cost of two more disks, I am able to counter the performance cost of redundancy, and actually see a positive performance effect instead. > So in this situation - if configured correctly with a good controller > (driver for software RAID etc) a single 4 disk RAID 1+0 could > outperform two 2 disk RAID 1 setups with data/OS+WAL split between the > two. > Is the real world speeds so different that this theory is real fantasy > or has hardware reached a point performance wise where this is close > to fact?? I think it depends on the balance. If every second operation requires a WAL write, having separate might make sense. However, if the balance is less than even, one would end up with one of the 2 disk RAID 1 setups being more idle than the other. It's not an exact science when it comes to the various compromises being made. :-) If you can only put 4 disks in to the system (either cost, or because of the system size), I would suggest RAID 1+0 on all four as sensible compromise. If you can put more in - start to consider breaking it up. :-) Cheers, mark -- Mark Mielke From pgsql@Sheeky.Biz Thu Dec 27 07:16:49 2007 Received: from localhost (unknown [200.46.204.182]) by postgresql.org (Postfix) with ESMTP id DC28F2E01BF for ; Thu, 27 Dec 2007 03:16:49 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.182]) (amavisd-maia, port 10024) with ESMTP id 19061-03 for ; Thu, 27 Dec 2007 03:16:38 -0400 (AST) Received: from ipmail01.adl6.internode.on.net (ipmail01.adl6.internode.on.net [203.16.214.146]) by postgresql.org (Postfix) with ESMTP id 2A4282E01A7 for ; Thu, 27 Dec 2007 03:16:43 -0400 (AST) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Aj0KAPDfckfLevdH/2dsb2JhbAAsgSuqNA X-IronPort-AV: E=Sophos;i="4.24,210,1196602200"; d="scan'208";a="13597057" Received: from ppp247-71.static.internode.on.net (HELO DevBox.local) ([203.122.247.71]) by ipmail01.adl6.internode.on.net with ESMTP; 27 Dec 2007 17:46:40 +1030 Message-ID: <47735156.2080403@Sheeky.Biz> Date: Thu, 27 Dec 2007 17:46:38 +1030 From: Shane Ambler User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: Mark Mielke CC: Fernando Hevia , 'pgsql-performance' Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 References: <0f9701c84801$399dcff0$8f01010a@iptel.com.ar> <47731D32.3040403@Sheeky.Biz> <47732BCB.2090302@mark.mielke.cc> In-Reply-To: <47732BCB.2090302@mark.mielke.cc> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/401 X-Sequence-Number: 28217 Mark Mielke wrote: > Shane Ambler wrote: >> So in a perfect setup (probably 1+0) 4x 300MB/s SATA drives could >> deliver 1200MB/s of data to RAM, which is also assuming that all 4 >> channels have their own data path to RAM and aren't sharing. >> (anyone know how segregated the on board controllers such as these >> are?) >> (do some pci controllers offer better throughput?) >> We all know that doesn't happen in the real world ;-) Let's say we >> are restricted to 80% - 1000MB/s - and some of that (10%) gets used >> by the system - so we end up with 900MB/s delivered off disk to >> postgres - that would still be more than the perfect rate at which >> 2x 300MB/s drives can deliver. > > I achieve something closer to +20% - +60% over the theoretical > performance of a single disk with my four disk RAID 1+0 partitions. If a good 4 disk SATA RAID 1+0 can achieve 60% more throughput than a single SATA disk, what sort of percentage can be achieved from a good SCSI controller with 4 disks in RAID 1+0? Are we still hitting the bus limits at this point or can a SCSI RAID still outperform in raw data throughput? I would still think that SCSI would still provide the better reliability that it always has, but performance wise is it still in front of SATA? -- Shane Ambler pgSQL (at) Sheeky (dot) Biz Get Sheeky @ http://Sheeky.Biz From pgsql@Sheeky.Biz Thu Dec 27 07:40:00 2007 Received: from localhost (unknown [200.46.204.183]) by postgresql.org (Postfix) with ESMTP id 9FAAE2E01D9 for ; Thu, 27 Dec 2007 03:40:00 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 16767-02 for ; Thu, 27 Dec 2007 03:39:56 -0400 (AST) Received: from ipmail01.adl6.internode.on.net (ipmail01.adl6.internode.on.net [203.16.214.146]) by postgresql.org (Postfix) with ESMTP id 0D8D42E01C6 for ; Thu, 27 Dec 2007 03:39:55 -0400 (AST) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Aj0KAILjckfLevdH/2dsb2JhbAAsgSuqMQ X-IronPort-AV: E=Sophos;i="4.24,210,1196602200"; d="scan'208";a="13605048" Received: from ppp247-71.static.internode.on.net (HELO DevBox.local) ([203.122.247.71]) by ipmail01.adl6.internode.on.net with ESMTP; 27 Dec 2007 18:09:53 +1030 Message-ID: <477356C7.6010302@Sheeky.Biz> Date: Thu, 27 Dec 2007 18:09:51 +1030 From: Shane Ambler User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: Greg Smith CC: 'pgsql-performance' Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 References: <0f9701c84801$399dcff0$8f01010a@iptel.com.ar> <47731D32.3040403@Sheeky.Biz> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/402 X-Sequence-Number: 28218 Greg Smith wrote: > On Thu, 27 Dec 2007, Shane Ambler wrote: > >> So in theory a modern RAID 1 setup can be configured to get similar >> read speeds as RAID 0 but would still drop to single disk speeds (or >> similar) when writing, but RAID 0 can get the faster write performance. > > The trick is, you need a perfect controller that scatters individual > reads evenly across the two disks as sequential reads move along the > disk to pull this off, bouncing between a RAID 1 pair to use all the > bandwidth available. There are caches inside the disk, read-ahead > strategies as well, and that all has to line up just right for a single > client to get all the bandwidth. Real-world disks and controllers don't > quite behave well enough for that to predictably deliver what you might > expect from theory. With RAID 0, getting the full read speed of > 2Xsingle drive is much more likely to actually happen than in RAID 1. Kind of makes the point for using 1+0 >> So in a perfect setup (probably 1+0) 4x 300MB/s SATA drives could >> deliver 1200MB/s of data to RAM, which is also assuming that all 4 >> channels have their own data path to RAM and aren't sharing. > OK, first off, beyond the occasional trivial burst you'll be hard > pressed to ever sustain over 60MB/s out of any single SATA drive. So > the theoretical max 4-channel speed is closer to 240MB/s. > > A regular PCI bus tops out at a theoretical 133MB/s, and you sure can > saturate one with 4 disks and a good controller. This is why server > configurations have controller cards that use PCI-X (1024MB/s) or lately > PCI-e aka PCI/Express (250MB/s for each channel with up to 16 being > common). If your SATA cards are on a motherboard, that's probably using So I guess as far as performance goes your motherboard will determine how far you can take it. (talking from a db only server view on things) A PCI system will have little benefit from more than 2 disks but would need 4 to get both reliability and performance. PCI-X can benefit from up to 17 disks PCI-e (with 16 channels) can benefit from 66 disks The trick there will be dividing your db over a large number of disk sets to balance the load among them (I don't see 66 disks being setup in one array), so this would be of limited use to anyone but the most dedicated DBA's. For most servers these days the number of disks are added to reach a performance level not a storage requirement. > While your numbers are off by a bunch, the reality for database use > means these computations don't matter much anyway. The seek related > behavior drives a lot of this more than sequential throughput, and > decisions like whether to split out the OS or WAL or whatever need to > factor all that, rather than just the theoretical I/O. > So this is where solid state disks come in - lack of seek times (basically) means they can saturate your bus limits. -- Shane Ambler pgSQL (at) Sheeky (dot) Biz Get Sheeky @ http://Sheeky.Biz From wmoran@collaborativefusion.com Thu Dec 27 13:50:17 2007 Received: from localhost (unknown [200.46.204.183]) by postgresql.org (Postfix) with ESMTP id 093D12E2D97 for ; Thu, 27 Dec 2007 09:50:17 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 95715-01 for ; Thu, 27 Dec 2007 09:50:12 -0400 (AST) Received: from mx00.pub.collaborativefusion.com (mx00.pub.collaborativefusion.com [206.210.89.199]) by postgresql.org (Postfix) with ESMTP id AAF852E2D9A for ; Thu, 27 Dec 2007 09:50:11 -0400 (AST) Received: from vanquish.pitbpa0.priv.collaborativefusion.com (vanquish.pitbpa0.priv.collaborativefusion.com [192.168.2.61]) (SSL: TLSv1/SSLv3,256bits,AES256-SHA) by wingspan with esmtp; Thu, 27 Dec 2007 08:50:10 -0500 id 00056407.4773AD92.000177BF Date: Thu, 27 Dec 2007 08:50:09 -0500 From: Bill Moran To: Mark Mielke Cc: david@lang.hm, Florian Weimer , Fernando Hevia , 'pgsql-performance' Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 Message-Id: <20071227085009.94ea3bef.wmoran@collaborativefusion.com> In-Reply-To: <4772D3DA.10302@mark.mielke.cc> References: <0f8201c847f6$08e15090$8f01010a@iptel.com.ar> <87zlvxnpkl.fsf@mid.deneb.enyo.de> <4772CA58.5050406@mark.mielke.cc> <4772CE34.2040608@mark.mielke.cc> <20071226171105.889f6146.wmoran@collaborativefusion.com> <4772D3DA.10302@mark.mielke.cc> Organization: Collaborative Fusion X-Mailer: Sylpheed 2.4.7 (GTK+ 2.12.1; i386-portbld-freebsd6.2) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/407 X-Sequence-Number: 28223 In response to Mark Mielke : > Bill Moran wrote: > > > >> What do you mean "heard of"? Which raid system do you know of that reads > >> all drives for RAID 1? > >> > > > > I'm fairly sure that FreeBSD's GEOM does. Of course, it couldn't be doing > > consistency checking at that point. > > > According to this: > > http://www.freebsd.org/cgi/man.cgi?query=gmirror&apropos=0&sektion=8&manpath=FreeBSD+6-current&format=html > > There is a -b (balance) option that seems pretty clear that it does not > read from all drives if it does not have to: From mark@mark.mielke.cc Thu Dec 27 15:27:52 2007 Received: from localhost (unknown [200.46.204.182]) by postgresql.org (Postfix) with ESMTP id 17E8D2E0207 for ; Thu, 27 Dec 2007 11:27:52 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.182]) (amavisd-maia, port 10024) with ESMTP id 65382-01 for ; Thu, 27 Dec 2007 11:27:37 -0400 (AST) Received: from ironport2-out.pppoe.ca (ironport2-out.pppoe.ca [206.248.154.182]) by postgresql.org (Postfix) with ESMTP id 6FA6B2E0156 for ; Thu, 27 Dec 2007 11:27:48 -0400 (AST) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ah4FAE5Tc0fO+Igp/2dsb2JhbACCOzSpKw X-IronPort-AV: E=Sophos;i="4.24,211,1196658000"; d="scan'208,217";a="12007269" Received: from smtp.pppoe.ca ([65.39.196.238]) by ironport2-out.pppoe.ca with ESMTP; 27 Dec 2007 10:27:46 -0500 Received: from [206.248.136.41] ([206.248.136.41]) by smtp.pppoe.ca (Internet Mail Server v1.0) with ESMTP id GBC00046; Thu, 27 Dec 2007 10:27:46 -0500 Message-ID: <4773C3AE.1060602@mark.mielke.cc> Date: Thu, 27 Dec 2007 10:24:30 -0500 From: Mark Mielke User-Agent: Thunderbird 2.0.0.9 (X11/20071115) MIME-Version: 1.0 To: Bill Moran CC: david@lang.hm, Florian Weimer , Fernando Hevia , 'pgsql-performance' Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 References: <0f8201c847f6$08e15090$8f01010a@iptel.com.ar> <87zlvxnpkl.fsf@mid.deneb.enyo.de> <4772CA58.5050406@mark.mielke.cc> <4772CE34.2040608@mark.mielke.cc> <20071226171105.889f6146.wmoran@collaborativefusion.com> <4772D3DA.10302@mark.mielke.cc> <20071227085009.94ea3bef.wmoran@collaborativefusion.com> In-Reply-To: <20071227085009.94ea3bef.wmoran@collaborativefusion.com> Content-Type: multipart/alternative; boundary="------------070309030209030703060907" X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/410 X-Sequence-Number: 28226 This is a multi-part message in MIME format. --------------070309030209030703060907 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Bill Moran wrote: > In response to Mark Mielke : > > >> Bill Moran wrote: >> >>> I'm fairly sure that FreeBSD's GEOM does. Of course, it couldn't be doing >>> consistency checking at that point. >>> >> According to this: >> >> http://www.freebsd.org/cgi/man.cgi?query=gmirror&apropos=0&sektion=8&manpath=FreeBSD+6-current&format=html >> >> There is a -b (balance) option that seems pretty clear that it does not >> read from all drives if it does not have to: >> > > >From where did you draw that conclusion? Note that the "split" algorithm > (which is the default) divides requests up among multiple drives. I'm > unclear as to how you reached a conclusion opposite of what the man page > says -- did you test and find it not to work? > Perhaps you and I are speaking slightly different languages? :-) When I say "does not read from all drives", I mean "it will happily read from any of the drives to satisfy the request, and allows some level of configuration as to which drive it will select. It does not need to read all of the drives to satisfy the request." Cheers, mark -- Mark Mielke --------------070309030209030703060907 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Bill Moran wrote: 
In response to Mark Mielke <mark@mark.mielke.cc>:

  
Bill Moran wrote:
    
I'm fairly sure that FreeBSD's GEOM does.  Of course, it couldn't be doing
consistency checking at that point.
      
According to this:

http://www.freebsd.org/cgi/man.cgi?query=gmirror&apropos=0&sektion=8&manpath=FreeBSD+6-current&format=html

There is a -b (balance) option that seems pretty clear that it does not 
read from all drives if it does not have to:
    

>From where did you draw that conclusion?  Note that the "split" algorithm
(which is the default) divides requests up among multiple drives.  I'm
unclear as to how you reached a conclusion opposite of what the man page
says -- did you test and find it not to work?
Perhaps you and I are speaking slightly different languages? :-) When I say "does not read from all drives", I mean "it will happily read from any of the drives to satisfy the request, and allows some level of configuration as to which drive it will select. It does not need to read all of the drives to satisfy the request."

Cheers,
mark

-- 
Mark Mielke <mark@mielke.cc>
--------------070309030209030703060907-- From jeandavid8@verizon.net Thu Dec 27 16:36:46 2007 Received: from localhost (unknown [200.46.204.184]) by postgresql.org (Postfix) with ESMTP id 4F5442E3D1B for ; Thu, 27 Dec 2007 12:36:46 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.184]) (amavisd-maia, port 10024) with ESMTP id 82916-01-6 for ; Thu, 27 Dec 2007 12:36:38 -0400 (AST) Received: from vms040pub.verizon.net (vms040pub.verizon.net [206.46.252.40]) by postgresql.org (Postfix) with ESMTP id D78492E3DA9 for ; Thu, 27 Dec 2007 12:19:13 -0400 (AST) Received: from [192.168.0.251] ([71.251.196.74]) by vms040.mailsrvcs.net (Sun Java System Messaging Server 6.2-6.01 (built Apr 3 2006)) with ESMTPA id <0JTP005CUUPAKRH1@vms040.mailsrvcs.net> for pgsql-performance@postgresql.org; Thu, 27 Dec 2007 10:19:59 -0600 (CST) Date: Thu, 27 Dec 2007 11:18:52 -0500 From: Jean-David Beyer Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 In-reply-to: <47735156.2080403@Sheeky.Biz> To: 'pgsql-performance' Message-id: <4773D06C.2020205@verizon.net> Organization: Institute for Regimented Whimsey MIME-version: 1.0 Content-type: text/plain; charset=ISO-8859-1 Content-transfer-encoding: 7bit X-Enigmail-Version: 0.94.0.0 References: <0f9701c84801$399dcff0$8f01010a@iptel.com.ar> <47731D32.3040403@Sheeky.Biz> <47732BCB.2090302@mark.mielke.cc> <47735156.2080403@Sheeky.Biz> User-Agent: Thunderbird 1.5.0.9 (X11/20070317) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/413 X-Sequence-Number: 28229 Shane Ambler wrote: >> I achieve something closer to +20% - +60% over the theoretical >> performance of a single disk with my four disk RAID 1+0 partitions. > > If a good 4 disk SATA RAID 1+0 can achieve 60% more throughput than a > single SATA disk, what sort of percentage can be achieved from a good > SCSI controller with 4 disks in RAID 1+0? > > Are we still hitting the bus limits at this point or can a SCSI RAID > still outperform in raw data throughput? > > I would still think that SCSI would still provide the better reliability > that it always has, but performance wise is it still in front of SATA? > I have a SuperMicro X5DP8-G2 motherboard with two hyperthreaded microprocessors on it. This motherboard has 5 PCI-X busses (not merely 5 sockets: in fact it has 6 sockets, but also a dual Ultra/320 SCSI controller chip, a dual gigabit ethernet chip. So I hook up my 4 10,000 rpm database hard drives on one SCSI controller and the two other 10,000 rpm hard drives on the other. Nothing else is on the SCSI controller or its PCI-X bus that goes to the main memory except the other SCSI controller. These PCI-X busses are 133 MHz, and the memory as 266 MHz but the FSB runs at 533MHz as the memory modules are run in parallel; i.e., there are 8 modules and they run two at a time. Nothing else is on the other SCSI controller. Of the two hard drives on the second controller, one has the WAL on it, but when my database is running something (it is up all the time, but frequently idle) nothing else uses that drive much. So in theory, I should be able to get about 320 megabytes/second through each SCSI controller, though I have never seen that. I do get over 60 megabytes/second for brief (several second) periods though. I do not run RAID. I think it is probably very difficult to generalize how things go without a good knowledge of how the motherboard is organized, the amounts and types of caching that take place (both sortware and hardware), the speeds of the various devices and their controllers, the bandwidths of the various communication paths, and so on. -- .~. Jean-David Beyer Registered Linux User 85642. /V\ PGP-Key: 9A2FC99A Registered Machine 241939. /( )\ Shrewsbury, New Jersey http://counter.li.org ^^-^^ 11:00:01 up 10 days, 11:30, 2 users, load average: 4.20, 4.20, 4.25 From vivek@khera.org Thu Dec 27 16:46:49 2007 Received: from localhost (unknown [200.46.204.187]) by postgresql.org (Postfix) with ESMTP id AD8D72E02BB for ; Thu, 27 Dec 2007 12:46:49 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.187]) (amavisd-maia, port 10024) with ESMTP id 03204-02 for ; Thu, 27 Dec 2007 12:46:42 -0400 (AST) Received: from yertle.kcilink.com (thingy.kcilink.com [74.92.149.59]) by postgresql.org (Postfix) with ESMTP id F1DE12E3FAB for ; Thu, 27 Dec 2007 12:24:33 -0400 (AST) Received: from host-121.int.kcilink.com (host-121.int.kcilink.com [192.168.7.121]) by yertle.kcilink.com (Postfix) with ESMTP id 95BB6C943A for ; Thu, 27 Dec 2007 11:24:30 -0500 (EST) Message-Id: <94FAC00E-BF43-4EE9-9E23-C02E71CFC683@khera.org> From: Vivek Khera To: pgsql-performance In-Reply-To: <20071226102105.57d9b502.wmoran@collaborativefusion.com> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v915) Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 Date: Thu, 27 Dec 2007 11:24:30 -0500 References: <0ea101c847d2$16f94bc0$8f01010a@iptel.com.ar> <20071226102105.57d9b502.wmoran@collaborativefusion.com> X-Mailer: Apple Mail (2.915) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/414 X-Sequence-Number: 28230 On Dec 26, 2007, at 10:21 AM, Bill Moran wrote: > I snipped the rest of your message because none of it matters. > Never use > RAID 5 on a database system. Ever. There is absolutely NO reason to > every put yourself through that much suffering. If you hate yourself > that much just commit suicide, it's less drastic. Once you hit 14 or more spindles, the difference between RAID10 and RAID5 (or preferably RAID6) is minimal. In your 4 disk scenario, I'd vote RAID10. From vivek@khera.org Thu Dec 27 16:48:47 2007 Received: from localhost (unknown [200.46.204.183]) by postgresql.org (Postfix) with ESMTP id A87E12E3BB0 for ; Thu, 27 Dec 2007 12:48:47 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 79786-01 for ; Thu, 27 Dec 2007 12:48:43 -0400 (AST) Received: from yertle.kcilink.com (thingy.kcilink.com [74.92.149.59]) by postgresql.org (Postfix) with ESMTP id 9E6DF2E39E5 for ; Thu, 27 Dec 2007 12:26:59 -0400 (AST) Received: from host-121.int.kcilink.com (host-121.int.kcilink.com [192.168.7.121]) by yertle.kcilink.com (Postfix) with ESMTP id 0123DC943A for ; Thu, 27 Dec 2007 11:26:58 -0500 (EST) Message-Id: <2EA4C65E-0FF6-4439-BFEA-3C8586943244@khera.org> From: Vivek Khera To: pgsql-performance Performance In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v915) Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 Date: Thu, 27 Dec 2007 11:26:58 -0500 References: <0f8201c847f6$08e15090$8f01010a@iptel.com.ar> X-Mailer: Apple Mail (2.915) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/415 X-Sequence-Number: 28231 On Dec 26, 2007, at 4:28 PM, david@lang.hm wrote: > now, if you can afford solid-state drives which don't have noticable > seek times, things are completely different ;-) Who makes one with "infinite" lifetime? The only ones I know of are built using RAM and have disk drive backup with internal monitoring are *really* expensive. I've pondered building a raid enclosure using these new SATA flash drives, but that would be an expensive brick after a short period as one of my DB servers... From wmoran@collaborativefusion.com Thu Dec 27 17:05:44 2007 Received: from localhost (unknown [200.46.204.183]) by postgresql.org (Postfix) with ESMTP id 5C7992E2F87 for ; Thu, 27 Dec 2007 13:05:44 -0400 (AST) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 85111-01-10 for ; Thu, 27 Dec 2007 13:05:38 -0400 (AST) Received: from mx00.pub.collaborativefusion.com (mx00.pub.collaborativefusion.com [206.210.89.199]) by postgresql.org (Postfix) with ESMTP id 787F32E3584 for ; Thu, 27 Dec 2007 12:54:46 -0400 (AST) Received: from vanquish.pitbpa0.priv.collaborativefusion.com (vanquish.pitbpa0.priv.collaborativefusion.com [192.168.2.61]) (SSL: TLSv1/SSLv3,256bits,AES256-SHA) by wingspan with esmtp; Thu, 27 Dec 2007 11:54:45 -0500 id 00056407.4773D8D5.00000FB0 Date: Thu, 27 Dec 2007 11:54:44 -0500 From: Bill Moran To: Mark Mielke Cc: david@lang.hm, Florian Weimer , Fernando Hevia , 'pgsql-performance' Subject: Re: With 4 disks should I go for RAID 5 or RAID 10 Message-Id: <20071227115444.07a9af3f.wmoran@collaborativefusion.com> In-Reply-To: <4773C3AE.1060602@mark.mielke.cc> References: <0f8201c847f6$08e15090$8f01010a@iptel.com.ar> <87zlvxnpkl.fsf@mid.deneb.enyo.de> <4772CA58.5050406@mark.mielke.cc> <4772CE34.2040608@mark.mielke.cc> <20071226171105.889f6146.wmoran@collaborativefusion.com> <4772D3DA.10302@mark.mielke.cc> <20071227085009.94ea3bef.wmoran@collaborativefusion.com> <4773C3AE.1060602@mark.mielke.cc> Organization: Collaborative Fusion X-Mailer: Sylpheed 2.4.7 (GTK+ 2.12.1; i386-portbld-freebsd6.2) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200712/417 X-Sequence-Number: 28233 In response to Mark Mielke : > Bill Moran wrote: > > In response to Mark Mielke : > > > > > >> Bill Moran wrote: > >> > >>> I'm fairly sure that FreeBSD's GEOM does. Of course, it couldn't be doing > >>> consistency checking at that point. > >>> > >> According to this: > >> > >> http://www.freebsd.org/cgi/man.cgi?query=gmirror&apropos=0&sektion=8&manpath=FreeBSD+6-current&format=html > >> > >> There is a -b (balance) option that seems pretty clear that it does not > >> read from all drives if it does not have to: > >> > > > > >From where did you draw that conclusion? Note that the "split" algorithm > > (which is the default) divides requests up among multiple drives. I'm > > unclear as to how you reached a conclusion opposite of what the man page > > says -- did you test and find it not to work? > > > Perhaps you and I are speaking slightly different languages? :-) When I > say "does not read from all drives", I mean "it will happily read from > any of the drives to satisfy the request, and allows some level of > configuration as to which drive it will select. It does not need to read > all of the drives to satisfy the request." Ahh ... I did misunderstand you. -- Bill Moran Collaborative Fusion Inc. http://people.collaborativefusion.com/~wmoran/ wmoran@collaborativefusion.com Phone: 412-422-3463x4023