Received: from localhost (unknown [200.46.204.183]) by postgresql.org (Postfix) with ESMTP id 3B3F364FD17 for ; Thu, 11 Sep 2008 17:44:46 -0300 (ADT) Received: from postgresql.org ([200.46.204.86]) by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 16407-01 for ; Thu, 11 Sep 2008 17:44:42 -0300 (ADT) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from mail-gx0-f17.google.com (mail-gx0-f17.google.com [209.85.217.17]) by postgresql.org (Postfix) with ESMTP id 5E1F364FD05 for ; Thu, 11 Sep 2008 17:44:42 -0300 (ADT) Received: by gxk10 with SMTP id 10so17356550gxk.19 for ; Thu, 11 Sep 2008 13:44:41 -0700 (PDT) Received: by 10.142.84.5 with SMTP id h5mr1132678wfb.339.1221165880968; Thu, 11 Sep 2008 13:44:40 -0700 (PDT) Received: by 10.143.39.19 with HTTP; Thu, 11 Sep 2008 13:44:40 -0700 (PDT) Message-ID: Date: Thu, 11 Sep 2008 13:44:40 -0700 From: "Scott Carey" To: "Greg Smith" Subject: Re: Effects of setting linux block device readahead size Cc: "Mark Wong" , "pgsql-performance@postgresql.org" , "Gabrielle Roth" , "Selena Deckelmann" In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_69362_11855526.1221165880979" References: <70c01d1d0809092353r4ae687a3ycf3d99d43a5680cf@mail.gmail.com> X-Google-Sender-Auth: a484bdeaae2b6cd6 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0.001 tagged_above=0 required=5 tests=HTML_MESSAGE=0.001 X-Spam-Level: X-Archive-Number: 200809/147 X-Sequence-Number: 31460 ------=_Part_69362_11855526.1221165880979 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Sorry, I forgot to mention the Linux kernel version I'm using, etc: 2.6.18-92.1.10.el5 #1 SMP x86_64 CentOS 5.2. The "adaptive" read-ahead, as well as other enhancements in the kernel, are taking place or coming soon in the most recent stuff. Some distributions offer the adaptive read-ahead as an add-on (Debian, for example). This is an area where much can be improved in Linux http://kerneltrap.org/node/6642 http://kernelnewbies.org/Linux_2_6_23#head-102af265937262a7a21766ae58fddc1a29a5d8d7 I obviously did not test how the new read-ahead stuff impacts these sorts of tests. On Thu, Sep 11, 2008 at 12:07 PM, Scott Carey wrote: > Hmm, I would expect this tunable to potentially be rather file system > dependent, and potentially raid controller dependant. The test was using > ext2, perhaps the others automatically prefetch or read ahead? Does it > vary by RAID controller? > > Well I went and found out, using ext3 and xfs. I have about 120+ data > points but here are a few interesting ones before I compile the rest and > answer a few other questions of my own. > > 1: readahead does not affect "pure" random I/O -- there seems to be a > heuristic trigger -- a single process or file probably has to request a > sequence of linear I/O of some size to trigger it. I set it to over 64MB of > read-ahead and random iops remained the same to prove this. > 2: File system matters more than you would expect. XFS sequential > transfers when readahead was tuned had TWICE the sequential throughput of > ext3, both for a single reader and 8 concurrent readers on 8 different > files. > 3: The RAID controller and its configuration make a pretty significant > difference as well. > > Hardware: > 12 7200RPM SATA (Seagate) in raid 10 on 3Ware 9650 (only ext3) > 12 7200RPM SATA ('nearline SAS' : Seagate ES.2) on PERC 6 in raid 10 (ext3, > xfs) > I also have some results with PERC raid 10 with 4x 15K SAS, not reporting > in this message though > > . . . {snip} ------=_Part_69362_11855526.1221165880979 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline
Sorry, I forgot to mention the Linux kernel version I'm using, etc:

2.6.18-92.1.10.el5 #1 SMP x86_64
CentOS 5.2.

The "adaptive" read-ahead, as well as other enhancements in the kernel, are taking place or coming soon in the most recent stuff.  Some distributions offer the adaptive read-ahead as an add-on (Debian, for example).  This is an area where much can be improved in Linux http://kerneltrap.org/node/6642
http://kernelnewbies.org/Linux_2_6_23#head-102af265937262a7a21766ae58fddc1a29a5d8d7

I obviously did not test how the new read-ahead stuff impacts these sorts of tests.

On Thu, Sep 11, 2008 at 12:07 PM, Scott Carey <scott@richrelevance.com> wrote:
Hmm, I would expect this tunable to potentially be rather file system dependent, and potentially raid controller dependant.  The test was using ext2, perhaps the others automatically prefetch or read ahead?   Does it vary by RAID controller?

Well I went and found out, using ext3 and xfs.  I have about 120+ data points but here are a few interesting ones before I compile the rest and answer a few other questions of my own.

1:  readahead does not affect "pure" random I/O -- there seems to be a heuristic trigger -- a single process or file probably has to request a sequence of linear I/O of some size to trigger it.  I set it to over 64MB of read-ahead and random iops remained the same to prove this.
2:  File system matters more than you would expect.  XFS sequential transfers when readahead was tuned had TWICE the sequential throughput of ext3, both for a single reader and 8 concurrent readers on 8 different files.
3:  The RAID controller and its configuration make a pretty significant difference as well.

Hardware:
12 7200RPM SATA (Seagate) in raid 10 on 3Ware 9650 (only ext3)
12 7200RPM SATA ('nearline SAS' : Seagate ES.2) on PERC 6 in raid 10 (ext3, xfs)
I also have some results with PERC raid 10 with 4x 15K SAS, not reporting in this message though

  . . . {snip}
------=_Part_69362_11855526.1221165880979--