Message-ID: <4F7DBF24.5010402@fuzzy.cz>
Date: Thu, 05 Apr 2012 17:49:56 +0200
From: Tomas Vondra <tv@fuzzy.cz>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
	rv:11.0) Gecko/20120312 Thunderbird/11.0
MIME-Version: 1.0
CC: "pgsql-performance@postgresql.org" <pgsql-performance@postgresql.org>
Subject: Re: H800 + md1200 Performance problem
References: 
 <CAMAsR=4acKwYjow-VWQ0xH0+WjXEtVJWxP0A0y4vx=oyr5fhpQ@mail.gmail.com>
	<920BB532-C321-45A3-851E-8B688A5963F7@yahoo.com>
	<CAMAsR=7g5PXU45+Z3aBW+ur-kPQKiW2MG=NJ3TOqNn6iqaVCVw@mail.gmail.com>
	<4F7AF966.20809@fuzzy.cz>
	<CAMAsR=637=zAtOSxGF=JH9apfOtU7FaUXR27+J2syzE_c-rdYw@mail.gmail.com>
	<4F7B3AEC.4000204@fuzzy.cz>
	<CAMAsR=7onjeWr--PtgHgfZv=yYSB8FVxf1BsYSwu2752YY0Q8w@mail.gmail.com>
	<CAHyXU0yAy6C__0xbfBH1t_UNai=VVSP9i5p-KKgDAYp3pxAM-Q@mail.gmail.com>
	<CAMAsR=7Xkzcr-G_NO-Jn2yTa+BSayuMMK7Gv7diXDNcC3WNVUw@mail.gmail.com>
	<4F7CA614.8060008@fuzzy.cz>
	<1333631401.49335.YahooMailNeo@web171401.mail.ir2.yahoo.com>
	<CAMAsR=4jszO0RT0poCbebzTod767Q3e53zBAc=08Wz-0p-yW0w@mail.gmail.com>
In-Reply-To: 
 <CAMAsR=4jszO0RT0poCbebzTod767Q3e53zBAc=08Wz-0p-yW0w@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 5.4.2012 17:17, Cesar Martin wrote:
> Well, I have installed megacli on server and attach the results in file
> megacli.txt. Also we have "Dell Open Manage" install in server, that can
> generate a log of H800. I attach to mail with name lsi_0403.
> 
> About dirty limits, I have default values:
> vm.dirty_background_ratio = 10
> vm.dirty_ratio = 20
> 
> I have compared with other server and values are the same, except in
> centos 5.4 database production server that have vm.dirty_ratio = 40

Do the other machines have the same amount of RAM? The point is that the
values that work with less memory don't work that well with large
amounts of memory (and the amount of RAM did grow a lot recently).

For example a few years ago the average amount of RAM was ~8GB. In that
case the

  vm.dirty_background_ratio = 10  => 800MB
  vm.dirty_ratio = 20 => 1600MB

which is all peachy if you have a decent controller with a write cache.
But turn that to 64GB and suddenly

  vm.dirty_background_ratio = 10  => 6.4GB
  vm.dirty_ratio = 20 => 12.8GB

The problem is that there'll be a lot of data waiting (for 30 seconds by
default), and then suddenly it starts writing all of them to the
controller. Such systems behave just as your system - short strokes of
writes interleaved with 'no activity'.

Greg Smith wrote a nice howto about this - it's from 2007 but all the
recommendations are still valid:

  http://www.westnet.com/~gsmith/content/linux-pdflush.htm

TL;DR:

  - decrease the dirty_background_ratio/dirty_ratio (or use *_bytes)

  - consider decreasing the dirty_expire_centiseconds


T.