DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=message-id:date:from:to:subject:in-reply-to:mime-version
	:content-type:content-transfer-encoding:content-disposition
	:references;
	b=UDO8mMG2PuiuL9DXeBwlDW6OH3xC4u0S4eW/yl6FCZ5frV4FHy7NRL5AWC1SwZUldZ
	dXpwiK0vgTmRuSZ4G3vNQwhTkiyj4IfubszkxZs8DHCCckvHdGnktgI+NKYKkg3XHwgm
	YcPRrUnkTturMqEpD/Np++P3PpJwIibrx4gfI=
Message-ID: <72dbd3150811051925n50ead40cn30354990ebb8bfb6@mail.gmail.com>
Date: Wed, 5 Nov 2008 19:25:31 -0800
From: "David Rees" <drees76@gmail.com>
To: pgsql-performance@postgresql.org
Subject: Re: Occasional Slow Commit
In-Reply-To: <72dbd3150810311714s51639bc3kfdbe03a7b51bc55c@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <72dbd3150810271723w57859013v9c4e3718ae86fba@mail.gmail.com>
	<Pine.GSO.4.64.0810290906150.19233@westnet.com>
	<72dbd3150810291530q7a95da64j2c39354de9ad91d1@mail.gmail.com>
	<72dbd3150810311714s51639bc3kfdbe03a7b51bc55c@mail.gmail.com>

On Fri, Oct 31, 2008 at 4:14 PM, David Rees <drees76@gmail.com> wrote:
> Well, I'm pretty sure the delays are not checkpoint related. None of
> the slow commits line up at all with the end of checkpoints.
>
> The period of high delays occur during the same period of time each
> week, but it's not during a particularly high load period on the
> systems.
>
> It really seems like there must be something running in the background
> that is not showing up on the system activity logs, like a background
> RAID scrub or something.

OK, I finally had a chance to dig at this problem some more, and after
futzing around with the MegaCli tools (major PITA, btw), I was able to
confirm that there is a feature called "Patrol Read" on this LSI
Megaraid SAS card which runs a weekly background read scan of the
disks looking for errors. It is during this time period that I get
lots of slow commits and transactions.

FWIW, the card identifies itself from lspci as this:

LSI Logic / Symbios Logic MegaRAID SAS
Subsystem: Intel Corporation SROMBSAS18E RAID Controller

I also found that my write cache was set to WriteThrough instead of
WriteBack, defeating the purpose of having a BBU and that my secondary
server apparently doesn't have a BBU on it. :-(

Anyway, has anyone done any benchmarking of MegaRAID SAS controllers?
I am configuring my arrays to use these settings:

Read Policy: Normal (Normal, Read ahead & Adaptive read head)
Write Policy: Writeback (Writeback, Writethrough)
Disable Writeback if bad BBU
IO Policy: Direct (Direct, Cached)
Disk Cache: Enable (Enable, Disable, Unchanged)

The only setting I'm really concerned about is the Disk Cache setting
- is it safe to assume that the controller will do the right thing
with regards to flushing the disk cache when appropriate to avoid data
loss? LSI RAID cards seem to be pretty well respected, so I'd have to
guess yes.

Thanks

Dave