Message-ID: <4E4DB552.2050605@catalyst.net.nz>
Date: Fri, 19 Aug 2011 12:58:58 +1200
From: Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
	rv:1.9.2.18) Gecko/20110617 Lightning/1.0b2 Thunderbird/3.1.11
MIME-Version: 1.0
To: Ogden <lists@darkstatic.com>
CC: pgsql-performance@postgresql.org
Subject: Re: Raid 5 vs Raid 10 Benchmarks Using bonnie++
References: <959BD2FA-0312-458B-BD5F-ED358D2C47ED@darkstatic.com>
	<f4e06a7cad34ee9217a3e57fd4eb0e67.squirrel@sq.gransy.com>
	<D89151D3-D2E5-47E3-8B90-4A573DAB5FFA@darkstatic.com>
	<75A85649-2B95-43D5-8404-4776D2E580EC@darkstatic.com>
	<4E4C2FB4.4010409@2ndQuadrant.com>
	<207E881B-73CD-4C31-A863-2C238838C41F@darkstatic.com>
	<4E4CA4BC.9050908@ringerc.id.au> <4E4CBA47.5070600@catalyst.net.nz>
	<CB8B0873-30DF-4B37-947C-093714543F62@darkstatic.com>
	<4E4DB3C4.5020307@catalyst.net.nz>
In-Reply-To: <4E4DB3C4.5020307@catalyst.net.nz>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 19/08/11 12:52, Mark Kirkwood wrote:
> On 19/08/11 02:09, Ogden wrote:
>> On Aug 18, 2011, at 2:07 AM, Mark Kirkwood wrote:
>>
>>> On 18/08/11 17:35, Craig Ringer wrote:
>>>> On 18/08/2011 11:48 AM, Ogden wrote:
>>>>> Isn't this very dangerous? I have the Dell PERC H700 card - I see 
>>>>> that it has 512Mb Cache. Is this the same thing and good enough to 
>>>>> switch to nobarrier? Just worried if a sudden power shut down, 
>>>>> then data can be lost on this option.
>>>>>
>>>>>
>>>> Yeah, I'm confused by that too. Shouldn't a write barrier flush 
>>>> data to persistent storage - in this case, the RAID card's battery 
>>>> backed cache? Why would it force a RAID controller cache flush to 
>>>> disk, too?
>>>>
>>>>
>>> If the card's cache has a battery, then the cache is preserved in 
>>> the advent of crash/power loss etc - provided it has enough charge, 
>>> so setting 'writeback' property on arrays is safe. The 
>>> PERC/SERVERRAID cards I'm familiar (LSI Megaraid rebranded models) 
>>> all switch to write-though mode if they detect the battery is 
>>> dangerously discharged so this is not normally a problem (but 
>>> commit/fsync performance will fall off a cliff when this happens)!
>>>
>>> Cheers
>>>
>>> Mark
>>
>> So a setting such as this:
>>
>> Device Name         : /dev/sdb
>> Type                : SAS
>> Read Policy         : No Read Ahead
>> Write Policy        : Write Back
>> Cache Policy        : Not Applicable
>> Stripe Element Size : 64 KB
>> Disk Cache Policy   : Enabled
>>
>>
>> Is sufficient to enable nobarrier then with these settings?
>>
>
>
> Hmm - that output looks different from the cards I'm familiar with. 
> I'd want to see the manual entries for  "Cache Policy=Not Applicable" 
> and "Disk Cache Policy=Enabled" to understand what the settings 
> actually mean. Assuming "Disk Cache Policy=Enabled" means what I think 
> it does (i.e writes are cached in the physical drives cache), this 
> setting seems wrong if your card has on board cache + battery, you 
> would want to only cache 'em in the *card's* cache  (too many caches 
> to keep straight in one's head, lol).
>

FWIW - here's what our ServerRaid (M5015) output looks like for a RAID 1 
array configured with writeback, reads not cached on the card's memory, 
physical disk caches disabled:

$ MegaCli64 -LDInfo -L0 -a0

Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name                :
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 67.054 GB
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 2
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache 
if Bad BBU
Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache 
if Bad BBU
Access Policy       : Read/Write
Disk Cache Policy   : Disabled
Encryption Type     : None