Message-ID: <4C07F41A.1080501@2ndquadrant.com>
Date: Thu, 03 Jun 2010 14:27:38 -0400
From: Greg Smith <greg@2ndquadrant.com>
User-Agent: Thunderbird 2.0.0.24 (X11/20100317)
MIME-Version: 1.0
To: Craig James <craig_james@emolecules.com>
CC: pgsql-performance@postgresql.org
Subject: Re: Weird XFS WAL problem
References: <AANLkTikS_QvlkYfv5_hbsXFH4SsTh1nP_WoTwRqpUUcz@mail.gmail.com>
	<AANLkTikegoyBvJXhqZGLZyTwRVyBs4qCuCQ7d1Vntk4w@mail.gmail.com>
	<AANLkTikjkEvj2hG-D1CMu49PQXXbxHVzpXm9JbLWzMQo@mail.gmail.com>
	<AANLkTimtdk11hrWd-OpvXGLrh-4pukGd9eHEUemEM3OR@mail.gmail.com>
	<AANLkTinAhrwwwQKh6NrX-nnvcpxT1EFFlZrUxhaDB87C@mail.gmail.com>
	<AANLkTinuOUTbuke9FTMtwSObcZBio0uOS1ysWrqFyK3t@mail.gmail.com>
	<4C06E994.2080905@emolecules.com>
	<4C06EC01.3060506@catalyst.net.nz>
	<4C07E103.9020909@emolecules.com>
In-Reply-To: <4C07E103.9020909@emolecules.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Craig James wrote:
> This is really hard to believe, because the bonnie++ numbers and dd(1) 
> numbers look good (see my original post).  But it's totally 
> repeatable.  It must be some really unfortunate "just missed the next 
> sector going by the write head" problem.

Commit performance is a separate number to measure that is not reflected 
in any benchmark that tests sequential performance.  I consider it the 
fourth axis of disk system performance (seq read, seq write, random 
IOPS, commit rate), and directly measure it with the sysbench fsync test 
I recommended already.  (You can do it with the right custom pgbench 
script too).

You only get one commit per rotation on a drive, which is exactly what 
you're seeing:  a bit under the 120 spins/second @ 7200 RPM.  Attempts 
to time things just right to catch more than one sector per spin are 
extremely difficult to accomplish, I spent a week on that once without 
making any good progress.  You can easily get 100MB/s on reads and 
writes but only manage 100 commits/second.

-- 
Greg Smith  2ndQuadrant US  Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com   www.2ndQuadrant.us