Message-ID: <515A5352.4030406@catalyst.net.nz>
Date: Tue, 02 Apr 2013 16:41:06 +1300
From: Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
	rv:17.0) Gecko/20130311 Thunderbird/17.0.4
MIME-Version: 1.0
To: Armand du Plessis <adp@bank.io>
CC: pgsql-performance <pgsql-performance@postgresql.org>
Subject: Re: Fwd: Problems with pg_locks explosion
References: 
 <CANf99sX6dK9cG9PDhXARq70QybO3Tn7EkRwt_YPxp8Lo-AFhmQ@mail.gmail.com>
	<515A2223.5080204@catalyst.net.nz>
	<CANf99sWN4LP_xF012miq=MghCYtbm369bup9tVT1+BiTOR9sUA@mail.gmail.com>
In-Reply-To: 
 <CANf99sWN4LP_xF012miq=MghCYtbm369bup9tVT1+BiTOR9sUA@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
Sender: pgsql-performance-owner@postgresql.org

Yeah, as I understand it you should have 6000 IOPS available for the md 
device (ideally).

The iostats you display certainly look benign... but the key time to be 
sampling would be when you see the lock list explode - could look very 
different then.

Re vm.dirty* - I would crank the values down by a factor of 5:

vm.dirty_background_ratio = 1 (down from 5)
vm.dirty_ratio = 2 (down from 10)

Assuming of course that you actually are seeing an IO stall (which 
should be catchable via iostat or iotop)... and not some other issue. 
Otherwise leave 'em alone and keep looking :-)

Cheers

Mark


On 02/04/13 13:31, Armand du Plessis wrote:
>
>
> I had a look at the iostat output (on a 5s interval) and pasted it
> below. The utilization and waits seems low. Included a sample below, #1
> taken during normal operation and then when the locks happen it
> basically drops to 0 across the board. My (mis)understanding of the IOPS
> was that it would be 1000 IOPS per/volume and when in RAID0 should give
> me quite a bit higher throughput than in a single EBS volume setup. (My
> naive envelop calculation was #volumes * PIOPS = Effective IOPS :/)
>
>
> I'm looking into  vm.dirty_background_ratio, vm.dirty_ratio sysctls. Is
> there any guidance or links available that would be useful as a starting
> point?
>
> Thanks again for the help, I really appreciate it.
>


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance