Message-ID: <515A79BC.9020006@catalyst.net.nz>
Date: Tue, 02 Apr 2013 19:25:00 +1300
From: Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
	rv:17.0) Gecko/20130311 Thunderbird/17.0.4
MIME-Version: 1.0
To: Jeff Janes <jeff.janes@gmail.com>
CC: Armand du Plessis <adp@bank.io>,
	pgsql-performance <pgsql-performance@postgresql.org>
Subject: Re: Problems with pg_locks explosion
References: 
 <CANf99sX6dK9cG9PDhXARq70QybO3Tn7EkRwt_YPxp8Lo-AFhmQ@mail.gmail.com>
	<515A2223.5080204@catalyst.net.nz>
	<CAMkU=1zFuG2PDzcZ4xwX20aNBF9NXzXNnz8c7Gsmrs3V1jXMWQ@mail.gmail.com>
In-Reply-To: 
 <CAMkU=1zFuG2PDzcZ4xwX20aNBF9NXzXNnz8c7Gsmrs3V1jXMWQ@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
Sender: pgsql-performance-owner@postgresql.org

On 02/04/13 19:08, Jeff Janes wrote:
> On Monday, April 1, 2013, Mark Kirkwood wrote:
>
>
>     Your provisioned volumes are much better than the default AWS ones,
>     but are still not hugely fast (i.e 1000 IOPS is about 8 MB/s worth
>     of Postgres 8k buffers). So you may need to look at adding more
>     volumes into the array, or adding some separate ones and putting
>     pg_xlog directory on 'em.
>
>     However before making changes I would recommend using iostat or sar
>     to monitor how volumes are handling the load (I usually choose a 1
>     sec granularity and look for 100% util and high - server hundred ms
>     - awaits). Also iotop could be enlightening.
>
>
> Hi Mark,
>
> Do you have experience using these tools with AWS?  When using non-DAS
> in other contexts, I've noticed that these tools often give deranged
> results, because the kernel doesn't correctly know what time to
> attribute to "network" and what to attribute to "disk".  But I haven't
> looked into it on AWS EBS, maybe they do a better job there.
> Thanks for any insight,
>


Hi Jeff,

That is a very good point. I did notice a reasonable amount of network 
traffic on the graphs posted previously, along with a suspiciously low 
amount of IO for md127 (which I assume is the raid0 array)...and 
wondered if iostat was not seeing IO fully, however it slipped my mind 
(I am on leave with kittens - so claim that for the purrrfect excuse)!

However I don't recall there being a problem with the io tools for 
standard EBS volumes - but I haven't benchmarked AWS for a over a year, 
so things could be different now - and I have no experience with these 
new provisioned volumes.

Armand - it might be instructive to do some benchmarking (with another 
host and volume set) where you do something like:

$ dd if=/dev/zero of=file bs=8k count=1000000

and see if iostat and friends actually show you doing IO as expected!


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance