Message-ID: <1bbfac04d1ec4a2708e47593dc84ba8c.squirrel@sq.gransy.com>
In-Reply-To: <008e01cd0817$06418680$12c49380$@com>
References: <006301cd07fd$358ebe40$a0ac3ac0$@com> <4F6AF3FE.30407@gmail.com>
	<dc1f59dfd626334d6f37d5a6ffb97bb9.squirrel@sq.gransy.com>
	<008e01cd0817$06418680$12c49380$@com>
Date: Thu, 22 Mar 2012 11:43:08 +0100
Subject: Re: Write workload is causing severe slowdown in
 Production
From: "Tomas Vondra" <tv@fuzzy.cz>
To: gnanam@zoniac.com
Cc: "'Tomas Vondra'" <tv@fuzzy.cz>, "'Vitalii Tymchyshyn'" <tivv00@gmail.com>,
	pgsql-performance@postgresql.org
User-Agent: SquirrelMail/1.4.22
MIME-Version: 1.0
Content-Type: text/plain;charset=utf-8
Importance: Normal
Content-Transfer-Encoding: quoted-printable

On 22 B=C5=99ezen 2012, 11:32, Gnanakumar wrote:
>> There's a checkpoint_warning option. Set it to 3600 and you should get
>> messages in the log. Correlate those to the issues (do they happen at
>> the
>> same time?).
> After setting "checkpoint_warning" to 3600, can you explain on how do I
> correlate with the messages?

Well, you do know when the issues happened, so that you can check the log=
s
and see if there are messages at that time. Try to install the applicatio=
n
and watch the logs / IO performance.

>> If you can, install iotop and watch the processes that cause the I/O.
> I tried installing "iotop", but it failed to run because it requires Li=
nux
> >=3D 2.6.20.  Our CentOS5.2 is 2.6.18-8.
>
>> What we need is more details about your setup, especially
>>   - checkpoint_segments
>>   - checkpoint_timeout
>>   - shared_buffers
> # - Memory -
> shared_buffers=3D1536MB
>
> # - Planner Cost Constants -
> effective_cache_size =3D 4GB

So, what else is running on the system? Because if there's 35GB RAM and
the shared buffers are 1.5GB, then there's about 33GB for page cache.
Something like 16GB would be a conservative setting.

I'm not saying this will fix the issues, but maybe it shows that somethin=
g
else is running on the box and maybe that's the culprit, not PostgreSQL?

>> also it'd be nice to have samples from the vmstat/iostat and messages
>> from
>> the log.
> Unfortunately, I don't have "exact" logs when the problem actually
> happened

Then install the app again and collect as much info as possible. Otherwis=
e
it's all just wild guesses.

Tomas