Received: from maia.hub.org (unknown [200.46.208.211])
	by mail.postgresql.org (Postfix) with ESMTP id 86415632636
	for <pgsql-performance-postgresql.org@mail.postgresql.org>;
	Fri, 26 Mar 2010 21:32:14 -0300 (ADT)
Received: from mail.postgresql.org ([200.46.204.86])
	by maia.hub.org (mx1.hub.org [200.46.208.211]) (amavisd-maia,
	port 10024) with ESMTP id 21349-04
	for <pgsql-performance-postgresql.org@mail.postgresql.org>;
	Sat, 27 Mar 2010 00:31:55 +0000 (UTC)
X-Greylist: delayed 00:06:16.715856 by SQLgrey-1.7.6
Received: from mail-iw0-f202.google.com (mail-iw0-f202.google.com
	[209.85.223.202])
	by mail.postgresql.org (Postfix) with ESMTP id 42E67632415
	for <pgsql-performance@postgresql.org>;
	Fri, 26 Mar 2010 21:32:03 -0300 (ADT)
Received: by iwn40 with SMTP id 40so2276736iwn.1
	for <pgsql-performance@postgresql.org>;
	Fri, 26 Mar 2010 17:32:01 -0700 (PDT)
Received: by 10.231.174.140 with SMTP id t12mr810953ibz.89.1269649544140;
	Fri, 26 Mar 2010 17:25:44 -0700 (PDT)
Received: from Craig-Jamess-MacBook-Pro.local
	(wsip-70-167-124-126.sd.sd.cox.net [70.167.124.126])
	by mx.google.com with ESMTPS id d2sm1246624ibr.15.2010.03.26.17.25.42
	(version=SSLv3 cipher=RC4-MD5); Fri, 26 Mar 2010 17:25:43 -0700 (PDT)
Message-ID: <4BAD5085.2080102@emolecules.com>
Date: Fri, 26 Mar 2010 17:25:41 -0700
From: Craig James <craig_james@emolecules.com>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: pgsql-performance@postgresql.org
Subject: Re: why does swap not recover?
References: <04C98397-261D-4A50-99CB-61FA3E4A974B@richyen.com>
In-Reply-To: <04C98397-261D-4A50-99CB-61FA3E4A974B@richyen.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Scanned: Maia Mailguard 1.0.1
X-Spam-Status: No, hits=-1.36 tagged_above=-10 required=5 tests=AWL=-0.620,
	BAYES_20=-0.74
X-Spam-Level: 
X-Archive-Number: 201003/343
X-Sequence-Number: 38168

On 3/26/10 4:57 PM, Richard Yen wrote:
> Hi everyone,
>
> We've recently encountered some swapping issues on our CentOS 64GB Nehalem machine, running postgres 8.4.2.  Unfortunately, I was foolish enough to set shared_buffers to 40GB.  I was wondering if anyone would have any insight into why the swapping suddenly starts, but never recovers?
>
> <img src="http://richyen.com/i/swap.png">
>
> Note, the machine has been up and running since mid-December 2009.  It was only a March 8 that this swapping began, and it's never recovered.
>
> If we look at dstat, we find the following:
>
> <img src="http://richyen.com/i/dstat.png">
>
> Note that it is constantly paging in, but never paging out.

This happens when you have too many processes using too much space to fit in real memory, but none of them are changing their memory image.  If the system swaps a process in, but that process doesn't change anything in memory, then there are no dirty pages and the kernel can just kick the process out of memory without writing anything back to the swap disk -- the data in the swap are still valid.

It's a classic problem when processes are running round-robin. Say you have space for 100 processes, but you're running 101 process.  When you get to the #101, #1 is the oldest so it swaps out.  Then #1 runs, and #2 is the oldest, so it gets kicked out.  Then #2 runs and kicks out #3 ... and so forth.  Going from 100 to 101 process brings the system nearly to a halt.

Some operating systems try to use tricks to keep this from happening, but it's a hard problem to solve.

Craig