Message-ID: <435D5654.2080209@modgraph-usa.com>
Date: Mon, 24 Oct 2005 14:47:00 -0700
From: "Craig A. James" <cjames@modgraph-usa.com>
User-Agent: Mozilla Thunderbird 1.0.6 (X11/20050716)
MIME-Version: 1.0
To: Scott Marlowe <smarlowe@g2switchworks.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: Used Memory
References: <435CBC9202000025000001AB@gwmta.wicourts.gov>	
	<435D1314.5040807@modgraph-usa.com>
	<1130185499.15546.120.camel@state.g2switchworks.com>
In-Reply-To: <1130185499.15546.120.camel@state.g2switchworks.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Scott Marlowe wrote:
>>What's needed is a way for the application developer to explicitely say,
>> "This object is frequenly used, and I want it kept in memory."
>  
> There's an interesting conversation happening on the linux kernel
> hackers mailing list right about now that applies:
> 
> http://www.gossamer-threads.com/lists/linux/kernel/580789

Thanks for the pointer.  If you're a participant in that mailing list, maybe you could forward this comment...

A fundamental flaw in the kernel, which goes WAY back to early UNIX implementations, is that the nice(1) setting of a program only applies to CPU usage, not to other resources.  In this case, the file-system cache has no priority, so even if I set postmaster's nice(1) value to a very high priority, any pissant process with the lowest priority possible can come along with a "cat some-big-file >/dev/null" and trash my cached file-system pages.  It's essentially a denial-of-service mechanism that's built in to the kernel.

The kernel group's discussion on the heuristics of how and when to toss stale cache pages should have a strong nice(1) component to it.  A process with a low priority should not be allowed to toss memory from a higher-priority process unless there is no other source of memory.

Getting back to Postgres, the same points that the linux kernel group are discussing apply to Postgres.  There is simply no way to devise a heuristic that comes even close to what the app developer can tell you.  A mechanism that allowed an application to say, "Keep this table in memory" is the only way.  App developers should be advised to use it sparingly, because most of the time the system is pretty good at memory management, and such a mechanism hobbles the system's ability to manage.  But when it's needed, there is no substitute.

Craig