pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Scott Marlowe <smarlowe@g2switchworks.com>
To: Craig A. James <cjames@modgraph-usa.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: Used Memory
Date: Mon, 24 Oct 2005 15:24:59 -0500
Message-ID: <1130185499.15546.120.camel@state.g2switchworks.com> (raw)
In-Reply-To: <435D1314.5040807@modgraph-usa.com>
References: <435CBC9202000025000001AB@gwmta.wicourts.gov>
	<435D1314.5040807@modgraph-usa.com>

On Mon, 2005-10-24 at 12:00, Craig A. James wrote:
> Kevin Grittner wrote:
> > In addition to what Mark pointed out, there is the possibility that a
> > query
> > is running which is scanning a large table or otherwise bringing in a
> > large number of pages from disk.  That would first use up all available
> > unused cache space, and then may start replacing some of your
> > frequently used data.  
> 
> An LRU cache is often a bad strategy for database applications.  There are two illustrations that show why.
> 
> 1. You have an index that's used for EVERY search, but each search returns a large and unique set of rows.  If it happens that the rows returned exceed the systems cache size, the part or all of your index will be flushed with EVERY query.
> 
> 2. You do a sequential scan of a table that's one block bigger than the file system cache, then you do it again.  At the beginning of the second scan, the first block of the table will have just been swapped out because it was the oldest, so the file system brings it back in, replacing the second block, which is now the oldest.  As you scan the table, each block of the table is swapped out JUST BEFORE you get to it.  At the start of your query, the file system might have had 99.9% of the relevant data in memory, but it swaps out 100% of it as your query progresses.
> 
> Scenario 2 above is interesting because a system that is performing very well can suddenly experience a catastrophic performance decline when the size of the data exceeds a critical limit - the file system's avaliable cache.
> 
> LRU works well if your frequently-used data is used often enough to keep it in memory.  But many applications don't have that luxury.  It's often the case that a single query will exceed the file system's cache size.  The file system cache is "dumb" -- it's strategy is too simple for a relational database.
> 
> What's needed is a way for the application developer to explicitely say, "This object is frequenly used, and I want it kept in memory."

There's an interesting conversation happening on the linux kernel
hackers mailing list right about now that applies:

http://www.gossamer-threads.com/lists/linux/kernel/580789



view thread (19+ messages)  latest in thread

Message-ID: <1130185499.15546.120.camel@state.g2switchworks.com>
Permalink:  ../1130185499.15546.120.camel@state.g2switchworks.com/
Also on:    postgresql.org/message-id/1130185499.15546.120.camel@state.g2switchworks.com

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-performance@postgresql.org
  Cc: smarlowe@g2switchworks.com, cjames@modgraph-usa.com
  Subject: Re: Used Memory
  In-Reply-To: <1130185499.15546.120.camel@state.g2switchworks.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox