From: Greg Stark <gsstark@mit.edu>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Greg Stark <gsstark@mit.edu>
Cc: Mark Aufflick <mark@pumptheory.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: seq scan cache vs. index cache smackdown
Date: 15 Feb 2005 03:10:39 -0500
Message-ID: <87vf8u2qgw.fsf@stark.xeocode.com> (raw)
In-Reply-To: <9326.1108452962@sss.pgh.pa.us>
References: <FC8ED065-7EE0-11D9-904E-003065D62456@pumptheory.com>
<87d5v247z8.fsf@stark.xeocode.com>
<9326.1108452962@sss.pgh.pa.us>
Tom Lane <tgl@sss.pgh.pa.us> writes:
> Greg Stark <gsstark@mit.edu> writes:
> > 8.0, on the other hand, has a new algorithm that specifically tries to
> > protect against the shared buffers being blown out by a sequential
> > scan. But that will only help if it's the shared buffers being
> > thrashed that's hurting you, not the entire OS file system cache.
>
> Something we ought to think about sometime: what are the performance
> implications of the real-world situation that we have another level of
> caching sitting underneath us?
It seems inevitable that Postgres will eventually eliminate that redundant
layer of buffering. Since mmap is not workable, that means using O_DIRECT to
read table and index data.
Every other database eventually goes this direction, and for good reason.
Having two layers of caching and buffering is inherently inefficient. It also
makes it impossible for Postgres to offer any application-specific hints to
the caching replacement algorithms.
In that world you would configure Postgres much like you configure Oracle,
with shared_buffers taking up as much of your memory as you can afford. And
the OS file system cache is kept entirely out of the loop.
> AFAIK all the theoretical studies we've looked at consider only a single
> level of caching. But for example, if our buffer management algorithm
> recognizes an index page as being heavily hit and therefore keeps it in
> cache for a long time, then when it does fall out of cache you can be sure
> it's going to need to be read from disk when it's next used, because the
> OS-level buffer cache has not seen a call for that page in a long time.
> Contrariwise a page that we think is only on the fringe of usefulness is
> going to stay in the OS cache because we repeatedly drop it and then have to
> ask for it again.
Hum. Is it clear that that's bad? By the same logic it's the ones on the
fringe that you're likely to have to read again anyways. The ones that are
being heavily used are likely not to have to be read again anyways.
--
greg
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: gsstark@mit.edu, tgl@sss.pgh.pa.us, mark@pumptheory.com
Subject: Re: seq scan cache vs. index cache smackdown
In-Reply-To: <87vf8u2qgw.fsf@stark.xeocode.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox