pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedWhat gets cached?
9+ messages / 7 participants
[nested] [flat]
* What gets cached?
@ 2005-10-21 12:34 Martin Nickel <martin@portant.com>
0 siblings, 4 replies; 9+ messages in thread
From: Martin Nickel @ 2005-10-21 12:34 UTC (permalink / raw)
To: pgsql-performance
I was reading a comment in another posting and it started me thinking
about this. Let's say I startup an Oracle server. All my queries are a
little bit (sometimes a lot bit) slow until it gets its "normal" things in
memory, then it's up to speed. The "normal" things would include some
small lookup tables and the indexes for the most frequently used tables.
Let's say I do the same thing in Postgres. I'm likely to have my very
fastest performance for the first few queries until memory gets filled up.
The only time Postgres seems to take advantage of cached data is when I
repeat the same (or substantially the same) query. I don't know of any
way to view what is actually cached at any point in time, but it seems
like "most recently used" rather than "most frequently used".
Does this seem true?
s
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: What gets cached?
@ 2005-10-21 12:40 Steinar H. Gunderson <sgunderson@bigfoot.com>
parent: Martin Nickel <martin@portant.com>
3 siblings, 0 replies; 9+ messages in thread
From: Steinar H. Gunderson @ 2005-10-21 12:40 UTC (permalink / raw)
To: pgsql-performance
On Fri, Oct 21, 2005 at 07:34:30AM -0500, Martin Nickel wrote:
> Let's say I do the same thing in Postgres. I'm likely to have my very
> fastest performance for the first few queries until memory gets filled up.
> The only time Postgres seems to take advantage of cached data is when I
> repeat the same (or substantially the same) query. I don't know of any
> way to view what is actually cached at any point in time, but it seems
> like "most recently used" rather than "most frequently used".
What version are you using? There have been significant improvements to the
buffer manager in the last few versions. Most of the caching is done by your
OS, though, so that would probably influence the results quite a bit.
/* Steinar */
--
Homepage: http://www.sesse.net/
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: What gets cached?
@ 2005-10-21 15:19 Alex Turner <armtuk@gmail.com>
parent: Martin Nickel <martin@portant.com>
3 siblings, 0 replies; 9+ messages in thread
From: Alex Turner @ 2005-10-21 15:19 UTC (permalink / raw)
To: Martin Nickel <martin@portant.com>; +Cc: pgsql-performance
Oracle uses LRU caching algorithm also, not LFU.
Alex
On 10/21/05, Martin Nickel <martin@portant.com> wrote:
>
> I was reading a comment in another posting and it started me thinking
> about this. Let's say I startup an Oracle server. All my queries are a
> little bit (sometimes a lot bit) slow until it gets its "normal" things in
> memory, then it's up to speed. The "normal" things would include some
> small lookup tables and the indexes for the most frequently used tables.
>
> Let's say I do the same thing in Postgres. I'm likely to have my very
> fastest performance for the first few queries until memory gets filled up.
> The only time Postgres seems to take advantage of cached data is when I
> repeat the same (or substantially the same) query. I don't know of any
> way to view what is actually cached at any point in time, but it seems
> like "most recently used" rather than "most frequently used".
>
> Does this seem true?
> s
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Have you searched our list archives?
>
> http://archives.postgresql.org
>
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: What gets cached?
@ 2005-10-21 15:59 Michael Fuhr <mike@fuhr.org>
parent: Martin Nickel <martin@portant.com>
3 siblings, 0 replies; 9+ messages in thread
From: Michael Fuhr @ 2005-10-21 15:59 UTC (permalink / raw)
To: Martin Nickel <martin@portant.com>; +Cc: pgsql-performance
On Fri, Oct 21, 2005 at 07:34:30AM -0500, Martin Nickel wrote:
> I don't know of any way to view what is actually cached at any point in time
In 8.1 (currently in beta) you can use contrib/pg_buffercache. Code
for older versions is available on PgFoundry:
http://pgfoundry.org/projects/pgbuffercache/
Note that pg_buffercache shows only pages in PostgreSQL's buffer
cache; it doesn't show your operating system's cache.
--
Michael Fuhr
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: What gets cached?
@ 2005-10-21 21:44 Neil Conway <neilc@samurai.com>
parent: Martin Nickel <martin@portant.com>
3 siblings, 1 reply; 9+ messages in thread
From: Neil Conway @ 2005-10-21 21:44 UTC (permalink / raw)
To: Martin Nickel <martin@portant.com>; +Cc: pgsql-performance
On Fri, 2005-21-10 at 07:34 -0500, Martin Nickel wrote:
> Let's say I do the same thing in Postgres. I'm likely to have my very
> fastest performance for the first few queries until memory gets filled up.
No, you're not: if a query doesn't hit the cache (both the OS cache and
the Postgres userspace cache), it will run slower. If the caches are
empty when Postgres starts up (which is true for the userspace cache and
might be true of the OS cache), the first queries that are run should be
slower, not faster.
> The only time Postgres seems to take advantage of cached data is when I
> repeat the same (or substantially the same) query.
Caching is done on a page-by-page basis -- the source text of the query
itself is not relevant. If two different queries happen to hit a similar
set of pages, they will probably both benefit from the same set of
cached pages.
> I don't know of any way to view what is actually cached at any point
> in time, but it seems like "most recently used" rather than "most
> frequently used".
The cache replacement policy in 7.4 and older releases is simple LRU.
The policy in 8.0 is ARC (essentially a version of LRU modified to try
to retain hot pages more accurately). The policy in 8.1 is a clock-based
algorithm.
-Neil
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: What gets cached?
@ 2005-10-24 15:09 Alex Turner <armtuk@gmail.com>
parent: Neil Conway <neilc@samurai.com>
0 siblings, 1 reply; 9+ messages in thread
From: Alex Turner @ 2005-10-24 15:09 UTC (permalink / raw)
To: Neil Conway <neilc@samurai.com>; +Cc: Martin Nickel <martin@portant.com>; pgsql-performance
Just to play devils advocate here for as second, but if we have an algorithm
that is substational better than just plain old LRU, which is what I believe
the kernel is going to use to cache pages (I'm no kernel hacker), then why
don't we apply that and have a significantly larger page cache a la Oracle?
Alex
On 10/21/05, Neil Conway <neilc@samurai.com> wrote:
>
> On Fri, 2005-21-10 at 07:34 -0500, Martin Nickel wrote:
> > Let's say I do the same thing in Postgres. I'm likely to have my very
> > fastest performance for the first few queries until memory gets filled
> up.
>
> No, you're not: if a query doesn't hit the cache (both the OS cache and
> the Postgres userspace cache), it will run slower. If the caches are
> empty when Postgres starts up (which is true for the userspace cache and
> might be true of the OS cache), the first queries that are run should be
> slower, not faster.
>
> > The only time Postgres seems to take advantage of cached data is when I
> > repeat the same (or substantially the same) query.
>
> Caching is done on a page-by-page basis -- the source text of the query
> itself is not relevant. If two different queries happen to hit a similar
> set of pages, they will probably both benefit from the same set of
> cached pages.
>
> > I don't know of any way to view what is actually cached at any point
> > in time, but it seems like "most recently used" rather than "most
> > frequently used".
>
> The cache replacement policy in 7.4 and older releases is simple LRU.
> The policy in 8.0 is ARC (essentially a version of LRU modified to try
> to retain hot pages more accurately). The policy in 8.1 is a clock-based
> algorithm.
>
> -Neil
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: explain analyze is your friend
>
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: What gets cached?
@ 2005-10-24 15:32 Steinar H. Gunderson <sgunderson@bigfoot.com>
parent: Alex Turner <armtuk@gmail.com>
0 siblings, 1 reply; 9+ messages in thread
From: Steinar H. Gunderson @ 2005-10-24 15:32 UTC (permalink / raw)
To: Alex Turner <armtuk@gmail.com>; +Cc: Neil Conway <neilc@samurai.com>; Martin Nickel <martin@portant.com>; pgsql-performance
On Mon, Oct 24, 2005 at 11:09:55AM -0400, Alex Turner wrote:
> Just to play devils advocate here for as second, but if we have an algorithm
> that is substational better than just plain old LRU, which is what I believe
> the kernel is going to use to cache pages (I'm no kernel hacker), then why
> don't we apply that and have a significantly larger page cache a la Oracle?
There have (AFAIK) been reports of setting huge amounts of shared_buffers
(close to the total amount of RAM) performing much better in 8.1 than in
earlier versions, so this might actually be okay these days.
I haven't heard of anybody reporting increase setting such values, though.
/* Steinar */
--
Homepage: http://www.sesse.net/
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: What gets cached?
@ 2005-10-27 20:41 PostgreSQL <martin@portant.com>
parent: Steinar H. Gunderson <sgunderson@bigfoot.com>
0 siblings, 1 reply; 9+ messages in thread
From: PostgreSQL @ 2005-10-27 20:41 UTC (permalink / raw)
To: pgsql-performance
Thank each of you for your replies. I'm just beginning to understand the
scope of my opportunities.
Someone (I apologize, I forgot who) recently posted this query:
SELECT oid::regclass, reltuples, relpages
FROM pg_class
ORDER BY 3 DESC
Though the application is a relatively low-volume TP system, it is
structured a lot like a data warehouse with one primary table that
everything else hangs off. What the query above shows is that my largest
table, at 34 million rows, takes almost 1.4 million pages or 10+ Gb if my
math is good. The same table has 14 indexes, totaling another 12Gb. All
this is running on a box with 4Gb of memory.
So what I believe I see happening is that almost every query is clearing out
memory to load the particular index it needs. Hence my "first queries are
the fastest" observation at the beginning of this thread.
There are certainly design improvements to be done, but I've already started
the process of getting the memory increased on our production db server. We
are btw running 8.1 beta 3.
""Steinar H. Gunderson"" <sgunderson@bigfoot.com> wrote in message
news:20051024153248.GA24601@samfundet.no...
> On Mon, Oct 24, 2005 at 11:09:55AM -0400, Alex Turner wrote:
>> Just to play devils advocate here for as second, but if we have an
>> algorithm
>> that is substational better than just plain old LRU, which is what I
>> believe
>> the kernel is going to use to cache pages (I'm no kernel hacker), then
>> why
>> don't we apply that and have a significantly larger page cache a la
>> Oracle?
>
> There have (AFAIK) been reports of setting huge amounts of shared_buffers
> (close to the total amount of RAM) performing much better in 8.1 than in
> earlier versions, so this might actually be okay these days.
>
> I haven't heard of anybody reporting increase setting such values, though.
>
> /* Steinar */
> --
> Homepage: http://www.sesse.net/
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster
>
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: What gets cached?
@ 2005-10-27 22:48 Jim C. Nasby <jnasby@pervasive.com>
parent: PostgreSQL <martin@portant.com>
0 siblings, 0 replies; 9+ messages in thread
From: Jim C. Nasby @ 2005-10-27 22:48 UTC (permalink / raw)
To: PostgreSQL <martin@portant.com>; +Cc: pgsql-performance
Did the patch that allows multiple seqscans to piggyback on each other
make it into 8.1? It might help in this situation.
BTW, if a query requires loading more than a few percent of an index
PostgreSQL will usually go with a sequential scan instead. You should
check explain/explain analyze on your queries and see what's actually
happening. If you've got stats turned on you can also look at
pg_stat_user_indexes to get a better idea of what indexes are and aren't
being used.
On Thu, Oct 27, 2005 at 03:41:10PM -0500, PostgreSQL wrote:
> Thank each of you for your replies. I'm just beginning to understand the
> scope of my opportunities.
>
> Someone (I apologize, I forgot who) recently posted this query:
> SELECT oid::regclass, reltuples, relpages
> FROM pg_class
> ORDER BY 3 DESC
>
> Though the application is a relatively low-volume TP system, it is
> structured a lot like a data warehouse with one primary table that
> everything else hangs off. What the query above shows is that my largest
> table, at 34 million rows, takes almost 1.4 million pages or 10+ Gb if my
> math is good. The same table has 14 indexes, totaling another 12Gb. All
> this is running on a box with 4Gb of memory.
>
> So what I believe I see happening is that almost every query is clearing out
> memory to load the particular index it needs. Hence my "first queries are
> the fastest" observation at the beginning of this thread.
>
> There are certainly design improvements to be done, but I've already started
> the process of getting the memory increased on our production db server. We
> are btw running 8.1 beta 3.
>
> ""Steinar H. Gunderson"" <sgunderson@bigfoot.com> wrote in message
> news:20051024153248.GA24601@samfundet.no...
> > On Mon, Oct 24, 2005 at 11:09:55AM -0400, Alex Turner wrote:
> >> Just to play devils advocate here for as second, but if we have an
> >> algorithm
> >> that is substational better than just plain old LRU, which is what I
> >> believe
> >> the kernel is going to use to cache pages (I'm no kernel hacker), then
> >> why
> >> don't we apply that and have a significantly larger page cache a la
> >> Oracle?
> >
> > There have (AFAIK) been reports of setting huge amounts of shared_buffers
> > (close to the total amount of RAM) performing much better in 8.1 than in
> > earlier versions, so this might actually be okay these days.
> >
> > I haven't heard of anybody reporting increase setting such values, though.
> >
> > /* Steinar */
> > --
> > Homepage: http://www.sesse.net/
> >
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 2: Don't 'kill -9' the postmaster
> >
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly
>
--
Jim C. Nasby, Sr. Engineering Consultant jnasby@pervasive.com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461
^ permalink raw reply [nested|flat] 9+ messages in thread
end of thread, other threads:[~2005-10-27 22:48 UTC | newest]
Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2005-10-21 12:34 What gets cached? Martin Nickel <martin@portant.com>
2005-10-21 12:40 ` Steinar H. Gunderson <sgunderson@bigfoot.com>
2005-10-21 15:19 ` Alex Turner <armtuk@gmail.com>
2005-10-21 15:59 ` Michael Fuhr <mike@fuhr.org>
2005-10-21 21:44 ` Neil Conway <neilc@samurai.com>
2005-10-24 15:09 ` Alex Turner <armtuk@gmail.com>
2005-10-24 15:32 ` Steinar H. Gunderson <sgunderson@bigfoot.com>
2005-10-27 20:41 ` PostgreSQL <martin@portant.com>
2005-10-27 22:48 ` Jim C. Nasby <jnasby@pervasive.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox