From: David Boreham <david_list@boreham.org>
To: pgsql-performance@postgresql.org
Subject: Re: measuring shared memory usage on Windows
Date: Fri, 20 Oct 2006 08:33:05 -0600
Message-ID: <4538DE21.9010200@boreham.org> (raw)
In-Reply-To: <45370607.4080308@paradise.net.nz>
References: <7be3f35d0610160215v21edbe48vc144200a83f41dd0@mail.gmail.com>
<4533815E.3050301@boreham.org>
<7be3f35d0610161312r3ff84fcy4dcb85e9c71fa1e2@mail.gmail.com>
<7be3f35d0610161313y7d9b5a6ewff090db6fb25585@mail.gmail.com>
<45370607.4080308@paradise.net.nz>
Unfortunately often operating system virtual memory
and filesystem caching code that does exactly the opposite of
what a database application would like.
For some reason the kernel guys don't see it that way ;)
Over the years there have been various kernel features added
with the overall goal of solving problems in this area : O_DIRECT,
ckrm, flags to mmap() and so on. So far I'm not sure any of
them has really succeeded. Hence the real-world wisdom to
'let the filesystem cache do its thing' and configure a small
shared memory cache in the application.
Ideally one would want to use O_DIRECT or equivalent
to bypass the OS's cleverness and manage the filesystem
caching oneself. However it turns out that enabling O_DIRECT
makes things much worse not better (YMMV obviously).
It's hard to achieve the level of concurrency that the kernel
can get for disk I/O, from user mode.
Another approach is to make the application cache size
dynamic, with the goal that it can grow and shrink to
reach the size that provides the best overall performance.
I've seen attempts to drive the sizing using memory access
latency measurements done from user mode inside the application.
However I'm not sure that anyone has taken this approach
beyond the science project stage.
So AFAIK this is still a generally unsolved problem.
NT (Windows) is particularly interesting because it drives the
filesystem cache sizing with a signal that it mesures from the
VM pages evicted per second counter. In order to keep its
feedback loop stable, the OS wants to see a non-zero value
for this signal at all times. So you will see that even under
ideal conditions the system will still page a little.
(Unless that code has changed in Win2003 -- it's been a
while since I checked). So don't drive yourself crazy trying
to get it to stop paging ;)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: david_list@boreham.org
Subject: Re: measuring shared memory usage on Windows
In-Reply-To: <4538DE21.9010200@boreham.org>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox