pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedLinux machine aggressively clearing cache
8+ messages / 6 participants
[nested] [flat]
* Linux machine aggressively clearing cache
@ 2012-03-27 20:06 Joshua Berkus <josh@agliodbs.com>
0 siblings, 2 replies; 8+ messages in thread
From: Joshua Berkus @ 2012-03-27 20:06 UTC (permalink / raw)
To: pgsql-performance
Have run across some memory behavior on Linux I've never seen before.
Server running RHEL6 with 96GB of RAM.
Kernel 2.6.32
PostgreSQL 9.0
208GB database with fairly random accesses over 50% of the database.
Now, here's the weird part: even after a week of uptime, only 21 to 25GB of cache is ever used, and there's constantly 20GB to 35GB free memory. This would mean a small working set, except that we see constant reads from disk (1 to 15MB/s) and around 1/3 of queries are slowed by iowaits.
In an effort to test this, we deliberately ran a pg_dump. This did grow the cache to all available memory, but Linux rapidly cleared the cache (flushing to disk) down to 25GB within an hour.
sys.kernel.vm parameters are all defaults. None of the parameters seem to specifically relate to the size of the page cache.
Has anyone ever seen this before? What did you do about it?
--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
San Francisco
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Linux machine aggressively clearing cache
@ 2012-03-27 20:10 Claudio Freire <klaussfreire@gmail.com>
parent: Joshua Berkus <josh@agliodbs.com>
1 sibling, 0 replies; 8+ messages in thread
From: Claudio Freire @ 2012-03-27 20:10 UTC (permalink / raw)
To: Joshua Berkus <josh@agliodbs.com>; +Cc: pgsql-performance
On Tue, Mar 27, 2012 at 5:06 PM, Joshua Berkus <josh@agliodbs.com> wrote:
> In an effort to test this, we deliberately ran a pg_dump. This did grow the cache to all available memory, but Linux rapidly cleared the cache (flushing to disk) down to 25GB within an hour.
This would happen if some queries (or some program) briefly uses that
much memory (pushing the cache off RAM).
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Linux machine aggressively clearing cache
@ 2012-03-27 20:40 Dave Crooke <dcrooke@gmail.com>
parent: Joshua Berkus <josh@agliodbs.com>
1 sibling, 1 reply; 8+ messages in thread
From: Dave Crooke @ 2012-03-27 20:40 UTC (permalink / raw)
To: Joshua Berkus <josh@agliodbs.com>; +Cc: pgsql-performance
This may just be a typo, but if you really did create write (dirty) block
device cache by writing the pg_dump file somewhere, then that is what it's
supposed to do ;) Linux is more aggressive about write cache and will allow
more of it to build up than e.g. HP-UX which will start to throttle
process-to-cache writes to avoid getting too far behind.
Read cache of course does not need to be flushed and can simply be dumped
when the memory is needed, and so Linux will keep more or less unlimited
amounts of read cache until it needs the memory for something else ....
here is an output from "free" on my laptop, showing ~2.5GB of read cache
that can be freed almost instantly if needed for process memory, write
cache, kernel buffers, etc. The -/+ line shows a net of what is being used
by processes.
dave:~$ free
total used free shared buffers cached
Mem: 8089056 7476424 612632 0 603508 2556584
-/+ buffers/cache: 4316332 3772724
Swap: 24563344 1176284 23387060
redirecting *pg_dump >/dev/null* will read the DB without writing
anything, but it's pretty resource intensive .... if you just want to get
the database tables into the OS read cache you can do it much more cheaply
with *sudo tar cvf - /var/lib/postgresql/8.4/main/base | cat >/dev/null *or
similar (GNU tar somehow detects if you connect its stdout directly to
/dev/null and then it cheats and doesn't do the reads)
In the second "free" output below, the kernel has grabbed what it can for
cache, leaving only ~64MB of actual free memory for instant use.
dave:~$ pg_dump -F c hyper9db >/dev/null
dave:~$ free
total used free shared buffers cached
Mem: 8089056 8024252 64804 0 287432 3797956
-/+ buffers/cache: 3938864 4150192
Swap: 24563344 1166556 23396788
dave:~$
Cheers
Dave
On Tue, Mar 27, 2012 at 3:06 PM, Joshua Berkus <josh@agliodbs.com> wrote:
> ... but Linux rapidly cleared the cache *(flushing to disk)* down to 25GB
> within an hour.
>
> --
> Josh Berkus
> PostgreSQL Experts Inc.
> http://pgexperts.com
> San Francisco
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Linux machine aggressively clearing cache
@ 2012-03-28 18:37 Joshua Berkus <josh@agliodbs.com>
parent: Dave Crooke <dcrooke@gmail.com>
0 siblings, 1 reply; 8+ messages in thread
From: Joshua Berkus @ 2012-03-28 18:37 UTC (permalink / raw)
To: Dave Crooke <dcrooke@gmail.com>; +Cc: pgsql-performance
> This may just be a typo, but if you really did create write (dirty)
> block device cache by writing the pg_dump file somewhere, then that
> is what it's supposed to do ;)
The pgdump was across the network. So the only caching on the machine was read caching.
> Read cache of course does not need to be flushed and can simply be
> dumped when the memory is needed, and so Linux will keep more or
> less unlimited amounts of read cache until it needs the memory for
> something else ....
Right, that's the normal behavior. Except not on this machine.
--Josh
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Linux machine aggressively clearing cache
@ 2012-03-31 00:51 Josh Berkus <josh@agliodbs.com>
parent: Joshua Berkus <josh@agliodbs.com>
0 siblings, 1 reply; 8+ messages in thread
From: Josh Berkus @ 2012-03-31 00:51 UTC (permalink / raw)
To: pgsql-performance
>> Read cache of course does not need to be flushed and can simply be
>> dumped when the memory is needed, and so Linux will keep more or
>> less unlimited amounts of read cache until it needs the memory for
>> something else ....
>
> Right, that's the normal behavior. Except not on this machine.
So this turned out to be a Linux kernel issue. Will document it on
www.databasesoup.com.
--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Linux machine aggressively clearing cache
@ 2012-04-12 15:47 Steve Crawford <scrawford@pinpointresearch.com>
parent: Josh Berkus <josh@agliodbs.com>
0 siblings, 1 reply; 8+ messages in thread
From: Steve Crawford @ 2012-04-12 15:47 UTC (permalink / raw)
To: Josh Berkus <josh@agliodbs.com>; +Cc: pgsql-performance
On 03/30/2012 05:51 PM, Josh Berkus wrote:
>
> So this turned out to be a Linux kernel issue. Will document it on
> www.databasesoup.com.
Anytime soon? About to build two PostgreSQL servers and wondering if you
have uncovered a kernel version or similar issue to avoid.
Cheers,
Steve
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Linux machine aggressively clearing cache
@ 2012-04-19 00:09 Josh Berkus <josh@agliodbs.com>
parent: Steve Crawford <scrawford@pinpointresearch.com>
0 siblings, 1 reply; 8+ messages in thread
From: Josh Berkus @ 2012-04-19 00:09 UTC (permalink / raw)
To: Steve Crawford <scrawford@pinpointresearch.com>; +Cc: pgsql-performance
On 4/12/12 8:47 AM, Steve Crawford wrote:
> On 03/30/2012 05:51 PM, Josh Berkus wrote:
>>
>> So this turned out to be a Linux kernel issue. Will document it on
>> www.databasesoup.com.
> Anytime soon? About to build two PostgreSQL servers and wondering if you
> have uncovered a kernel version or similar issue to avoid.
Yeah, I'll blog it.
--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Linux machine aggressively clearing cache
@ 2012-09-13 19:00 Ross Reedstrom <reedstrm@rice.edu>
parent: Josh Berkus <josh@agliodbs.com>
0 siblings, 0 replies; 8+ messages in thread
From: Ross Reedstrom @ 2012-09-13 19:00 UTC (permalink / raw)
To: pgsql-performance
On Wed, Apr 18, 2012 at 05:09:29PM -0700, Josh Berkus wrote:
> On 4/12/12 8:47 AM, Steve Crawford wrote:
> > On 03/30/2012 05:51 PM, Josh Berkus wrote:
> >>
> >> So this turned out to be a Linux kernel issue. Will document it on
> >> www.databasesoup.com.
> > Anytime soon? About to build two PostgreSQL servers and wondering if you
> > have uncovered a kernel version or similar issue to avoid.
>
> Yeah, I'll blog it.
Since I'm doing some backlog catchup, I'll do some community/archive service
and provide the link:
http://www.databasesoup.com/2012/04/red-hat-kernel-cache-clearing-issue.html
Ross
--
Ross Reedstrom, Ph.D. reedstrm@rice.edu
Systems Engineer & Admin, Research Scientist phone: 713-348-6166
Connexions http://cnx.org fax: 713-348-3665
Rice University MS-375, Houston, TX 77005
GPG Key fingerprint = F023 82C8 9B0E 2CC6 0D8E F888 D3AE 810E 88F0 BEDE
^ permalink raw reply [nested|flat] 8+ messages in thread
end of thread, other threads:[~2012-09-13 19:00 UTC | newest]
Thread overview: 8+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2012-03-27 20:06 Linux machine aggressively clearing cache Joshua Berkus <josh@agliodbs.com>
2012-03-27 20:10 ` Claudio Freire <klaussfreire@gmail.com>
2012-03-27 20:40 ` Dave Crooke <dcrooke@gmail.com>
2012-03-28 18:37 ` Joshua Berkus <josh@agliodbs.com>
2012-03-31 00:51 ` Josh Berkus <josh@agliodbs.com>
2012-04-12 15:47 ` Steve Crawford <scrawford@pinpointresearch.com>
2012-04-19 00:09 ` Josh Berkus <josh@agliodbs.com>
2012-09-13 19:00 ` Ross Reedstrom <reedstrm@rice.edu>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox