pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedshared_buffers vs Linux file cache
6+ messages / 5 participants
[nested] [flat]
* shared_buffers vs Linux file cache
@ 2015-01-15 11:30 Huan Ruan <huan.ruan.it@gmail.com>
2015-01-15 17:10 ` Re: shared_buffers vs Linux file cache Glyn Astill <glynastill@yahoo.co.uk>
2015-01-15 22:22 ` Re: shared_buffers vs Linux file cache Jeff Janes <jeff.janes@gmail.com>
0 siblings, 2 replies; 6+ messages in thread
From: Huan Ruan @ 2015-01-15 11:30 UTC (permalink / raw)
To: pgsql-performance
Hi All
I thought 'shared_buffers' sets how much memory that is dedicated to
PostgreSQL to use for caching data, therefore not available to other
applications.
However, as shown in the following screenshots, The server (CentOS 6.6
64bit) has 64GB of RAM, and 'shared_buffer' is set to 32GB, but the
free+buffer+cache is 60GB.
Shouldn't the maximum value for free+buffer+cache be 32GB ( 64 - 32)?
Is 'shared_buffers' pre allocated to Postgres, and Postgres only?
Thanks
Huan
[image: Inline images 2]
[image: Inline images 1]
Attachments:
image.png (image/png, 4.2K) download
view image
image.png (image/png, 16.5K) download
view image
* Re: shared_buffers vs Linux file cache
2015-01-15 11:30 shared_buffers vs Linux file cache Huan Ruan <huan.ruan.it@gmail.com>
@ 2015-01-15 17:10 ` Glyn Astill <glynastill@yahoo.co.uk>
1 sibling, 0 replies; 6+ messages in thread
From: Glyn Astill @ 2015-01-15 17:10 UTC (permalink / raw)
To: Huan Ruan <huan.ruan.it@gmail.com>; pgsql-performance
> From: Huan Ruan <huan.ruan.it@gmail.com>
>To: pgsql-performance@postgresql.org
>Sent: Thursday, 15 January 2015, 11:30
>Subject: [PERFORM] shared_buffers vs Linux file cache
>
>
>
>Hi All
>
>
>I thought 'shared_buffers' sets how much memory that is dedicated to PostgreSQL to use for caching data, therefore not available to other applications.
>
>
>However, as shown in the following screenshots, The server (CentOS 6.6 64bit) has 64GB of RAM, and 'shared_buffer' is set to 32GB, but the free+buffer+cache is 60GB.
>
>
>Shouldn't the maximum value for free+buffer+cache be 32GB ( 64 - 32)?
>Is 'shared_buffers' pre allocated to Postgres, and Postgres only?
>
I've not looked at the images, but I think you're getting PostgreSQL shared_buffers and the OS buffercache mixed up; they are not the same.
PostgreSQL shared_buffers is specific to postgres, whereas the OS buffercache will just use free memory to cache data pages from disk, and this is what you're seeing.
Some reading for you:
http://www.tldp.org/LDP/sag/html/buffer-cache.html
Glyn
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: shared_buffers vs Linux file cache
2015-01-15 11:30 shared_buffers vs Linux file cache Huan Ruan <huan.ruan.it@gmail.com>
@ 2015-01-15 22:22 ` Jeff Janes <jeff.janes@gmail.com>
2015-01-15 23:24 ` Re: shared_buffers vs Linux file cache Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 1 reply; 6+ messages in thread
From: Jeff Janes @ 2015-01-15 22:22 UTC (permalink / raw)
To: Huan Ruan <huan.ruan.it@gmail.com>; +Cc: pgsql-performance
On Thu, Jan 15, 2015 at 3:30 AM, Huan Ruan <huan.ruan.it@gmail.com> wrote:
> Hi All
>
> I thought 'shared_buffers' sets how much memory that is dedicated to
> PostgreSQL to use for caching data, therefore not available to other
> applications.
>
> However, as shown in the following screenshots, The server (CentOS 6.6
> 64bit) has 64GB of RAM, and 'shared_buffer' is set to 32GB, but the
> free+buffer+cache is 60GB.
>
> Shouldn't the maximum value for free+buffer+cache be 32GB ( 64 - 32)?
> Is 'shared_buffers' pre allocated to Postgres, and Postgres only?
>
While PostgreSQL has reserves the right to use 32GB, as long as PostgreSQL
has not actually dirtied that RAM yet, then the kernel is free to keep
using it to cache files.
Cheers,
Jeff
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: shared_buffers vs Linux file cache
2015-01-15 11:30 shared_buffers vs Linux file cache Huan Ruan <huan.ruan.it@gmail.com>
2015-01-15 22:22 ` Re: shared_buffers vs Linux file cache Jeff Janes <jeff.janes@gmail.com>
@ 2015-01-15 23:24 ` Tom Lane <tgl@sss.pgh.pa.us>
2015-01-16 03:32 ` Re: shared_buffers vs Linux file cache Huan Ruan <huan.ruan.it@gmail.com>
0 siblings, 1 reply; 6+ messages in thread
From: Tom Lane @ 2015-01-15 23:24 UTC (permalink / raw)
To: Jeff Janes <jeff.janes@gmail.com>; +Cc: Huan Ruan <huan.ruan.it@gmail.com>; pgsql-performance
Jeff Janes <jeff.janes@gmail.com> writes:
> On Thu, Jan 15, 2015 at 3:30 AM, Huan Ruan <huan.ruan.it@gmail.com> wrote:
>> I thought 'shared_buffers' sets how much memory that is dedicated to
>> PostgreSQL to use for caching data, therefore not available to other
>> applications.
> While PostgreSQL has reserves the right to use 32GB, as long as PostgreSQL
> has not actually dirtied that RAM yet, then the kernel is free to keep
> using it to cache files.
Another thing to keep in mind is that, even if Postgres *has* used the
RAM, the kernel might decide to swap parts of it out if it's not being
used heavily. This is pretty disastrous from a performance standpoint,
so it's advisable to not make shared_buffers very much larger than what
your application will keep "hot".
Ideally we'd lock the shared buffer arena into RAM to prevent that,
but such facilities are often unavailable or restricted to root.
regards, tom lane
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: shared_buffers vs Linux file cache
2015-01-15 11:30 shared_buffers vs Linux file cache Huan Ruan <huan.ruan.it@gmail.com>
2015-01-15 22:22 ` Re: shared_buffers vs Linux file cache Jeff Janes <jeff.janes@gmail.com>
2015-01-15 23:24 ` Re: shared_buffers vs Linux file cache Tom Lane <tgl@sss.pgh.pa.us>
@ 2015-01-16 03:32 ` Huan Ruan <huan.ruan.it@gmail.com>
2015-01-16 07:49 ` Re: shared_buffers vs Linux file cache Roman Konoval <rkonoval@gmail.com>
0 siblings, 1 reply; 6+ messages in thread
From: Huan Ruan @ 2015-01-16 03:32 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Jeff Janes <jeff.janes@gmail.com>; pgsql-performance
Thanks very much, Glyn, Jeff, and Tom. That was very clearly explained.
A related case, see the following top dump. The Postgres process is using
87g residential memory, which I thought was the physical memory consumed by
a process that can't be shared with others. While, the free+cached is about
155gb. But, (87 + 155) is bigger than the total available 198g RAM. Does
this mean some of the residential memory used by Postgres is actually
shareable to others?
>> Mem: 198311880k total, 183836408k used, 14475472k free, 8388k buffers
>> Swap: 4194300k total, 314284k used, 3880016k free, 141105408k cached
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 15338 postgres 20 0 97.9g 87g 87g S 0.3 46.4 21:47.44
>> postgres: checkpointer process
>> 27473 postgres 20 0 98.1g 29g 29g S 0.0 15.8 2:14.93
>> postgres: xxxx idle
>> 4710 postgres 20 0 98.1g 24g 23g S 0.0 12.7 1:17.41
>> postgres: xxxx idle
>> 26587 postgres 20 0 98.0g 15g 15g S 0.0 8.0 1:21.24
>
>
^ permalink raw reply [nested|flat] 6+ messages in thread
* Re: shared_buffers vs Linux file cache
2015-01-15 11:30 shared_buffers vs Linux file cache Huan Ruan <huan.ruan.it@gmail.com>
2015-01-15 22:22 ` Re: shared_buffers vs Linux file cache Jeff Janes <jeff.janes@gmail.com>
2015-01-15 23:24 ` Re: shared_buffers vs Linux file cache Tom Lane <tgl@sss.pgh.pa.us>
2015-01-16 03:32 ` Re: shared_buffers vs Linux file cache Huan Ruan <huan.ruan.it@gmail.com>
@ 2015-01-16 07:49 ` Roman Konoval <rkonoval@gmail.com>
0 siblings, 0 replies; 6+ messages in thread
From: Roman Konoval @ 2015-01-16 07:49 UTC (permalink / raw)
To: Huan Ruan <huan.ruan.it@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Jeff Janes <jeff.janes@gmail.com>; pgsql-performance
Huan,
Residential memory is part of the process memory that is now swapped and is
in RAM. This includes also memory shared with other processes so sum of RES
for all processes may be greater that total physical memory.
I recommend this article
http://www.depesz.com/2012/06/09/how-much-ram-is-postgresql-using/ to get
better understanding how linux manages memory.
Regards,
Roman
On Fri, Jan 16, 2015 at 5:32 AM, Huan Ruan <huan.ruan.it@gmail.com> wrote:
> Thanks very much, Glyn, Jeff, and Tom. That was very clearly explained.
>
> A related case, see the following top dump. The Postgres process is using
> 87g residential memory, which I thought was the physical memory consumed by
> a process that can't be shared with others. While, the free+cached is about
> 155gb. But, (87 + 155) is bigger than the total available 198g RAM. Does
> this mean some of the residential memory used by Postgres is actually
> shareable to others?
>
>
> >> Mem: 198311880k total, 183836408k used, 14475472k free, 8388k buffers
> >> Swap: 4194300k total, 314284k used, 3880016k free, 141105408k cached
> >>
> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> >> 15338 postgres 20 0 97.9g 87g 87g S 0.3 46.4 21:47.44
> >> postgres: checkpointer process
> >> 27473 postgres 20 0 98.1g 29g 29g S 0.0 15.8 2:14.93
> >> postgres: xxxx idle
> >> 4710 postgres 20 0 98.1g 24g 23g S 0.0 12.7 1:17.41
> >> postgres: xxxx idle
>
> >> 26587 postgres 20 0 98.0g 15g 15g S 0.0 8.0 1:21.24
>
>
>>
>>
>
>
>
^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2015-01-16 07:49 UTC | newest]
Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2015-01-15 11:30 shared_buffers vs Linux file cache Huan Ruan <huan.ruan.it@gmail.com>
2015-01-15 17:10 ` Glyn Astill <glynastill@yahoo.co.uk>
2015-01-15 22:22 ` Jeff Janes <jeff.janes@gmail.com>
2015-01-15 23:24 ` Tom Lane <tgl@sss.pgh.pa.us>
2015-01-16 03:32 ` Huan Ruan <huan.ruan.it@gmail.com>
2015-01-16 07:49 ` Roman Konoval <rkonoval@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox