Message-ID: <508EAC0B.7010609@optionshouse.com>
Date: Mon, 29 Oct 2012 11:17:15 -0500
From: Shaun Thomas <sthomas@optionshouse.com>
Reply-To: <sthomas@optionshouse.com>
Organization: Peak6
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
	rv:16.0) Gecko/20121011 Thunderbird/16.0.1
MIME-Version: 1.0
To: =?UTF-8?B?0JLRltGC0LDQu9GW0Lkg0KLQuNC80YfQuNGI0LjQvQ==?=
 <tivv00@gmail.com>
CC: "pgsql-performance@postgresql.org" <pgsql-performance@postgresql.org>
Subject: Re: Tons of free RAM. Can't make it go away.
References: <508583E4.8080102@optionshouse.com>
	<CABWW-d37ut8uWFDvLrv9Pa9wwiX2425F3bn4cVJXwt-jj2pJRg@mail.gmail.com>
In-Reply-To: 
 <CABWW-d37ut8uWFDvLrv9Pa9wwiX2425F3bn4cVJXwt-jj2pJRg@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Length: 1530
Content-Transfer-Encoding: quoted-printable

On 10/27/2012 10:49 PM, =D0=92=D1=96=D1=82=D0=B0=D0=BB=D1=96=D0=B9 =D0=A2=D0=B8=D0=BC=D1=87=D0=B8=D1=88=D0=B8=D0=BD wrote:

> It can be that some query(s) use a lot of work mem, either because of
> high work_mem setting or because of planner error. In this case the
> moment query runs it will need memory that will later be returned and
> become free. Usually this can be seen as active memory spike with a lot
> of free memory after.

Yeah, I had briefly considered that. But our work-mem is only 16MB, and 
even a giant query would have trouble allocating 10+GB with that size of 
work-mem buckets.

That's why I later listed the numa info. In our case, processor 0 is 
heavily unbalanced with its memory accesses compared to processor 1. I 
think the theory that we didn't start with interleave put an 8GB (our 
shared_buffers) segment all on processor 0, which unbalanced a lot of 
other stuff.

Of course, that leaves 4-6GB unaccounted for. And numactl still shows a 
heavy preference for freeing memory from proc 0. It seems to only do it 
on this node, so we're going to switch nodes soon and see if the problem 
reappears. We may have to perform a node hardware audit if this persists.

Thanks for your input, though. :)

-- 
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-444-8534
sthomas@optionshouse.com

______________________________________________

See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email