Received: from localhost (maia-3.hub.org [200.46.204.184]) by postgresql.org (Postfix) with ESMTP id 92E309FA557 for ; Tue, 28 Aug 2007 21:36:47 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.184]) (amavisd-maia, port 10024) with ESMTP id 34264-01 for ; Tue, 28 Aug 2007 21:36:35 -0300 (ADT) X-Greylist: delayed 00:05:08.202601 by SQLgrey-1.7.5 Received: from mout.perfora.net (mout.perfora.net [74.208.4.194]) by postgresql.org (Postfix) with ESMTP id 827B69FA5B6 for ; Tue, 28 Aug 2007 21:36:38 -0300 (ADT) Received: from pustefix136.perfora.net [172.19.56.140] (helo=pustefix136.perfora.net) by mrelay.perfora.net (node=mrus1) with ESMTP (Nemesis), id 0MKpCa-1IQBSx3noy-0007VS; Tue, 28 Aug 2007 20:31:28 -0400 Message-Id: <29896230.882031188347487962.JavaMail.servlet@perfora> From: jallgood@the-allgoods.net To: Subject: Re: How to monitor resources on Linux. Cc: Cc: Cc: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Priority: 3 X-Binford: 6100 (more power) X-Mailer: Webmail X-Originating-From: 32485842 X-Routing: US X-Message-Id: <32485842$1188347487913172.19.56.1403780632@pustefix136.perfora.net-539324407> X-Received: from pustefix136.perfora.net by 68.209.119.52 with HTTP id 32485842 for [tgl@sss.pgh.pa.us]; Tue, 28 Aug 2007 20:31:27 EDT Date: Tue, 28 Aug 2007 20:31:27 -0400 X-Provags-ID: V01U2FsdGVkX1+jVToGO89NHNWNXwOxLARtaXeu0vz+IZ7qoka fSsimqsH7TD5QTTK2VdOuDjlaq5oZGwC4h6Vnc2Q4Xaxkzzsjs 6OD4kmF3fk= X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200708/375 X-Sequence-Number: 26706 Hello Again This is interesting. We are running a 32bit kernel. I will send another output from free in the morning when the load on the server is greater. We have been using this equipment since April and because of another issue which required reboots everynight. This is really the first time that we have had any uptime on this machine. It was up for 18days. Before we started having any issues. The output from free -l -m. I believe the High and Low are like watermarks for lack of another word. Thanks John R Allgood writes: > Here is the output from free on one of the nodes. Hmmm ... I'm not exactly a kernel jock, but I find the presence of these lines in the output to be mighty suspicious: > Low: 821 510 310 > High: 7294 5459 1835 In a true 64-bit system you should not have any distinction between low and high memory (and indeed "free" doesn't print any such thing on my x86_64 box). Maybe you are running a 32-bit kernel? Anyway it seems likely that the out-of-memory situation was due to oversubscribed lowmem rather than being out of memory globally, especially since you are running at zero swap usage. This is not the trace of a system that's under any memory pressure overall: > total used free > Mem: 8116 5969 2146 > -/+ buffers/cache: 1506 6609 > Swap: 2000 0 1999 Something else to check is whether having swap only a quarter the size of physical RAM is a good idea or not. I'm not sure what the latest configuration theories are for Linux, but the old-school-Unix theory was always that you should have more swap space than RAM. When memory overcommit is disabled, having plenty of swap space available may be necessary even if you're seemingly not using it --- the kernel needs to be sure that there would be someplace to put everything if it had to materialize all the virtual copy-on-write pages that the current process set is sharing. regards, tom lane