Received: from localhost (maia-1.hub.org [200.46.204.191]) by postgresql.org (Postfix) with ESMTP id 32CCD9FA295 for ; Tue, 28 Aug 2007 18:32:35 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.191]) (amavisd-maia, port 10024) with ESMTP id 47688-01 for ; Tue, 28 Aug 2007 18:32:30 -0300 (ADT) X-Greylist: from auto-whitelisted by SQLgrey-1.7.5 Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) by postgresql.org (Postfix) with ESMTP id EDA409F945E for ; Tue, 28 Aug 2007 18:32:29 -0300 (ADT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss.pgh.pa.us (8.14.1/8.14.1) with ESMTP id l7SLWQ2N004246; Tue, 28 Aug 2007 17:32:26 -0400 (EDT) To: John R Allgood cc: Scott Marlowe , Andrew Sullivan , pgsql-admin@postgresql.org Subject: Re: How to monitor resources on Linux. In-reply-to: <46D489B6.8050804@the-allgoods.net> References: <46D4353F.2040209@the-allgoods.net> <815.1188324416@sss.pgh.pa.us> <46D470D8.2030605@the-allgoods.net> <20070828191143.GE13868@alvh.no-ip.org> <46D47A13.3060907@the-allgoods.net> <20070828195554.GB1683@phlogiston.dyndns.org> <46D489B6.8050804@the-allgoods.net> Comments: In-reply-to John R Allgood message dated "Tue, 28 Aug 2007 16:46:46 -0400" Date: Tue, 28 Aug 2007 17:32:26 -0400 Message-ID: <4245.1188336746@sss.pgh.pa.us> From: Tom Lane X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200708/368 X-Sequence-Number: 26699 John R Allgood writes: > Here is the output from free on one of the nodes. Hmmm ... I'm not exactly a kernel jock, but I find the presence of these lines in the output to be mighty suspicious: > Low: 821 510 310 > High: 7294 5459 1835 In a true 64-bit system you should not have any distinction between low and high memory (and indeed "free" doesn't print any such thing on my x86_64 box). Maybe you are running a 32-bit kernel? Anyway it seems likely that the out-of-memory situation was due to oversubscribed lowmem rather than being out of memory globally, especially since you are running at zero swap usage. This is not the trace of a system that's under any memory pressure overall: > total used free > Mem: 8116 5969 2146 > -/+ buffers/cache: 1506 6609 > Swap: 2000 0 1999 Something else to check is whether having swap only a quarter the size of physical RAM is a good idea or not. I'm not sure what the latest configuration theories are for Linux, but the old-school-Unix theory was always that you should have more swap space than RAM. When memory overcommit is disabled, having plenty of swap space available may be necessary even if you're seemingly not using it --- the kernel needs to be sure that there would be someplace to put everything if it had to materialize all the virtual copy-on-write pages that the current process set is sharing. regards, tom lane