Received: from localhost (maia-4.hub.org [200.46.204.183]) by postgresql.org (Postfix) with ESMTP id 0629E9FA519 for ; Tue, 28 Aug 2007 18:06:40 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 35665-03 for ; Tue, 28 Aug 2007 18:06:35 -0300 (ADT) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.5 Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.238]) by postgresql.org (Postfix) with ESMTP id 81C359FA48B for ; Tue, 28 Aug 2007 18:06:28 -0300 (ADT) Received: by nz-out-0506.google.com with SMTP id l8so1319836nzf for ; Tue, 28 Aug 2007 14:06:27 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=KBiq2JyGgjSEFRjIRS/cx0InLIbKxQgyXFYpJ69JO3TRkAQWOiVzc2IhjDBkRauHld+yVQHHhhwslD0Uk0IFU5I0Dq68bdvPh2QlNtZRrqyv2T6qmrSlSnVGe3Brb5jyR9XBar7fh5GSX8rE2gv/4fIOzTSfomCizCokoDsng5A= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=f7ZRcsH1KWsuW8FnvYt0zjZrOwXpe9jCHfZOZov+vRFlajPQhJMz3lB1DB6Ys6wWfQ/GEUD446uzLX6QBOAVf08RKb9vbOxjFu7fR9lh4L1VS5pc56rjcCPRoa0u14t9vKNG2yzv+wfwNoS4oTZP/W7i44silax2wSSDDr44J6s= Received: by 10.141.52.5 with SMTP id e5mr2105511rvk.1188335186560; Tue, 28 Aug 2007 14:06:26 -0700 (PDT) Received: by 10.141.159.11 with HTTP; Tue, 28 Aug 2007 14:06:26 -0700 (PDT) Message-ID: Date: Tue, 28 Aug 2007 16:06:26 -0500 From: "Scott Marlowe" To: "John R Allgood" Subject: Re: How to monitor resources on Linux. Cc: "Andrew Sullivan" , pgsql-admin@postgresql.org In-Reply-To: <46D489B6.8050804@the-allgoods.net> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <46D4353F.2040209@the-allgoods.net> <815.1188324416@sss.pgh.pa.us> <46D470D8.2030605@the-allgoods.net> <20070828191143.GE13868@alvh.no-ip.org> <46D47A13.3060907@the-allgoods.net> <20070828195554.GB1683@phlogiston.dyndns.org> <46D489B6.8050804@the-allgoods.net> X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200708/367 X-Sequence-Number: 26698 On 8/28/07, John R Allgood wrote: > > Here is the output from free on one of the nodes. I have seen free mem go > as low as 15 and then go back up. Like I was saying earlier my concern was > why the kernel started killing my postmasters. Here is the kernel message > "kernel: oom-killer: gfp_mask=0xd0". This started happening when were > running our midday backup. After backup runs vacuum will start up right > after. We rebooted the servers last night and today backup and vacuum ran > fine. Below the free out put I have added the logging out of > /var/log/messages. Thanks this is a great list. > > i total used free shared > buffers cached > Mem: 8116 5969 2146 0 144 4318 > Low: 821 510 310 > High: 7294 5459 1835 > -/+ buffers/cache: 1506 6609 > Swap: 2000 0 1999 I'm assuming those numbers are in megabytes. If so, then they're pretty realistic. you've 2 Gig free, and 4 Gig cached, with 144 Meg buffer mem. Very reasonable. Have you got the output of free when things are going wrong? > ug 27 12:24:42 gan-lxc-01 kernel: Swap cache: add 2104, delete 2017, find > 829/1136, race 0+0 > Aug 27 12:24:42 gan-lxc-01 kernel: 1229 bounce buffer pages > Aug 27 12:24:42 gan-lxc-01 kernel: Free swap: 2047424kB > Aug 27 12:24:42 gan-lxc-01 kernel: 2260992 pages of RAM > Aug 27 12:24:42 gan-lxc-01 kernel: 1867512 pages of HIGHMEM > Aug 27 12:24:42 gan-lxc-01 kernel: 183273 reserved pages > Aug 27 12:24:42 gan-lxc-01 kernel: 942026 pages shared > Aug 27 12:24:42 gan-lxc-01 kernel: 87 pages swap cached > Aug 27 12:24:42 gan-lxc-01 kernel: Out of Memory: Killed process 19383 > (postmaster). IS there any other context to go with this, like something from the postgres logs at the same time, maybe top output sorted by memory... I'm wondering if you're just running a few queries that fire really big sorts off and that's what's getting you.