Received: from magus.postgresql.org (magus.postgresql.org [87.238.57.229]) by mail.postgresql.org (Postfix) with ESMTP id 3E5DA16D90F8 for ; Tue, 27 Mar 2012 17:06:45 -0300 (ADT) Received: from smtp.01.com ([199.36.142.181]) by magus.postgresql.org with esmtp (Exim 4.72) (envelope-from ) id 1SCcf4-00084X-KI for pgsql-performance@postgresql.org; Tue, 27 Mar 2012 20:06:42 +0000 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp-out-1.01.com (Postfix) with ESMTP id 298453F54F3 for ; Tue, 27 Mar 2012 15:06:18 -0500 (CDT) X-Virus-Scanned: amavisd-new at smtp-out-1.01.com Received: from smtp.01.com ([127.0.0.1]) by localhost (smtp-out-1.01.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3+ByKkbNf39B for ; Tue, 27 Mar 2012 15:06:18 -0500 (CDT) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp-out-1.01.com (Postfix) with ESMTP id 080123F5926 for ; Tue, 27 Mar 2012 15:06:18 -0500 (CDT) Received: from mail-1.01.com (mail.01.com [172.18.30.178]) by smtp-out-1.01.com (Postfix) with ESMTP id DB1BF3F57D6 for ; Tue, 27 Mar 2012 15:06:17 -0500 (CDT) Date: Tue, 27 Mar 2012 15:06:17 -0500 (CDT) From: Joshua Berkus To: postgres performance list Message-ID: <160356272.159746.1332878777645.JavaMail.root@mail-1.01.com> In-Reply-To: <958857849.158900.1332878339328.JavaMail.root@mail-1.01.com> Subject: Linux machine aggressively clearing cache MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [72.177.2.103] X-Mailer: Zimbra 7.1.3_GA_3374 (ZimbraWebClient - FF3.0 (Linux)/7.1.3_GA_3346) X-Pg-Spam-Score: -1.9 (-) X-Archive-Number: 201203/168 X-Sequence-Number: 46618 Have run across some memory behavior on Linux I've never seen before. Server running RHEL6 with 96GB of RAM. Kernel 2.6.32 PostgreSQL 9.0 208GB database with fairly random accesses over 50% of the database. Now, here's the weird part: even after a week of uptime, only 21 to 25GB of cache is ever used, and there's constantly 20GB to 35GB free memory. This would mean a small working set, except that we see constant reads from disk (1 to 15MB/s) and around 1/3 of queries are slowed by iowaits. In an effort to test this, we deliberately ran a pg_dump. This did grow the cache to all available memory, but Linux rapidly cleared the cache (flushing to disk) down to 25GB within an hour. sys.kernel.vm parameters are all defaults. None of the parameters seem to specifically relate to the size of the page cache. Has anyone ever seen this before? What did you do about it? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com San Francisco