X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 017FE329FAF for ; Mon, 18 Oct 2004 22:19:49 +0100 (BST) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 31757-06 for ; Mon, 18 Oct 2004 21:19:38 +0000 (GMT) Received: from smtp018.mail.yahoo.com (smtp018.mail.yahoo.com [216.136.174.115]) by svr1.postgresql.org (Postfix) with SMTP id 53ADB329EBB for ; Mon, 18 Oct 2004 22:19:38 +0100 (BST) Received: from unknown (HELO jupiter.black-lion.info) (janwieck@68.80.245.191 with login) by smtp018.mail.yahoo.com with SMTP; 18 Oct 2004 21:19:37 -0000 Received: from [172.21.8.18] (ismtp.afilias.com [216.217.55.254]) (authenticated bits=0) by jupiter.black-lion.info (8.12.10/8.12.9) with ESMTP id i9ILJT94032117; Mon, 18 Oct 2004 17:19:30 -0400 (EDT) (envelope-from JanWieck@Yahoo.com) Message-ID: <41743355.9060005@Yahoo.com> Date: Mon, 18 Oct 2004 17:19:17 -0400 From: Jan Wieck User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.1) Gecko/20040707 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Jan Wieck Cc: simon@2ndquadrant.com, Tom Lane , josh@agliodbs.com, pgsql-hackers@postgresql.org, pgsql-performance@postgresql.org Subject: Autotuning of shared buffer size (was: Re: Getting rid of AtEOXact Buffers (was Re: [Testperf-general] Re: [PERFORM] First set of OSDL Shared Memscalability results, some wierdness ...)) References: <28292295$10980398474172c227e88c75.95384212@config16.schlund.de> <41742943.4050603@Yahoo.com> In-Reply-To: <41742943.4050603@Yahoo.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=1.6 tagged_above=0.0 required=5.0 tests=RCVD_IN_NJABL_DUL, RCVD_IN_SORBS_DUL X-Spam-Level: * X-Archive-Number: 200410/614 X-Sequence-Number: 60117 Trying to think a little out of the box, how "common" is it in modern operating systems to be able to swap out shared memory? Maybe we're not using the ARC algorithm correctly after all. The ARC algorithm does not consider the second level OS buffer cache in it's design. Maybe the total size of the ARC cache directory should not be 2x the size of what is configured as the shared buffer cache, but rather 2x the size of the effective cache size (in 8k pages). If we assume that the job of the T1 queue is better done by the OS buffers anyway (and this is what this discussion seems to point out), we shouldn't hold them in shared buffers (only very few of them and evict them ASAP). We just account for them and assume that the OS will have those cached that we find in our T1 directory. I think with the right configuration for effective cache size, this is a fair assumption. The T2 queue represents the frequently used blocks. If our implementation would cause the unused/used portions of the buffers not to move around, the OS will swap out currently unused portions of the shared buffer cache and utilize those as OS buffers. To verify this theory it would be interesting what the ARC strategy after a long DBT run with a "large" buffer cache thinks a good T2 size would be. Enabling the strategy debug message and running postmaster with -d1 will show that. In theory, this size should be anywhere near the sweet spot. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #