X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 5DC1BEAD7E2; Fri, 22 Oct 2004 21:09:51 +0100 (BST) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 48953-07; Fri, 22 Oct 2004 20:09:50 +0000 (GMT) Received: from is.rice.edu (is.rice.edu [128.42.42.24]) by svr1.postgresql.org (Postfix) with ESMTP id C7DD1EAE493; Fri, 22 Oct 2004 21:09:41 +0100 (BST) Received: from localhost (localhost [127.0.0.1]) by localhost.is.rice.edu (Postfix) with ESMTP id 85679419ED; Fri, 22 Oct 2004 15:10:01 -0500 (CDT) Received: from is.rice.edu ([127.0.0.1]) by localhost (it.is.rice.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 02747-02; Fri, 22 Oct 2004 15:09:55 -0500 (CDT) Received: by is.rice.edu (Postfix, from userid 18612) id C5AB6419C1; Fri, 22 Oct 2004 15:09:55 -0500 (CDT) Date: Fri, 22 Oct 2004 15:09:55 -0500 From: Kenneth Marshall To: Jan Wieck Cc: Simon Riggs , pgsql-patches@postgresql.org, pgsql-hackers@postgresql.org, josh@agliodbs.com Subject: Re: ARC Memory Usage analysis Message-ID: <20041022200955.GC4382@it.is.rice.edu> References: <1098471059.20926.49.camel@localhost.localdomain> <41796115.2020501@Yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <41796115.2020501@Yahoo.com> User-Agent: Mutt/1.4.2i X-Virus-Scanned: by amavis-20030314-p2 at is.rice.edu X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.0 tagged_above=0.0 required=5.0 tests= X-Spam-Level: X-Archive-Number: 200410/802 X-Sequence-Number: 60305 On Fri, Oct 22, 2004 at 03:35:49PM -0400, Jan Wieck wrote: > On 10/22/2004 2:50 PM, Simon Riggs wrote: > > >I've been using the ARC debug options to analyse memory usage on the > >PostgreSQL 8.0 server. This is a precursor to more complex performance > >analysis work on the OSDL test suite. > > > >I've simplified some of the ARC reporting into a single log line, which > >is enclosed here as a patch on freelist.c. This includes reporting of: > >- the total memory in use, which wasn't previously reported > >- the cache hit ratio, which was slightly incorrectly calculated > >- a useful-ish value for looking at the "B" lists in ARC > >(This is a patch against cvstip, but I'm not sure whether this has > >potential for inclusion in 8.0...) > > > >The total memory in use is useful because it allows you to tell whether > >shared_buffers is set too high. If it is set too high, then memory usage > >will continue to grow slowly up to the max, without any corresponding > >increase in cache hit ratio. If shared_buffers is too small, then memory > >usage will climb quickly and linearly to its maximum. > > > >The last one I've called "turbulence" in an attempt to ascribe some > >useful meaning to B1/B2 hits - I've tried a few other measures though > >without much success. Turbulence is the hit ratio of B1+B2 lists added > >together. By observation, this is zero when ARC gives smooth operation, > >and goes above zero otherwise. Typically, turbulence occurs when > >shared_buffers is too small for the working set of the database/workload > >combination and ARC repeatedly re-balances the lengths of T1/T2 as a > >result of "near-misses" on the B1/B2 lists. Turbulence doesn't usually > >cut in until the cache is fully utilized, so there is usually some delay > >after startup. > > > >We also recently discussed that I would add some further memory analysis > >features for 8.1, so I've been trying to figure out how. > > > >The idea that B1, B2 represent something really useful doesn't seem to > >have been borne out - though I'm open to persuasion there. > > > >I originally envisaged a "shadow list" operating in extension of the > >main ARC list. This will require some re-coding, since the variables and > >macros are all hard-coded to a single set of lists. No complaints, just > >it will take a little longer than we all thought (for me, that is...) > > > >My proposal is to alter the code to allow an array of memory linked > >lists. The actual list would be [0] - other additional lists would be > >created dynamically as required i.e. not using IFDEFs, since I want this > >to be controlled by a SIGHUP GUC to allow on-site tuning, not just lab > >work. This will then allow reporting against the additional lists, so > >that cache hit ratios can be seen with various other "prototype" > >shared_buffer settings. > > All the existing lists live in shared memory, so that dynamic approach > suffers from the fact that the memory has to be allocated during ipc_init. > > What do you think about my other theory to make C actually 2x effective > cache size and NOT to keep T1 in shared buffers but to assume T1 lives > in the OS buffer cache? > > > Jan > Jan,