Feedback-ID: id4a34324:Fastmail
Date: Wed, 20 Jul 2022 09:34:24 -0700
From: Andres Freund <andres@anarazel.de>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Melanie Plageman <melanieplageman@gmail.com>,
	Greg Stark <stark@mit.edu>,
	Kyotaro Horiguchi <horikyota.ntt@gmail.com>,
	Justin Pryzby <pryzby@telsasoft.com>,
	Thomas Munro <thomas.munro@gmail.com>,
	"David G. Johnston" <david.g.johnston@gmail.com>,
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: shared-memory based stats collector - v70
Message-ID: <20220720163424.ag2cdhruhbwkneym@alap3.anarazel.de>
References: 
 <CAAKRu_YsFzW3jsQUD7MVUY8c+87OiKW_Y93TsPAfLr3zi_jX9w@mail.gmail.com>
 <20220325.142256.1649716600998659084.horikyota.ntt@gmail.com>
 <20220325.172418.1055357550509939415.horikyota.ntt@gmail.com>
 <20220402082102.ojdd2e6cobsmljvh@alap3.anarazel.de>
 <20220404041516.cctrvpadhuriawlq@alap3.anarazel.de>
 <20220405030506.lfdhbu5zf4tzdpux@alap3.anarazel.de>
 <20220406030008.2qxipjxo776dwnqs@alap3.anarazel.de>
 <CAM-w4HN=4ss5dS2DH6Cof=zocJXeGOjRNKxEGgUZdJPMndyNzA@mail.gmail.com>
 <CAAKRu_Z18jYbarKA4GZMOO00KtUqKtBqgQ8SgB=wSvSqKq3iiA@mail.gmail.com>
 <1736122.1658333315@sss.pgh.pa.us>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1736122.1658333315@sss.pgh.pa.us>
Archived-At: 
 <https://www.postgresql.org/message-id/20220720163424.ag2cdhruhbwkneym%40alap3.anarazel.de>
Precedence: bulk

Hi,

On 2022-07-20 12:08:35 -0400, Tom Lane wrote:
> AFAIR, the previous stats collector implementation had no such provision
> either: it'd just keep adding hashtable entries as it received info about
> new objects.

Yep.


> The only thing that's changed is that now those entries are in shared
> memory instead of process-local memory.  We'd be well advised to be
> sure that memory can be swapped out under pressure, but otherwise I'm
> not seeing that things have gotten worse.

FWIW, I ran a few memory usage benchmarks. Without stats accesses the
memory usage with shared memory stats was sometimes below, sometimes
above the "old" memory usage, depending on the number of objects. As
soon as there's stats access, it's well below (that includes things like
autovac workers).

I think there's quite a bit of memory usage reduction potential around
dsa.c - we occasionally end up with [nearly] unused dsm segments.

Greetings,

Andres Freund