Feedback-ID: id4a34324:Fastmail
Date: Thu, 19 Oct 2023 15:22:51 -0700
From: Andres Freund <andres@anarazel.de>
To: Stephen Frost <sfrost@snowman.net>
Cc: Andrei Lepikhov <a.lepikhov@postgrespro.ru>,
	reid.thompson@crunchydata.com, Arne Roland <A.Roland@index.de>,
	"pgsql-hackers@lists.postgresql.org" <pgsql-hackers@lists.postgresql.org>,
	vignesh C <vignesh21@gmail.com>,
	Justin Pryzby <pryzby@telsasoft.com>,
	Ibrar Ahmed <ibrar.ahmad@gmail.com>,
	"stephen.frost" <stephen.frost@crunchydata.com>
Subject: Re: Add the ability to limit the amount of memory that can be
 allocated to backends.
Message-ID: <20231019222251.4qg6udcbtrwglogw@awork3.anarazel.de>
References: <334ccda9d8790f2b80dc3fee787704b0621152da.camel@crunchydata.com>
 <BE1P281MB313901E262719C4CF69B224BFE9C9@BE1P281MB3139.DEUP281.PROD.OUTLOOK.COM>
 <94184680049d6bda0c85f1759af24d8127cfe895.camel@crunchydata.com>
 <7912c911af51d5cf28c611190bf3d463b9209343.camel@crunchydata.com>
 <4edafedc0f8acb12a2979088ac1317bd7dd42145.camel@crunchydata.com>
 <f62b4107-227d-49e7-a9be-104d1ceb03d9@postgrespro.ru>
 <268e0ac7-8a81-4d65-8b40-b62c4b3f1bf9@postgrespro.ru>
 <ZTArWsctGn5fEVPR@tamriel.snowman.net>
 <48548d40-634b-4943-a737-3c9d95eacf06@postgrespro.ru>
 <ZTGoUt+pH8uPucfm@tamriel.snowman.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZTGoUt+pH8uPucfm@tamriel.snowman.net>
Archived-At: <https://www.postgresql.org/message-id/20231019222251.4qg6udcbtrwglogw%40awork3.anarazel.de>
Precedence: bulk

Hi,

On 2023-10-19 18:06:10 -0400, Stephen Frost wrote:
> Ignoring such would defeat much of the point of this effort- which is to
> get to a position where we can say with some confidence that we're not
> going to go over some limit that the user has set and therefore not
> allow ourselves to end up getting OOM killed.

I think that is a good medium to long term goal. I do however think that we'd
be better off merging the visibility of memory allocations soon-ish and
implement the limiting later. There's a lot of hairy details to get right for
the latter, and even just having visibility will be a huge improvement.

I think even patch 1 is doing too much at once. I doubt the DSM stuff is
quite right.

I'm unconvinced it's a good idea to split the different types of memory
contexts out. That just exposes too much implementation detail stuff without a
good reason.

I think the overhead even just the tracking implies right now is likely too
high and needs to be optimized. It should be a single math operation, not
tracking things in multiple fields. I don't think pg_sub_u64_overflow() should
be in the path either, that suddenly adds conditional branches.  You really
ought to look at the difference in assembly for the hot functions.

Greetings,

Andres Freund