Message-ID: <48548d40-634b-4943-a737-3c9d95eacf06@postgrespro.ru>
Date: Thu, 19 Oct 2023 09:57:08 +0700
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Add the ability to limit the amount of memory that can be
 allocated to backends.
Content-Language: en-US
To: Stephen Frost <sfrost@snowman.net>
Cc: reid.thompson@crunchydata.com, Arne Roland <A.Roland@index.de>,
 Andres Freund <andres@anarazel.de>,
 "pgsql-hackers@lists.postgresql.org" <pgsql-hackers@lists.postgresql.org>,
 vignesh C <vignesh21@gmail.com>, Justin Pryzby <pryzby@telsasoft.com>,
 Ibrar Ahmed <ibrar.ahmad@gmail.com>,
 "stephen.frost" <stephen.frost@crunchydata.com>
References: <1a136d92e53eea52a4cde2163918ea91cf75e0b6.camel@crunchydata.com>
 <20230303002907.jktl5rddkvlazprp@alap3.anarazel.de>
 <457dd07b21810463d8dd282e48b99cd90c24a440.camel@crunchydata.com>
 <334ccda9d8790f2b80dc3fee787704b0621152da.camel@crunchydata.com>
 <BE1P281MB313901E262719C4CF69B224BFE9C9@BE1P281MB3139.DEUP281.PROD.OUTLOOK.COM>
 <94184680049d6bda0c85f1759af24d8127cfe895.camel@crunchydata.com>
 <7912c911af51d5cf28c611190bf3d463b9209343.camel@crunchydata.com>
 <4edafedc0f8acb12a2979088ac1317bd7dd42145.camel@crunchydata.com>
 <f62b4107-227d-49e7-a9be-104d1ceb03d9@postgrespro.ru>
 <268e0ac7-8a81-4d65-8b40-b62c4b3f1bf9@postgrespro.ru>
 <ZTArWsctGn5fEVPR@tamriel.snowman.net>
From: Andrei Lepikhov <a.lepikhov@postgrespro.ru>
Organization: Postgres Professional
In-Reply-To: <ZTArWsctGn5fEVPR@tamriel.snowman.net>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Archived-At: <https://www.postgresql.org/message-id/48548d40-634b-4943-a737-3c9d95eacf06%40postgrespro.ru>
Precedence: bulk

On 19/10/2023 02:00, Stephen Frost wrote:
> Greetings,
> 
> * Andrei Lepikhov (a.lepikhov@postgrespro.ru) wrote:
>> On 29/9/2023 09:52, Andrei Lepikhov wrote:
>>> On 22/5/2023 22:59, reid.thompson@crunchydata.com wrote:
>>>> Attach patches updated to master.
>>>> Pulled from patch 2 back to patch 1 a change that was also pertinent
>>>> to patch 1.
>>> +1 to the idea, have doubts on the implementation.
>>>
>>> I have a question. I see the feature triggers ERROR on the exceeding of
>>> the memory limit. The superior PG_CATCH() section will handle the error.
>>> As I see, many such sections use memory allocations. What if some
>>> routine, like the CopyErrorData(), exceeds the limit, too? In this case,
>>> we could repeat the error until the top PG_CATCH(). Is this correct
>>> behaviour? Maybe to check in the exceeds_max_total_bkend_mem() for
>>> recursion and allow error handlers to slightly exceed this hard limit?
> 
>> By the patch in attachment I try to show which sort of problems I'm worrying
>> about. In some PП_CATCH() sections we do CopyErrorData (allocate some
>> memory) before aborting the transaction. So, the allocation error can move
>> us out of this section before aborting. We await for soft ERROR message but
>> will face more hard consequences.
> 
> While it's an interesting idea to consider making exceptions to the
> limit, and perhaps we'll do that (or have some kind of 'reserve' for
> such cases), this isn't really any different than today, is it?  We
> might have a malloc() failure in the main path, end up in PG_CATCH() and
> then try to do a CopyErrorData() and have another malloc() failure.
> 
> If we can rearrange the code to make this less likely to happen, by
> doing a bit more work to free() resources used in the main path before
> trying to do new allocations, then, sure, let's go ahead and do that,
> but that's independent from this effort.

I agree that rearranging efforts can be made independently. The code in 
the letter above was shown just as a demo of the case I'm worried about.
IMO, the thing that should be implemented here is a recursion level for 
the memory limit. If processing the error, we fall into recursion with 
this limit - we should ignore it.
I imagine custom extensions that use PG_CATCH() and allocate some data 
there. At least we can raise the level of error to FATAL.

-- 
regards,
Andrey Lepikhov
Postgres Professional