Feedback-ID: ie0a040ee:Fastmail
Message-ID: <80e8ac9e-da2a-4d3e-8345-d16c77ac88f9@eisentraut.org>
Date: Wed, 16 Oct 2024 09:07:13 +0200
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Limitation relates to memory allocation
To: Ekaterina Kiryanova <e.kiryanova@postgrespro.ru>,
 pgsql-docs@lists.postgresql.org
References: <440403b2-5c91-45a8-88c9-db031f05ef4d@postgrespro.ru>
Content-Language: en-US
From: Peter Eisentraut <peter@eisentraut.org>
In-Reply-To: <440403b2-5c91-45a8-88c9-db031f05ef4d@postgrespro.ru>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Archived-At: 
 <https://www.postgresql.org/message-id/80e8ac9e-da2a-4d3e-8345-d16c77ac88f9%40eisentraut.org>
Precedence: bulk

On 14.10.24 08:03, Ekaterina Kiryanova wrote:
> We encountered an issue related to internal memory allocation limit:
> ERROR: invalid memory alloc request size
> 
> In the documentation on limits: https://www.postgresql.org/docs/17/ 
> limits.html
> the description suggests that only a single field is limited to 1GB, 
> which could imply that the total tuple size can be larger. So as we 
> planned to store three columns of 1GB each in a table and attempted to 
> insert this data, we got the error.
> 
> Our research showed that the limit is imposed by the palloc() function, 
> regardless of whether it is a tuple or not, and if the data is 
> serialized or dumped, the effective limit can be even lower, typically 
> around 512MB per row. So for allocations exceeding 1GB, the 
> palloc_extended() function can be used. Please correct me if I'm wrong.
> 
> I prepared a small patch for master, if it's worth clarifying, could you 
> please review the attachment?

The 1 GB limit in palloc() is a safety check, when the code shouldn't be 
allocating more than that.  Code that legitimately wants to allocate 
more than 1 GB can use the MCXT_ALLOC_HUGE flag.

If you see this error, then that could either be corruption somewhere 
(the kind of thing this safety check is meant to catch) or the code is 
buggy.

In either case, I don't know that it is appropriate to document this as 
an externally visible system limitation.