From: Peter Eisentraut <peter@eisentraut.org>
To: Ekaterina Kiryanova <e.kiryanova@postgrespro.ru>
To: pgsql-docs@lists.postgresql.org
Subject: Re: Limitation relates to memory allocation
Date: Wed, 16 Oct 2024 09:07:13 +0200
Message-ID: <80e8ac9e-da2a-4d3e-8345-d16c77ac88f9@eisentraut.org> (raw)
In-Reply-To: <440403b2-5c91-45a8-88c9-db031f05ef4d@postgrespro.ru>
References: <440403b2-5c91-45a8-88c9-db031f05ef4d@postgrespro.ru>
On 14.10.24 08:03, Ekaterina Kiryanova wrote:
> We encountered an issue related to internal memory allocation limit:
> ERROR: invalid memory alloc request size
>
> In the documentation on limits: https://www.postgresql.org/docs/17/
> limits.html
> the description suggests that only a single field is limited to 1GB,
> which could imply that the total tuple size can be larger. So as we
> planned to store three columns of 1GB each in a table and attempted to
> insert this data, we got the error.
>
> Our research showed that the limit is imposed by the palloc() function,
> regardless of whether it is a tuple or not, and if the data is
> serialized or dumped, the effective limit can be even lower, typically
> around 512MB per row. So for allocations exceeding 1GB, the
> palloc_extended() function can be used. Please correct me if I'm wrong.
>
> I prepared a small patch for master, if it's worth clarifying, could you
> please review the attachment?
The 1 GB limit in palloc() is a safety check, when the code shouldn't be
allocating more than that. Code that legitimately wants to allocate
more than 1 GB can use the MCXT_ALLOC_HUGE flag.
If you see this error, then that could either be corruption somewhere
(the kind of thing this safety check is meant to catch) or the code is
buggy.
In either case, I don't know that it is appropriate to document this as
an externally visible system limitation.
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-docs@postgresql.org
Cc: peter@eisentraut.org, e.kiryanova@postgrespro.ru, pgsql-docs@lists.postgresql.org
Subject: Re: Limitation relates to memory allocation
In-Reply-To: <80e8ac9e-da2a-4d3e-8345-d16c77ac88f9@eisentraut.org>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox