From: Ekaterina Kiryanova <e.kiryanova@postgrespro.ru>
To: pgsql-docs@lists.postgresql.org
Subject: Limitation relates to memory allocation
Date: Mon, 14 Oct 2024 09:03:36 +0300
Message-ID: <440403b2-5c91-45a8-88c9-db031f05ef4d@postgrespro.ru> (raw)
Hello!
We encountered an issue related to internal memory allocation limit:
ERROR: invalid memory alloc request size
In the documentation on limits:
https://www.postgresql.org/docs/17/limits.html
the description suggests that only a single field is limited to 1GB,
which could imply that the total tuple size can be larger. So as we
planned to store three columns of 1GB each in a table and attempted to
insert this data, we got the error.
Our research showed that the limit is imposed by the palloc() function,
regardless of whether it is a tuple or not, and if the data is
serialized or dumped, the effective limit can be even lower, typically
around 512MB per row. So for allocations exceeding 1GB, the
palloc_extended() function can be used. Please correct me if I'm wrong.
I prepared a small patch for master, if it's worth clarifying, could you
please review the attachment?
--
Ekaterina Kiryanova
Technical Writer
Postgres Professional
the Russian PostgreSQL Company
Attachments:
[text/x-patch] palloc-limitation.patch (1.0K, ../440403b2-5c91-45a8-88c9-db031f05ef4d@postgrespro.ru/2-palloc-limitation.patch)
download | inline diff:diff --git a/doc/src/sgml/limits.sgml b/doc/src/sgml/limits.sgmlindex f26f4466719..89c383d1db7 100644--- a/doc/src/sgml/limits.sgml+++ b/doc/src/sgml/limits.sgml@@ -71,7 +71,8 @@
<row>
<entry>field size</entry>
<entry>1 GB</entry>
- <entry></entry>+ <entry>limited by <function>palloc()</function> restriction; see note+ below</entry>
</row>
<row>
@@ -146,4 +147,14 @@
Typically, this is only an issue for tables containing many terabytes
of data; partitioning is a possible workaround.
</para>
++ <para>+ The 1 GB field size limit is imposed by the memory-allocating+ <function>palloc()</function> function. The practical limit is less than+ 1 GB, especially if the data is serialized or dumped. In cases where+ larger allocations are needed, <function>palloc_extended()</function>+ can be used, as it bypasses this restriction by invoking+ <function>malloc()</function> for large memory blocks, which allows+ requesting memory directly from the operating system.+ </para>
</appendix>
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-docs@postgresql.org
Cc: e.kiryanova@postgrespro.ru, pgsql-docs@lists.postgresql.org
Subject: Re: Limitation relates to memory allocation
In-Reply-To: <440403b2-5c91-45a8-88c9-db031f05ef4d@postgrespro.ru>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox