Content-Type: multipart/mixed; boundary="------------ZUTxBFJHQpyuN6JEN6p3xMy7"
Message-ID: <440403b2-5c91-45a8-88c9-db031f05ef4d@postgrespro.ru>
Date: Mon, 14 Oct 2024 09:03:36 +0300
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Content-Language: en-US
To: pgsql-docs@lists.postgresql.org
From: Ekaterina Kiryanova <e.kiryanova@postgrespro.ru>
Subject: Limitation relates to memory allocation
Archived-At: 
 <https://www.postgresql.org/message-id/440403b2-5c91-45a8-88c9-db031f05ef4d%40postgrespro.ru>
Precedence: bulk

This is a multi-part message in MIME format.
--------------ZUTxBFJHQpyuN6JEN6p3xMy7
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Hello!

We encountered an issue related to internal memory allocation limit:
ERROR: invalid memory alloc request size

In the documentation on limits: 
https://www.postgresql.org/docs/17/limits.html
the description suggests that only a single field is limited to 1GB, 
which could imply that the total tuple size can be larger. So as we 
planned to store three columns of 1GB each in a table and attempted to 
insert this data, we got the error.

Our research showed that the limit is imposed by the palloc() function, 
regardless of whether it is a tuple or not, and if the data is 
serialized or dumped, the effective limit can be even lower, typically 
around 512MB per row. So for allocations exceeding 1GB, the 
palloc_extended() function can be used. Please correct me if I'm wrong.

I prepared a small patch for master, if it's worth clarifying, could you 
please review the attachment?

-- 
Ekaterina Kiryanova
Technical Writer
Postgres Professional
the Russian PostgreSQL Company
--------------ZUTxBFJHQpyuN6JEN6p3xMy7
Content-Type: text/x-patch; charset=UTF-8; name="palloc-limitation.patch"
Content-Disposition: attachment; filename="palloc-limitation.patch"
Content-Transfer-Encoding: base64

ZGlmZiAtLWdpdCBhL2RvYy9zcmMvc2dtbC9saW1pdHMuc2dtbCBiL2RvYy9zcmMvc2dtbC9s
aW1pdHMuc2dtbAppbmRleCBmMjZmNDQ2NjcxOS4uODljMzgzZDFkYjcgMTAwNjQ0Ci0tLSBh
L2RvYy9zcmMvc2dtbC9saW1pdHMuc2dtbAorKysgYi9kb2Mvc3JjL3NnbWwvbGltaXRzLnNn
bWwKQEAgLTcxLDcgKzcxLDggQEAKICAgICA8cm93PgogICAgICA8ZW50cnk+ZmllbGQgc2l6
ZTwvZW50cnk+CiAgICAgIDxlbnRyeT4xIEdCPC9lbnRyeT4KLSAgICAgPGVudHJ5PjwvZW50
cnk+CisgICAgIDxlbnRyeT5saW1pdGVkIGJ5IDxmdW5jdGlvbj5wYWxsb2MoKTwvZnVuY3Rp
b24+IHJlc3RyaWN0aW9uOyBzZWUgbm90ZQorICAgICBiZWxvdzwvZW50cnk+CiAgICAgPC9y
b3c+CiAKICAgICA8cm93PgpAQCAtMTQ2LDQgKzE0NywxNCBAQAogICBUeXBpY2FsbHksIHRo
aXMgaXMgb25seSBhbiBpc3N1ZSBmb3IgdGFibGVzIGNvbnRhaW5pbmcgbWFueSB0ZXJhYnl0
ZXMKICAgb2YgZGF0YTsgcGFydGl0aW9uaW5nIGlzIGEgcG9zc2libGUgd29ya2Fyb3VuZC4K
ICA8L3BhcmE+CisKKyA8cGFyYT4KKyAgVGhlIDEgR0IgZmllbGQgc2l6ZSBsaW1pdCBpcyBp
bXBvc2VkIGJ5IHRoZSBtZW1vcnktYWxsb2NhdGluZworICA8ZnVuY3Rpb24+cGFsbG9jKCk8
L2Z1bmN0aW9uPiBmdW5jdGlvbi4gVGhlIHByYWN0aWNhbCBsaW1pdCBpcyBsZXNzIHRoYW4K
KyAgMSBHQiwgZXNwZWNpYWxseSBpZiB0aGUgZGF0YSBpcyBzZXJpYWxpemVkIG9yIGR1bXBl
ZC4gSW4gY2FzZXMgd2hlcmUKKyAgbGFyZ2VyIGFsbG9jYXRpb25zIGFyZSBuZWVkZWQsIDxm
dW5jdGlvbj5wYWxsb2NfZXh0ZW5kZWQoKTwvZnVuY3Rpb24+CisgIGNhbiBiZSB1c2VkLCBh
cyBpdCBieXBhc3NlcyB0aGlzIHJlc3RyaWN0aW9uIGJ5IGludm9raW5nCisgIDxmdW5jdGlv
bj5tYWxsb2MoKTwvZnVuY3Rpb24+IGZvciBsYXJnZSBtZW1vcnkgYmxvY2tzLCB3aGljaCBh
bGxvd3MKKyAgcmVxdWVzdGluZyBtZW1vcnkgZGlyZWN0bHkgZnJvbSB0aGUgb3BlcmF0aW5n
IHN5c3RlbS4KKyA8L3BhcmE+CiA8L2FwcGVuZGl4Pgo=

--------------ZUTxBFJHQpyuN6JEN6p3xMy7--