public inbox for [email protected]
help / color / mirror / Atom feedFrom: Maxim Zibitsker <[email protected]>
To: [email protected]
Subject: Support allocating memory for large strings
Date: Fri, 7 Nov 2025 21:15:22 -0500
Message-ID: <[email protected]> (raw)
PostgreSQL's MaxAllocSize limit prevents storing individual variable-length character strings exceeding ~1GB, causing "invalid memory alloc request size" errors during INSERT operations on tables with large text columns. Example reproduction included in artifacts.md.
This limitation also affects pg_dump when exporting a PostgreSQL database with such data. The attached patches demonstrates a proof of concept using palloc_extended with MCXT_ALLOC_HUGE in the write path. For the read path, there are a couple of possible approaches: extending existing functions to handle huge allocations, or implementing a chunked storage mechanism that avoids single large allocations.
Thoughts?
Maxim
Attachments:
[application/octet-stream] 0001-Support-allocating-memory-for-large-strings.patch (1001B, ../[email protected]/2-0001-Support-allocating-memory-for-large-strings.patch)
download | inline diff:
From 354473fe17fab22399f6ce48c53e13839a3823cb Mon Sep 17 00:00:00 2001
From: Maxim Zibitsker <[email protected]>
Date: Thu, 30 Oct 2025 11:00:38 -0400
Subject: [PATCH] Support allocating memory for large strings
---
src/backend/access/common/heaptuple.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c
index 1173a6d81b5..e615fa9790a 100644
--- a/src/backend/access/common/heaptuple.c
+++ b/src/backend/access/common/heaptuple.c
@@ -1163,7 +1163,7 @@ heap_form_tuple(TupleDesc tupleDescriptor,
* Allocate and zero the space needed. Note that the tuple body and
* HeapTupleData management structure are allocated in one chunk.
*/
- tuple = (HeapTuple) palloc0(HEAPTUPLESIZE + len);
+ tuple = (HeapTuple) palloc_extended(HEAPTUPLESIZE + len, MCXT_ALLOC_HUGE | MCXT_ALLOC_ZERO);
tuple->t_data = td = (HeapTupleHeader) ((char *) tuple + HEAPTUPLESIZE);
/*
--
2.32.1 (Apple Git-133)
[text/markdown] artifacts.md (6.1K, ../[email protected]/4-artifacts.md)
download | inline:
### Client-side
postgres=# CREATE TABLE wide_row (id int, a varchar, b varchar, c varchar, d varchar, e varchar, f varchar, g varchar, h varchar, i varchar, j varchar, k varchar, l varchar, m varchar, n varchar, o varchar);
CREATE TABLE
postgres=# INSERT INTO wide_row VALUES (1, repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int));
ERROR: invalid memory alloc request size 1500000112
postgres=#
postgres=# INSERT INTO wide_row (id, a, b, c, d, e, f, g, h, i) VALUES (1, repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int));
INSERT 0 1
postgres=# UPDATE wide_row SET j = repeat('x', (10^8)::int), k = repeat('x', (10^8)::int), l = repeat('x', (10^8)::int), m = repeat('x', (10^8)::int), n = repeat('x', (10^8)::int), o = repeat('x', (10^8)::int)
postgres-# WHERE id = 1;
UPDATE 2
### Server-side
2025-11-05 21:42:51.927 EST [77944] ERROR: invalid memory alloc request size 1500000112
2025-11-05 21:42:51.927 EST [77944] BACKTRACE:
2 postgres 0x00000001047b9804 MemoryContextSizeFailure + 72
3 postgres 0x00000001047b2ad0 AllocSetAllocFromNewBlock + 0
4 postgres 0x00000001047b9a58 palloc0 + 48
5 postgres 0x0000000104336018 heap_form_tuple + 200
6 postgres 0x00000001044c5208 tts_buffer_heap_copyslot + 248
7 postgres 0x00000001044e1fec ExecModifyTable + 1620
8 postgres 0x00000001044ba168 standard_ExecutorRun + 288
9 postgres 0x0000000104657b98 ProcessQuery + 156
10 postgres 0x0000000104657340 PortalRunMulti + 356
11 postgres 0x0000000104656dac PortalRun + 436
12 postgres 0x0000000104655fbc exec_simple_query + 1404
13 postgres 0x0000000104653978 PostgresMain + 2792
14 postgres 0x000000010464f594 BackendInitialize + 0
15 postgres 0x00000001045b5b0c postmaster_child_launch + 312
16 postgres 0x00000001045b9f2c ServerLoop + 5964
17 postgres 0x00000001045b7cac PostmasterMain + 3744
18 postgres 0x0000000104505164 main + 848
19 dyld 0x0000000198a37fd8 start + 2412
2025-11-05 21:42:51.927 EST [77944] STATEMENT: INSERT INTO wide_row VALUES (1, repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int));
### Client-side
./pg_dump -U Max -d postgres -p 5433 -F c -f dump1.sql
pg_dump: error: Dumping the contents of table "wide_row" failed: PQgetResult() failed.
pg_dump: detail: Error message from server: ERROR: string buffer exceeds maximum allowed length (1073741823 bytes)
DETAIL: Cannot enlarge string buffer containing 1000000012 bytes by 100000000 more bytes.
pg_dump: detail: Command was: COPY public.wide_row (id, a, b, c, d, e, f, g, h, i, j, k, l, m, n, o) TO stdout;
### Server-side
2025-11-05 21:50:30.839 EST [79615] ERROR: string buffer exceeds maximum allowed length (1073741823 bytes)
2025-11-05 21:50:30.839 EST [79615] DETAIL: Cannot enlarge string buffer containing 1000000012 bytes by 100000000 more bytes.
2025-11-05 21:50:30.839 EST [79615] BACKTRACE:
2 postgres 0x00000001046740c4 enlargeStringInfo.cold.2 + 100
3 postgres 0x00000001045cb010 appendStringInfoString + 0
4 postgres 0x00000001045cb0a0 appendBinaryStringInfo + 36
5 postgres 0x0000000104239fc0 CopyToTextOneRow + 208
6 postgres 0x0000000104238eb4 DoCopyTo + 752
7 postgres 0x0000000104233d00 DoCopy + 1172
8 postgres 0x00000001044480dc standard_ProcessUtility + 688
9 postgres 0x0000000104447b20 PortalRunUtility + 136
10 postgres 0x000000010444732c PortalRunMulti + 240
11 postgres 0x0000000104446e0c PortalRun + 436
12 postgres 0x000000010444601c exec_simple_query + 1404
13 postgres 0x00000001044439d8 PostgresMain + 2792
14 postgres 0x000000010443f5f4 BackendInitialize + 0
15 postgres 0x00000001043a5b6c postmaster_child_launch + 312
16 postgres 0x00000001043a9f8c ServerLoop + 5964
17 postgres 0x00000001043a7d0c PostmasterMain + 3744
18 postgres 0x00000001042f51c4 main + 848
19 dyld 0x0000000198a37fd8 start + 2412
2025-11-05 21:50:30.839 EST [79615] STATEMENT: COPY public.wide_row (id, a, b, c, d, e, f, g, h, i, j, k, l, m, n, o) TO stdout;
2025-11-05 21:50:30.848 EST [79615] LOG: could not send data to client: Broken pipe
2025-11-05 21:50:30.848 EST [79615] ERROR: canceling statement due to user request
2025-11-05 21:50:30.848 EST [79615] FATAL: connection to client lost
view thread (5+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected]
Subject: Re: Support allocating memory for large strings
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox