public inbox for [email protected]help / color / mirror / Atom feed
Support allocating memory for large strings 5+ messages / 4 participants [nested] [flat]
* Support allocating memory for large strings @ 2025-11-08 02:15 Maxim Zibitsker <[email protected]> 0 siblings, 2 replies; 5+ messages in thread From: Maxim Zibitsker @ 2025-11-08 02:15 UTC (permalink / raw) To: pgsql-hackers PostgreSQL's MaxAllocSize limit prevents storing individual variable-length character strings exceeding ~1GB, causing "invalid memory alloc request size" errors during INSERT operations on tables with large text columns. Example reproduction included in artifacts.md. This limitation also affects pg_dump when exporting a PostgreSQL database with such data. The attached patches demonstrates a proof of concept using palloc_extended with MCXT_ALLOC_HUGE in the write path. For the read path, there are a couple of possible approaches: extending existing functions to handle huge allocations, or implementing a chunked storage mechanism that avoids single large allocations. Thoughts? Maxim Attachments: [application/octet-stream] 0001-Support-allocating-memory-for-large-strings.patch (1001B, ../../[email protected]/2-0001-Support-allocating-memory-for-large-strings.patch) download | inline diff: From 354473fe17fab22399f6ce48c53e13839a3823cb Mon Sep 17 00:00:00 2001 From: Maxim Zibitsker <[email protected]> Date: Thu, 30 Oct 2025 11:00:38 -0400 Subject: [PATCH] Support allocating memory for large strings --- src/backend/access/common/heaptuple.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c index 1173a6d81b5..e615fa9790a 100644 --- a/src/backend/access/common/heaptuple.c +++ b/src/backend/access/common/heaptuple.c @@ -1163,7 +1163,7 @@ heap_form_tuple(TupleDesc tupleDescriptor, * Allocate and zero the space needed. Note that the tuple body and * HeapTupleData management structure are allocated in one chunk. */ - tuple = (HeapTuple) palloc0(HEAPTUPLESIZE + len); + tuple = (HeapTuple) palloc_extended(HEAPTUPLESIZE + len, MCXT_ALLOC_HUGE | MCXT_ALLOC_ZERO); tuple->t_data = td = (HeapTupleHeader) ((char *) tuple + HEAPTUPLESIZE); /* -- 2.32.1 (Apple Git-133) [text/markdown] artifacts.md (6.1K, ../../[email protected]/4-artifacts.md) download | inline: ### Client-side postgres=# CREATE TABLE wide_row (id int, a varchar, b varchar, c varchar, d varchar, e varchar, f varchar, g varchar, h varchar, i varchar, j varchar, k varchar, l varchar, m varchar, n varchar, o varchar); CREATE TABLE postgres=# INSERT INTO wide_row VALUES (1, repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int)); ERROR: invalid memory alloc request size 1500000112 postgres=# postgres=# INSERT INTO wide_row (id, a, b, c, d, e, f, g, h, i) VALUES (1, repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int)); INSERT 0 1 postgres=# UPDATE wide_row SET j = repeat('x', (10^8)::int), k = repeat('x', (10^8)::int), l = repeat('x', (10^8)::int), m = repeat('x', (10^8)::int), n = repeat('x', (10^8)::int), o = repeat('x', (10^8)::int) postgres-# WHERE id = 1; UPDATE 2 ### Server-side 2025-11-05 21:42:51.927 EST [77944] ERROR: invalid memory alloc request size 1500000112 2025-11-05 21:42:51.927 EST [77944] BACKTRACE: 2 postgres 0x00000001047b9804 MemoryContextSizeFailure + 72 3 postgres 0x00000001047b2ad0 AllocSetAllocFromNewBlock + 0 4 postgres 0x00000001047b9a58 palloc0 + 48 5 postgres 0x0000000104336018 heap_form_tuple + 200 6 postgres 0x00000001044c5208 tts_buffer_heap_copyslot + 248 7 postgres 0x00000001044e1fec ExecModifyTable + 1620 8 postgres 0x00000001044ba168 standard_ExecutorRun + 288 9 postgres 0x0000000104657b98 ProcessQuery + 156 10 postgres 0x0000000104657340 PortalRunMulti + 356 11 postgres 0x0000000104656dac PortalRun + 436 12 postgres 0x0000000104655fbc exec_simple_query + 1404 13 postgres 0x0000000104653978 PostgresMain + 2792 14 postgres 0x000000010464f594 BackendInitialize + 0 15 postgres 0x00000001045b5b0c postmaster_child_launch + 312 16 postgres 0x00000001045b9f2c ServerLoop + 5964 17 postgres 0x00000001045b7cac PostmasterMain + 3744 18 postgres 0x0000000104505164 main + 848 19 dyld 0x0000000198a37fd8 start + 2412 2025-11-05 21:42:51.927 EST [77944] STATEMENT: INSERT INTO wide_row VALUES (1, repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int)); ### Client-side ./pg_dump -U Max -d postgres -p 5433 -F c -f dump1.sql pg_dump: error: Dumping the contents of table "wide_row" failed: PQgetResult() failed. pg_dump: detail: Error message from server: ERROR: string buffer exceeds maximum allowed length (1073741823 bytes) DETAIL: Cannot enlarge string buffer containing 1000000012 bytes by 100000000 more bytes. pg_dump: detail: Command was: COPY public.wide_row (id, a, b, c, d, e, f, g, h, i, j, k, l, m, n, o) TO stdout; ### Server-side 2025-11-05 21:50:30.839 EST [79615] ERROR: string buffer exceeds maximum allowed length (1073741823 bytes) 2025-11-05 21:50:30.839 EST [79615] DETAIL: Cannot enlarge string buffer containing 1000000012 bytes by 100000000 more bytes. 2025-11-05 21:50:30.839 EST [79615] BACKTRACE: 2 postgres 0x00000001046740c4 enlargeStringInfo.cold.2 + 100 3 postgres 0x00000001045cb010 appendStringInfoString + 0 4 postgres 0x00000001045cb0a0 appendBinaryStringInfo + 36 5 postgres 0x0000000104239fc0 CopyToTextOneRow + 208 6 postgres 0x0000000104238eb4 DoCopyTo + 752 7 postgres 0x0000000104233d00 DoCopy + 1172 8 postgres 0x00000001044480dc standard_ProcessUtility + 688 9 postgres 0x0000000104447b20 PortalRunUtility + 136 10 postgres 0x000000010444732c PortalRunMulti + 240 11 postgres 0x0000000104446e0c PortalRun + 436 12 postgres 0x000000010444601c exec_simple_query + 1404 13 postgres 0x00000001044439d8 PostgresMain + 2792 14 postgres 0x000000010443f5f4 BackendInitialize + 0 15 postgres 0x00000001043a5b6c postmaster_child_launch + 312 16 postgres 0x00000001043a9f8c ServerLoop + 5964 17 postgres 0x00000001043a7d0c PostmasterMain + 3744 18 postgres 0x00000001042f51c4 main + 848 19 dyld 0x0000000198a37fd8 start + 2412 2025-11-05 21:50:30.839 EST [79615] STATEMENT: COPY public.wide_row (id, a, b, c, d, e, f, g, h, i, j, k, l, m, n, o) TO stdout; 2025-11-05 21:50:30.848 EST [79615] LOG: could not send data to client: Broken pipe 2025-11-05 21:50:30.848 EST [79615] ERROR: canceling statement due to user request 2025-11-05 21:50:30.848 EST [79615] FATAL: connection to client lost ^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Support allocating memory for large strings @ 2025-11-08 02:32 Tom Lane <[email protected]> parent: Maxim Zibitsker <[email protected]> 1 sibling, 1 reply; 5+ messages in thread From: Tom Lane @ 2025-11-08 02:32 UTC (permalink / raw) To: Maxim Zibitsker <[email protected]>; +Cc: pgsql-hackers Maxim Zibitsker <[email protected]> writes: > PostgreSQL's MaxAllocSize limit prevents storing individual variable-length character strings exceeding ~1GB, causing "invalid memory alloc request size" errors during INSERT operations on tables with large text columns. This is news to no one. We are not especially interested in trying to relax that limit, because doing so would bleed over into approximately everything in the backend, and create opportunities for integer-overflow bugs in many places that are perfectly okay today. The cost-benefit ratio for changing this decision is horrible. > The attached patches demonstrates a proof of concept using > palloc_extended with MCXT_ALLOC_HUGE in the write path. "Proof of concept"? This can't possibly fix your problem, because it does nothing for the fact that tuple size fields are still limited to 1GB, as are varlena headers for individual fields. A serious attack on this limitation, at a guess, would require a patch on the order of 100K lines, and that might be an underestimate. regards, tom lane ^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Support allocating memory for large strings @ 2025-11-08 11:17 Jose Luis Tallon <[email protected]> parent: Maxim Zibitsker <[email protected]> 1 sibling, 0 replies; 5+ messages in thread From: Jose Luis Tallon @ 2025-11-08 11:17 UTC (permalink / raw) To: Maxim Zibitsker <[email protected]>; pgsql-hackers On 8/11/25 3:15, Maxim Zibitsker wrote: > PostgreSQL's MaxAllocSize limit prevents storing individual variable-length character strings exceeding ~1GB, causing "invalid memory alloc request size" errors during INSERT operations on tables with large text columns. Example reproduction included in artifacts.md. Tom Lane's very appropriate response not withstanding.... a) Why is this a problem? (Please share a bit more about your intended use case) b) Why would someone need to store >1GB worth of TEXT (in a single string, no less!) in a column in an (albeit very flexible) Relational Database ? (I'm assuming no internal structure that would allow such amount of text to be split/spread over multiple records) c) There exists LObs (Large OBjects) intended for this use, precisely... why is this mechanism not a good solution to your need? d) Wouldn't a (journalling) File System (with a slim abstraction layer on top for directory hashing/indexing) not be a better solution for this particular application? Full Text Search on the stored data doesn't look like it would ever be performant... there exist specialized tools for that And... how did you get "invalid" data in the database, that pg_dump wouldn't process, in the first place? (maybe just speculating/projecting and I didn't pick up the nuance properly) Mostly curious about the problem / intended use case.... when we explored limits and limitations in Postgres almost 15 years ago, we never considered this even :o Thanks, -- Parkinson's Law: Work expands to fill the time alloted to it. ^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Support allocating memory for large strings @ 2025-11-10 20:35 Nathan Bossart <[email protected]> parent: Tom Lane <[email protected]> 0 siblings, 1 reply; 5+ messages in thread From: Nathan Bossart @ 2025-11-10 20:35 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: Maxim Zibitsker <[email protected]>; pgsql-hackers On Fri, Nov 07, 2025 at 09:32:45PM -0500, Tom Lane wrote: > Maxim Zibitsker <[email protected]> writes: >> PostgreSQL's MaxAllocSize limit prevents storing individual >> variable-length character strings exceeding ~1GB, causing "invalid >> memory alloc request size" errors during INSERT operations on tables >> with large text columns. > > This is news to no one. We are not especially interested in trying to > relax that limit, because doing so would bleed over into approximately > everything in the backend, and create opportunities for > integer-overflow bugs in many places that are perfectly okay today. > The cost-benefit ratio for changing this decision is horrible. FWIW something I am hearing about more often these days, and what I believe Maxim's patch is actually after, is the 1GB limit on row size. Even if each field doesn't exceed 1GB (which is what artifacts.md seems to demonstrate), heap_form_tuple() and friends can fail to construct the whole tuple. This doesn't seem to be covered in the existing documentation about limits [0]. [0] https://www.postgresql.org/docs/devel/limits.html -- nathan ^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Support allocating memory for large strings @ 2025-11-10 21:37 Tom Lane <[email protected]> parent: Nathan Bossart <[email protected]> 0 siblings, 0 replies; 5+ messages in thread From: Tom Lane @ 2025-11-10 21:37 UTC (permalink / raw) To: Nathan Bossart <[email protected]>; +Cc: Maxim Zibitsker <[email protected]>; pgsql-hackers Nathan Bossart <[email protected]> writes: > FWIW something I am hearing about more often these days, and what I believe > Maxim's patch is actually after, is the 1GB limit on row size. Even if > each field doesn't exceed 1GB (which is what artifacts.md seems to > demonstrate), heap_form_tuple() and friends can fail to construct the whole > tuple. This doesn't seem to be covered in the existing documentation about > limits [0]. Yeah. I think our hopes of relaxing the 1GB limit on individual field values are about zero, but maybe there is some chance of allowing tuples that are wider than that. The notion that it's a one-line fix is still ludicrous though :-( One big problem with a scheme like that is "what happens when I try to make a bigger-than-1GB tuple into a composite datum?". Another issue is what happens when a wider-than-1GB tuple needs to be sent to or from clients. I think there are assumptions in the wire protocol about message lengths fitting in an int, for example. Even if the protocol were okay with it, I wouldn't count on client libraries not to fall over. On the whole, it's a nasty can of worms, and I stand by the opinion that the cost-benefit ratio of removing the limit is pretty awful. regards, tom lane ^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2025-11-10 21:37 UTC | newest] Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2025-11-08 02:15 Support allocating memory for large strings Maxim Zibitsker <[email protected]> 2025-11-08 02:32 ` Tom Lane <[email protected]> 2025-11-10 20:35 ` Nathan Bossart <[email protected]> 2025-11-10 21:37 ` Tom Lane <[email protected]> 2025-11-08 11:17 ` Jose Luis Tallon <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox