From: Maxim Zibitsker <max.zibitsker@gmail.com>
Content-Type: multipart/mixed;
	boundary="Apple-Mail=_6BEA65A9-DA96-4C13-9BF5-3A4147099821"
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6.1.2\))
Subject: Support allocating memory for large strings
Message-Id: <5B146D62-35AE-4822-9B98-37E386AFB942@gmail.com>
Date: Fri, 7 Nov 2025 21:15:22 -0500
To: pgsql-hackers@postgresql.org
Archived-At: 
 <https://www.postgresql.org/message-id/5B146D62-35AE-4822-9B98-37E386AFB942%40gmail.com>
Precedence: bulk


--Apple-Mail=_6BEA65A9-DA96-4C13-9BF5-3A4147099821
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

PostgreSQL's MaxAllocSize limit prevents storing individual =
variable-length character strings exceeding ~1GB, causing "invalid =
memory alloc request size" errors during INSERT operations on tables =
with large text columns. Example reproduction included in artifacts.md.

This limitation also affects pg_dump when exporting a PostgreSQL =
database with such data. The attached patches demonstrates a proof of =
concept using palloc_extended with MCXT_ALLOC_HUGE in the write path. =
For the read path, there are a couple of possible approaches: extending =
existing functions to handle huge allocations, or implementing a chunked =
storage mechanism that avoids single large allocations.

Thoughts?

Maxim


--Apple-Mail=_6BEA65A9-DA96-4C13-9BF5-3A4147099821
Content-Disposition: attachment;
	filename=0001-Support-allocating-memory-for-large-strings.patch
Content-Type: application/octet-stream;
	x-unix-mode=0644;
	name="0001-Support-allocating-memory-for-large-strings.patch"
Content-Transfer-Encoding: quoted-printable

=46rom=20354473fe17fab22399f6ce48c53e13839a3823cb=20Mon=20Sep=2017=20=
00:00:00=202001=0AFrom:=20Maxim=20Zibitsker=20<max.zibitsker@gmail.com>=0A=
Date:=20Thu,=2030=20Oct=202025=2011:00:38=20-0400=0ASubject:=20[PATCH]=20=
Support=20allocating=20memory=20for=20large=20strings=0A=0A---=0A=20=
src/backend/access/common/heaptuple.c=20|=202=20+-=0A=201=20file=20=
changed,=201=20insertion(+),=201=20deletion(-)=0A=0Adiff=20--git=20=
a/src/backend/access/common/heaptuple.c=20=
b/src/backend/access/common/heaptuple.c=0Aindex=20=
1173a6d81b5..e615fa9790a=20100644=0A---=20=
a/src/backend/access/common/heaptuple.c=0A+++=20=
b/src/backend/access/common/heaptuple.c=0A@@=20-1163,7=20+1163,7=20@@=20=
heap_form_tuple(TupleDesc=20tupleDescriptor,=0A=20=09=20*=20Allocate=20=
and=20zero=20the=20space=20needed.=20=20Note=20that=20the=20tuple=20body=20=
and=0A=20=09=20*=20HeapTupleData=20management=20structure=20are=20=
allocated=20in=20one=20chunk.=0A=20=09=20*/=0A-=09tuple=20=3D=20=
(HeapTuple)=20palloc0(HEAPTUPLESIZE=20+=20len);=0A+=09tuple=20=3D=20=
(HeapTuple)=20palloc_extended(HEAPTUPLESIZE=20+=20len,=20MCXT_ALLOC_HUGE=20=
|=20MCXT_ALLOC_ZERO);=0A=20=09tuple->t_data=20=3D=20td=20=3D=20=
(HeapTupleHeader)=20((char=20*)=20tuple=20+=20HEAPTUPLESIZE);=0A=20=0A=20=
=09/*=0A--=20=0A2.32.1=20(Apple=20Git-133)=0A=0A=

--Apple-Mail=_6BEA65A9-DA96-4C13-9BF5-3A4147099821
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
	charset=us-ascii


--Apple-Mail=_6BEA65A9-DA96-4C13-9BF5-3A4147099821
Content-Disposition: attachment;
	filename=artifacts.md
Content-Type: text/markdown;
	x-unix-mode=0644;
	name="artifacts.md"
Content-Transfer-Encoding: 7bit

### Client-side
postgres=# CREATE TABLE wide_row (id int, a varchar, b varchar, c varchar, d varchar, e varchar, f varchar, g varchar, h varchar, i varchar, j varchar, k varchar, l varchar, m varchar, n varchar, o varchar);
CREATE TABLE

postgres=# INSERT INTO wide_row VALUES (1, repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int));

ERROR:  invalid memory alloc request size 1500000112
postgres=# 
postgres=# INSERT INTO wide_row (id, a, b, c, d, e, f, g, h, i) VALUES (1, repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int));
INSERT 0 1
postgres=# UPDATE wide_row SET j = repeat('x', (10^8)::int), k = repeat('x', (10^8)::int), l = repeat('x', (10^8)::int), m = repeat('x', (10^8)::int), n = repeat('x', (10^8)::int), o = repeat('x', (10^8)::int)
postgres-# WHERE id = 1;

UPDATE 2

### Server-side

2025-11-05 21:42:51.927 EST [77944] ERROR:  invalid memory alloc request size 1500000112
2025-11-05 21:42:51.927 EST [77944] BACKTRACE:  
        2   postgres                            0x00000001047b9804 MemoryContextSizeFailure + 72
        3   postgres                            0x00000001047b2ad0 AllocSetAllocFromNewBlock + 0
        4   postgres                            0x00000001047b9a58 palloc0 + 48
        5   postgres                            0x0000000104336018 heap_form_tuple + 200
        6   postgres                            0x00000001044c5208 tts_buffer_heap_copyslot + 248
        7   postgres                            0x00000001044e1fec ExecModifyTable + 1620
        8   postgres                            0x00000001044ba168 standard_ExecutorRun + 288
        9   postgres                            0x0000000104657b98 ProcessQuery + 156
        10  postgres                            0x0000000104657340 PortalRunMulti + 356
        11  postgres                            0x0000000104656dac PortalRun + 436
        12  postgres                            0x0000000104655fbc exec_simple_query + 1404
        13  postgres                            0x0000000104653978 PostgresMain + 2792
        14  postgres                            0x000000010464f594 BackendInitialize + 0
        15  postgres                            0x00000001045b5b0c postmaster_child_launch + 312
        16  postgres                            0x00000001045b9f2c ServerLoop + 5964
        17  postgres                            0x00000001045b7cac PostmasterMain + 3744
        18  postgres                            0x0000000104505164 main + 848
        19  dyld                                0x0000000198a37fd8 start + 2412
2025-11-05 21:42:51.927 EST [77944] STATEMENT:  INSERT INTO wide_row VALUES (1, repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int), repeat('x', (10^8)::int));


### Client-side
./pg_dump -U Max -d postgres  -p 5433 -F c -f dump1.sql
pg_dump: error: Dumping the contents of table "wide_row" failed: PQgetResult() failed.
pg_dump: detail: Error message from server: ERROR:  string buffer exceeds maximum allowed length (1073741823 bytes)
DETAIL:  Cannot enlarge string buffer containing 1000000012 bytes by 100000000 more bytes.
pg_dump: detail: Command was: COPY public.wide_row (id, a, b, c, d, e, f, g, h, i, j, k, l, m, n, o) TO stdout;


### Server-side
2025-11-05 21:50:30.839 EST [79615] ERROR:  string buffer exceeds maximum allowed length (1073741823 bytes)
2025-11-05 21:50:30.839 EST [79615] DETAIL:  Cannot enlarge string buffer containing 1000000012 bytes by 100000000 more bytes.
2025-11-05 21:50:30.839 EST [79615] BACKTRACE:  
        2   postgres                            0x00000001046740c4 enlargeStringInfo.cold.2 + 100
        3   postgres                            0x00000001045cb010 appendStringInfoString + 0
        4   postgres                            0x00000001045cb0a0 appendBinaryStringInfo + 36
        5   postgres                            0x0000000104239fc0 CopyToTextOneRow + 208
        6   postgres                            0x0000000104238eb4 DoCopyTo + 752
        7   postgres                            0x0000000104233d00 DoCopy + 1172
        8   postgres                            0x00000001044480dc standard_ProcessUtility + 688
        9   postgres                            0x0000000104447b20 PortalRunUtility + 136
        10  postgres                            0x000000010444732c PortalRunMulti + 240
        11  postgres                            0x0000000104446e0c PortalRun + 436
        12  postgres                            0x000000010444601c exec_simple_query + 1404
        13  postgres                            0x00000001044439d8 PostgresMain + 2792
        14  postgres                            0x000000010443f5f4 BackendInitialize + 0
        15  postgres                            0x00000001043a5b6c postmaster_child_launch + 312
        16  postgres                            0x00000001043a9f8c ServerLoop + 5964
        17  postgres                            0x00000001043a7d0c PostmasterMain + 3744
        18  postgres                            0x00000001042f51c4 main + 848
        19  dyld                                0x0000000198a37fd8 start + 2412
2025-11-05 21:50:30.839 EST [79615] STATEMENT:  COPY public.wide_row (id, a, b, c, d, e, f, g, h, i, j, k, l, m, n, o) TO stdout;
2025-11-05 21:50:30.848 EST [79615] LOG:  could not send data to client: Broken pipe
2025-11-05 21:50:30.848 EST [79615] ERROR:  canceling statement due to user request
2025-11-05 21:50:30.848 EST [79615] FATAL:  connection to client lost
--Apple-Mail=_6BEA65A9-DA96-4C13-9BF5-3A4147099821--