Date: Fri, 16 Aug 2019 14:14:18 -0400
From: Alvaro Herrera <alvherre@2ndquadrant.com>
To: pgsql-hackers@lists.postgresql.org
Subject: allocation limit for encoding conversion
Message-ID: <20190816181418.GA898@alvherre.pgsql>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="OXfL5xGRrasGEqWY"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
User-Agent: Mutt/1.9.4 (2018-02-28)
Precedence: bulk


--OXfL5xGRrasGEqWY
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit

Somebody ran into issues when generating large XML output (upwards of
256 MB) and then sending via a connection with a different
client_encoding.  This occurs because we pessimistically allocate 4x as
much memory as the string needs, and we run into the 1GB palloc
limitation.  ISTM we can do better now by using huge allocations, as per
the preliminary attached patch (which probably needs an updated overflow
check rather than have it removed altogether); but at least it is able
to process this query, which it wasn't without the patch:

select query_to_xml(
    'select a, cash_words(a::text::money) from generate_series(0, 2000000) a',
    true, false, '');

-- 
Álvaro Herrera

--OXfL5xGRrasGEqWY
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
 filename="v1-0001-Use-huge-pallocs-on-encoding-conversions.patch"

From a718f299d72617c366a08735895e0439482ede2f Mon Sep 17 00:00:00 2001
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
Date: Thu, 8 Aug 2019 15:36:53 -0400
Subject: [PATCH v1] Use huge pallocs on encoding conversions

---
 src/backend/utils/mb/mbutils.c | 26 ++++----------------------
 1 file changed, 4 insertions(+), 22 deletions(-)

diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index bec54bb5cb..e2853fcf6e 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -348,17 +348,8 @@ pg_do_encoding_conversion(unsigned char *src, int len,
 						pg_encoding_to_char(src_encoding),
 						pg_encoding_to_char(dest_encoding))));
 
-	/*
-	 * Allocate space for conversion result, being wary of integer overflow
-	 */
-	if ((Size) len >= (MaxAllocSize / (Size) MAX_CONVERSION_GROWTH))
-		ereport(ERROR,
-				(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
-				 errmsg("out of memory"),
-				 errdetail("String of %d bytes is too long for encoding conversion.",
-						   len)));
-
-	result = palloc(len * MAX_CONVERSION_GROWTH + 1);
+	result = palloc_extended((Size) len * MAX_CONVERSION_GROWTH + 1,
+							 MCXT_ALLOC_HUGE);
 
 	OidFunctionCall5(proc,
 					 Int32GetDatum(src_encoding),
@@ -681,17 +672,8 @@ perform_default_encoding_conversion(const char *src, int len,
 	if (flinfo == NULL)
 		return unconstify(char *, src);
 
-	/*
-	 * Allocate space for conversion result, being wary of integer overflow
-	 */
-	if ((Size) len >= (MaxAllocSize / (Size) MAX_CONVERSION_GROWTH))
-		ereport(ERROR,
-				(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
-				 errmsg("out of memory"),
-				 errdetail("String of %d bytes is too long for encoding conversion.",
-						   len)));
-
-	result = palloc(len * MAX_CONVERSION_GROWTH + 1);
+	result = palloc_extended((Size) len * MAX_CONVERSION_GROWTH + 1,
+							 MCXT_ALLOC_HUGE);
 
 	FunctionCall5(flinfo,
 				  Int32GetDatum(src_encoding),
-- 
2.17.1


--OXfL5xGRrasGEqWY--