From: Tom Lane <tgl@sss.pgh.pa.us>
To: Alvaro Herrera <alvherre@2ndquadrant.com>
cc: pgsql-hackers@lists.postgresql.org
Subject: Re: allocation limit for encoding conversion
In-reply-to: <20190816181418.GA898@alvherre.pgsql>
References: <20190816181418.GA898@alvherre.pgsql>
Comments: In-reply-to Alvaro Herrera <alvherre@2ndquadrant.com>
	message dated "Fri, 16 Aug 2019 14:14:18 -0400"
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <27989.1565991109.1@sss.pgh.pa.us>
Content-Transfer-Encoding: quoted-printable
Date: Fri, 16 Aug 2019 17:31:49 -0400
Message-ID: <27990.1565991109@sss.pgh.pa.us>
Precedence: bulk

Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Somebody ran into issues when generating large XML output (upwards of
> 256 MB) and then sending via a connection with a different
> client_encoding.  This occurs because we pessimistically allocate 4x as
> much memory as the string needs, and we run into the 1GB palloc
> limitation.  ISTM we can do better now by using huge allocations, as per
> the preliminary attached patch (which probably needs an updated overflow
> check rather than have it removed altogether); but at least it is able
> to process this query, which it wasn't without the patch:

> select query_to_xml(
>     'select a, cash_words(a::text::money) from generate_series(0, 200000=
0) a',
>     true, false, '');

I fear that allowing pg_do_encoding_conversion to return strings longer
than 1GB is just going to create failure cases somewhere else.

However, it's certainly true that 4x growth is a pretty unlikely worst
case.  Maybe we could do something like

1. If string is short (say up to a few megabytes), continue to do it
like now.  This avoids adding overhead for typical cases.

2. Otherwise, run some lobotomized form of encoding conversion that
just computes the space required (as an int64, I guess) without saving
the result anywhere.

3. If space required > 1GB, fail.

4. Otherwise, allocate just the space required, and convert.

			regards, tom lane