From: Tom Lane <tgl@sss.pgh.pa.us>
To: Alvaro Herrera <alvherre@2ndquadrant.com>
Cc: pgsql-hackers@lists.postgresql.org
Subject: Re: allocation limit for encoding conversion
Date: Fri, 16 Aug 2019 17:31:49 -0400
Message-ID: <27990.1565991109@sss.pgh.pa.us> (raw)
In-Reply-To: <20190816181418.GA898@alvherre.pgsql>
References: <20190816181418.GA898@alvherre.pgsql>
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> Somebody ran into issues when generating large XML output (upwards of
> 256 MB) and then sending via a connection with a different
> client_encoding. This occurs because we pessimistically allocate 4x as
> much memory as the string needs, and we run into the 1GB palloc
> limitation. ISTM we can do better now by using huge allocations, as per
> the preliminary attached patch (which probably needs an updated overflow
> check rather than have it removed altogether); but at least it is able
> to process this query, which it wasn't without the patch:
> select query_to_xml(
> 'select a, cash_words(a::text::money) from generate_series(0, 2000000) a',
> true, false, '');
I fear that allowing pg_do_encoding_conversion to return strings longer
than 1GB is just going to create failure cases somewhere else.
However, it's certainly true that 4x growth is a pretty unlikely worst
case. Maybe we could do something like
1. If string is short (say up to a few megabytes), continue to do it
like now. This avoids adding overhead for typical cases.
2. Otherwise, run some lobotomized form of encoding conversion that
just computes the space required (as an int64, I guess) without saving
the result anywhere.
3. If space required > 1GB, fail.
4. Otherwise, allocate just the space required, and convert.
regards, tom lane
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: tgl@sss.pgh.pa.us, alvherre@2ndquadrant.com, pgsql-hackers@lists.postgresql.org
Subject: Re: allocation limit for encoding conversion
In-Reply-To: <27990.1565991109@sss.pgh.pa.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox