From: Tom Lane <tgl@sss.pgh.pa.us>
To: Andres Freund <andres@anarazel.de>
cc: Alvaro Herrera <alvherre@2ndquadrant.com>,
        pgsql-hackers@lists.postgresql.org
Subject: Re: allocation limit for encoding conversion
In-reply-to: <20190924214204.mav4md77xg5u5wq6@alap3.anarazel.de>
References: <20190816181418.GA898@alvherre.pgsql>
 <27990.1565991109@sss.pgh.pa.us>
 <20190816220452.lg57jowanbsa2gf7@alap3.anarazel.de>
 <974.1569356381@sss.pgh.pa.us>
 <20190924214204.mav4md77xg5u5wq6@alap3.anarazel.de>
Comments: In-reply-to Andres Freund <andres@anarazel.de>
	message dated "Tue, 24 Sep 2019 14:42:04 -0700"
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <5883.1569362968.1@sss.pgh.pa.us>
Date: Tue, 24 Sep 2019 18:09:28 -0400
Message-ID: <5884.1569362968@sss.pgh.pa.us>
Precedence: bulk

Andres Freund <andres@anarazel.de> writes:
> On 2019-09-24 16:19:41 -0400, Tom Lane wrote:
>> Andres Freund <andres@anarazel.de> writes:
>>> It's probably too big a hammer for this specific case, but I think at
>>> some point we ought to stop using fixed size allocations for this kind
>>> of work. Instead we should use something roughly like our StringInfo,
>>> except that when exceeding the current size limit, the overflowing data
>>> is stored in a separate allocation. And only once we actually need the
>>> data in a consecutive form, we allocate memory that's large enough to
>>> store the all the separate allocations in their entirety.

>> That sounds pretty messy :-(.

> I don't think it's too bad - except for now allowing the .data member of
> such a 'chunked' StringInfo to be directly accessible, it'd be just
> about the same interface as the current stringinfo.

I dunno.  What you're describing would be a whole lotta work, and it'd
break a user-visible API, and no amount of finagling is going to prevent
it from making conversions somewhat slower, and the cases where it matters
to not preallocate a surely-large-enough buffer are really few and far
between.  I have to think that we have better ways to spend our time.

			regards, tom lane