From: Tom Lane <tgl@sss.pgh.pa.us>
To: Simon Riggs <simon@2ndquadrant.com>
cc: Andres Freund <andres@anarazel.de>, Tomas Vondra <tv@fuzzy.cz>,
        pgsql-committers <pgsql-committers@postgresql.org>
Subject: Re: pgsql: Generational memory allocator
In-reply-to: 
 <CANP8+jKRt0Yenw8Ln4J4u=UsDVkCBgPmjDE2buYou-H9VTMZjw@mail.gmail.com>
References: <E1eHa4J-0006hI-Q8@gemulon.postgresql.org>
 <20171123001649.y2nf7422iay6cw3v@alap3.anarazel.de>
 <CANP8+jKoT=Pg5W_3FcKiDf+FMrJhFYacxt_g5E0=tkmb5s8M6g@mail.gmail.com>
 <c903f275-2150-fa52-64bf-dca7b53ebf8d@fuzzy.cz>
 <16449.1511548313@sss.pgh.pa.us> <4618.1511573138@sss.pgh.pa.us>
 <bf84d940-90d4-de91-19dd-612e011007f4@fuzzy.cz>
 <6189.1511587105@sss.pgh.pa.us> <8991.1511639441@sss.pgh.pa.us>
 <20171125200014.qbewtip5oydqsklt@alap3.anarazel.de>
 <21466.1511644760@sss.pgh.pa.us> <2875.1511646362@sss.pgh.pa.us>
 <CANP8+jKRt0Yenw8Ln4J4u=UsDVkCBgPmjDE2buYou-H9VTMZjw@mail.gmail.com>
Comments: In-reply-to Simon Riggs <simon@2ndquadrant.com>
	message dated "Sun, 26 Nov 2017 20:33:01 +1100"
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <16796.1511718406.1@sss.pgh.pa.us>
Date: Sun, 26 Nov 2017 12:46:46 -0500
Message-ID: <16797.1511718406@sss.pgh.pa.us>
Precedence: bulk

Simon Riggs <simon@2ndquadrant.com> writes:
> On 26 November 2017 at 08:46, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> I've confirmed that the attached is sufficient to stop the valgrind crash
>> on my machine.  But as I said, I think we should be more aggressive at
>> resizing the buffer, to reduce resize cycles.  I'm inclined to start out
>> with a buffer size of 128 or 256 or so bytes and double it when needed.
>> Anybody have a feeling for a typical size for the "main data" part
>> of a WAL record?

> We reuse the buffer and only pfree/palloc when we need to enlarge the
> buffer, so not sure we need to do the doubling thing and it probably
> doesn't matter what the typical size is.

Well, I'm concerned about the possibility of a lot of palloc thrashing
if the first bunch of records it reads happen to have steadily increasing
sizes.  However, rather than doubling, it might be sufficient to set a
robust minimum on the first allocation, ie use something along the lines
of Max(1024, MAXALIGN(state->main_data_len)).

			regards, tom lane