Subject: Re: POC: converting Lists into arrays
To: Andres Freund <andres@anarazel.de>, Peter Geoghegan <pg@bowt.ie>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, Robert Haas <robertmhaas@gmail.com>,
 PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
References: <11587.1550975080@sss.pgh.pa.us>
 <CA+TgmoZVhSnXwXah-Atdn7mD3L+RNT_j4raq99wDejq7zR-vew@mail.gmail.com>
 <1325.1551118646@sss.pgh.pa.us>
 <CA+TgmoYft0QrxuwBhEXm5b9huB36KXEHJ+1cSyCifuQse7RLnw@mail.gmail.com>
 <2320.1551121176@sss.pgh.pa.us>
 <CAH2-Wz=xPuBfXezetjmrdH4HN0T8j7pzALtvu0+wnqs=xDKsSQ@mail.gmail.com>
 <20190225210344.26nf25wro3qt4q3o@alap3.anarazel.de>
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Message-ID: <3ccc28fb-a07c-3412-d48c-cc46de1ae42b@2ndquadrant.com>
Date: Mon, 25 Feb 2019 22:35:37 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
 Thunderbird/60.5.0
MIME-Version: 1.0
In-Reply-To: <20190225210344.26nf25wro3qt4q3o@alap3.anarazel.de>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk


On 2/25/19 10:03 PM, Andres Freund wrote:
> Hi,
> 
> On 2019-02-25 11:59:06 -0800, Peter Geoghegan wrote:
>> On Mon, Feb 25, 2019 at 10:59 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> Because it involves touching ten times more code (and that's a very
>>> conservative estimate).  Excluding changes in pg_list.h + list.c,
>>> what I posted touches approximately 600 lines of code (520 insertions,
>>> 644 deletions to be exact).  For comparison's sake, there are about
>>> 1800 uses of foreach in the tree, each of which would require at least
>>> 3 changes to replace (the foreach itself, the ListCell variable
>>> declaration, and at least one lfirst() reference in the loop body).
>>
>> If we knew that the list code was the bottleneck in a handful of
>> cases, then I'd come down on Robert's side here. It would then be
>> possible to update the relevant bottlenecked code in an isolated
>> fashion, while getting the lion's share of the benefit. However, I
>> don't think that that's actually possible. The costs of using Lists
>> everywhere are real and measurable, but it's also highly distributed.
>> At least, that's my recollection from previous discussion from several
>> years back. I remember talking about this with Andres in early 2016.
> 
> It's distributed, but not *that* distributed. The largest source of
> "cost" at execution time used to be all-over expression evaluation, but
> that's gone now. That was a lot of places, but it's not outside of reach
> of a targeted change. Now it's targetlist handling, which'd have to
> change together with plan time, where it's a large issue.
> 

So let's say we want to measure the improvement this patch gives us.
What would be some reasonable (and corner) cases to benchmark? I do have
some ideas, but as you've been looking at this in the past, perhaps you
have something better.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services