Message-ID: <70b689dc-7ad4-138c-2c0b-9fcc131a1aaf@enterprisedb.com>
Date: Thu, 16 Dec 2021 18:00:56 +0100
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.2.0
Subject: Re: Use generation context to speed up tuplesorts
Content-Language: en-US
To: Ronan Dunklau <ronan.dunklau@aiven.io>,
 David Rowley <dgrowleyml@gmail.com>, pgsql-hackers@lists.postgresql.org
Cc: Andres Freund <andres@anarazel.de>, Tomas Vondra <tv@fuzzy.cz>
References: 
 <CAApHDvoH4ASzsAOyHcxkuY01Qf++8JJ0paw+03dk+W25tQEcNQ@mail.gmail.com>
 <567d1ea7-bfeb-bd97-1a7f-b13d2770258c@enterprisedb.com>
 <2740198.88bMQJbFj6@aivenronan> <36444569.XM6RcZxFsP@aivenronan>
From: Tomas Vondra <tomas.vondra@enterprisedb.com>
In-Reply-To: <36444569.XM6RcZxFsP@aivenronan>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Archived-At: 
 <https://www.postgresql.org/message-id/70b689dc-7ad4-138c-2c0b-9fcc131a1aaf%40enterprisedb.com>
Precedence: bulk

On 12/16/21 17:03, Ronan Dunklau wrote:
> Le jeudi 16 décembre 2021, 11:56:15 CET Ronan Dunklau a écrit :
>> I will follow up with a benchmark of the test sorting a table with a width
>> varying from 1 to 32 columns.
>>
> 
> So please find attached another benchmark for that case.
> 
> The 3 different patchsets tested are:
> 
>   - master
>   - fixed (David's original patch)
>   - adjust (Thomas growing blocks patch)
> 

Presumably Thomas is me, right?

> So it looks like tuning malloc for this would be very benificial for any kind
> of allocation, and by doing so we reduce the problems seen with the growing
> blocks patch to next to nothing, while keeping the ability to not allocate too
> much memory from the get go.
> 

Thanks for running those tests and investigating the glibc behavior! I 
find those results very interesting. My conclusions from this is that 
the interaction interaction between "our" allocator and the allocator in 
malloc (e.g. glibc) can be problematic. Which makes benchmarking and 
optimization somewhat tricky because code changes may trigger behavior 
change in glibc (or whatever allocator backs malloc).

I think it's worth exploring if we can tune this in a reasonable way, 
but I have a couple concerns related to that:

1) I wonder how glibc-specific this is - I'd bet it applies to other 
allocators (either on another OS or just different allocator on Linux) 
too. Tweaking glibc parameters won't affect those systems, of course, 
but maybe we should allow tweaking those systems too ...

2) In fact, I wonder if different glibc versions behave differently? 
Hopefully it's not changing that much, though. Ditto kernel versions, 
but the mmap/sbrk interface is likely more stable. We can test this.

3) If we bump the thresholds, won't that work against reusing the 
memory? I mean, if we free a whole block (from any allocator we have), 
glibc might return it to kernel, depending on mmap threshold value. It's 
not guaranteed, but increasing the malloc thresholds will make that even 
less likely. So we might just as well increase the minimum block size, 
with about the same effect, no?


> I would like to try to implement some dynamic glibc malloc tuning, if that is
> something we don't reject on principle from the get go.
> 

+1 to that


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company