From: Tomas Vondra <tomas.vondra@enterprisedb.com>
To: Ronan Dunklau <ronan.dunklau@aiven.io>
To: David Rowley <dgrowleyml@gmail.com>
To: pgsql-hackers@lists.postgresql.org
Cc: Andres Freund <andres@anarazel.de>
Cc: Tomas Vondra <tv@fuzzy.cz>
Subject: Re: Use generation context to speed up tuplesorts
Date: Thu, 16 Dec 2021 18:00:56 +0100
Message-ID: <70b689dc-7ad4-138c-2c0b-9fcc131a1aaf@enterprisedb.com> (raw)
In-Reply-To: <36444569.XM6RcZxFsP@aivenronan>
References: <CAApHDvoH4ASzsAOyHcxkuY01Qf++8JJ0paw+03dk+W25tQEcNQ@mail.gmail.com>
<567d1ea7-bfeb-bd97-1a7f-b13d2770258c@enterprisedb.com>
<2740198.88bMQJbFj6@aivenronan>
<36444569.XM6RcZxFsP@aivenronan>
On 12/16/21 17:03, Ronan Dunklau wrote:
> Le jeudi 16 décembre 2021, 11:56:15 CET Ronan Dunklau a écrit :
>> I will follow up with a benchmark of the test sorting a table with a width
>> varying from 1 to 32 columns.
>>
>
> So please find attached another benchmark for that case.
>
> The 3 different patchsets tested are:
>
> - master
> - fixed (David's original patch)
> - adjust (Thomas growing blocks patch)
>
Presumably Thomas is me, right?
> So it looks like tuning malloc for this would be very benificial for any kind
> of allocation, and by doing so we reduce the problems seen with the growing
> blocks patch to next to nothing, while keeping the ability to not allocate too
> much memory from the get go.
>
Thanks for running those tests and investigating the glibc behavior! I
find those results very interesting. My conclusions from this is that
the interaction interaction between "our" allocator and the allocator in
malloc (e.g. glibc) can be problematic. Which makes benchmarking and
optimization somewhat tricky because code changes may trigger behavior
change in glibc (or whatever allocator backs malloc).
I think it's worth exploring if we can tune this in a reasonable way,
but I have a couple concerns related to that:
1) I wonder how glibc-specific this is - I'd bet it applies to other
allocators (either on another OS or just different allocator on Linux)
too. Tweaking glibc parameters won't affect those systems, of course,
but maybe we should allow tweaking those systems too ...
2) In fact, I wonder if different glibc versions behave differently?
Hopefully it's not changing that much, though. Ditto kernel versions,
but the mmap/sbrk interface is likely more stable. We can test this.
3) If we bump the thresholds, won't that work against reusing the
memory? I mean, if we free a whole block (from any allocator we have),
glibc might return it to kernel, depending on mmap threshold value. It's
not guaranteed, but increasing the malloc thresholds will make that even
less likely. So we might just as well increase the minimum block size,
with about the same effect, no?
> I would like to try to implement some dynamic glibc malloc tuning, if that is
> something we don't reject on principle from the get go.
>
+1 to that
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: tomas.vondra@enterprisedb.com, ronan.dunklau@aiven.io, dgrowleyml@gmail.com, pgsql-hackers@lists.postgresql.org, andres@anarazel.de, tv@fuzzy.cz
Subject: Re: Use generation context to speed up tuplesorts
In-Reply-To: <70b689dc-7ad4-138c-2c0b-9fcc131a1aaf@enterprisedb.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox