From: Ronan Dunklau <ronan.dunklau@aiven.io>
To: Andres Freund <andres@anarazel.de>
Cc: pgsql-hackers@postgresql.org, Peter Eisentraut <peter@eisentraut.org>,
 tomas.vondra@enterprisedb.com
Subject: Re: Add GUC to tune glibc's malloc implementation.
Date: Tue, 27 Jun 2023 08:35:28 +0200
Message-ID: <8235070.T7Z3S40VBb@aivenlaptop>
In-Reply-To: <20230626210348.rk4h3bscdtcwbe7o@awork3.anarazel.de>
References:
 <3424675.QJadu78ljV@aivenlaptop> <2285512.ElGaqSPkdT@aivenlaptop>
 <20230626210348.rk4h3bscdtcwbe7o@awork3.anarazel.de>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="nextPart10293421.nUPlyArG6x"
Content-Transfer-Encoding: 7Bit
Archived-At: <https://www.postgresql.org/message-id/8235070.T7Z3S40VBb%40aivenlaptop>
Precedence: bulk

This is a multi-part message in MIME format.

--nextPart10293421.nUPlyArG6x
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="iso-8859-1"

Le lundi 26 juin 2023, 23:03:48 CEST Andres Freund a =E9crit :
> Hi,
>=20
> On 2023-06-26 08:38:35 +0200, Ronan Dunklau wrote:
> > I hope what I'm trying to achieve is clearer that way. Maybe this patch=
 is
> > not the best way to go about this, but since the memory allocator
> > behaviour can have such an impact it's a bit sad we have to leave half
> > the performance on the table because of it when there are easily
> > accessible knobs to avoid it.
> I'm *quite* doubtful this patch is the way to go.  If we want to more
> tightly control memory allocation patterns, because we have more
> information than glibc, we should do that, rather than try to nudge glibc=
's
> malloc in random direction.  In contrast a generic malloc() implementation
> we can have much more information about memory lifetimes etc due to memory
> contexts.

Yes this is probably much more appropriate, but a much larger change with=20
greater risks of regression. Especially as we have to make sure we're not=20
overfitting our own code for a specific malloc implementation, to the detri=
ment=20
of others. Except if you hinted we should write our own directly instead ?

>=20
> We e.g. could keep a larger number of memory blocks reserved
> ourselves. Possibly by delaying the release of additionally held blocks
> until we have been idle for a few seconds or such.

I think keeping work_mem around after it has been used a couple times make=
=20
sense. This is the memory a user is willing to dedicate to operations, afte=
r=20
all.

>=20
>=20
> WRT to the difference in TPS in the benchmark you mention - I suspect that
> we are doing something bad that needs to be improved regardless of the
> underlying memory allocator implementation.  Due to the lack of detailed
> instructions I couldn't reproduce the results immediately.

I re-attached the simple script I used. I've run this script with different=
=20
values for glibc_malloc_max_trim_threshold.=20

Best regards,

=2D-
Ronan Dunklau

--nextPart10293421.nUPlyArG6x
Content-Disposition: attachment; filename="bench.sh"
Content-Transfer-Encoding: base64
Content-Type: application/x-shellscript; name="bench.sh"

IyEvYmluL2Jhc2gKCnNlYz01CmRibmFtZT1wb3N0Z3JlcwpyZWNvcmRzPTEwMDAwCm1vZD0xMDAK
CnBzcWwgLWMgImRyb3AgdGFibGUgaWYgZXhpc3RzIHQiICRkYm5hbWUKcHNxbCAtYyAiY3JlYXRl
IHRhYmxlIHQgKGEgYmlnaW50IG5vdCBudWxsKSIgJGRibmFtZQpwc3FsIC1jICJpbnNlcnQgaW50
byB0IHNlbGVjdCB4ICUgJG1vZCBmcm9tIGdlbmVyYXRlX3NlcmllcygxLCRyZWNvcmRzKSB4IiAk
ZGJuYW1lCnBzcWwgLWMgInZhY3V1bSBmcmVlemUgdCIgJGRibmFtZQoKZm9yIGkgaW4gezEuLjMy
fQpkbwoJZWNobyAkaQoJcHNxbCAtYyAiZXhwbGFpbiBhbmFseXplIHNlbGVjdCAqIGZyb20gdCBv
cmRlciBieSBhIG9mZnNldCAxMDAwMDAwMDAwIiAkZGJuYW1lIHwgZ3JlcCAtRSAiTWVtb3J5fERp
c2siCgllY2hvICJzZWxlY3QgKiBmcm9tIHQgb3JkZXIgYnkgYSBvZmZzZXQgMTAwMDAwMDAwMCIg
PiBiZW5jaC5zcWwKCWZvciBsb29wcyBpbiB7MS4uM30KCWRvCgkJcGdiZW5jaCAtbiAtTSBwcmVw
YXJlZCAtVCAkc2VjIC1mIGJlbmNoLnNxbCAkZGJuYW1lIHwgZ3JlcCB0cHMKCWRvbmUKCglwc3Fs
IC1jICJhbHRlciB0YWJsZSB0IGFkZCBjb2x1bW4gYyRpIGJpZ2ludCIgJGRibmFtZQoJcHNxbCAt
YyAidXBkYXRlIHQgc2V0IGMkaSA9IGEiICRkYm5hbWUKCXBzcWwgLWMgInZhY3V1bSBmdWxsIHQi
ICRkYm5hbWUKCXBzcWwgLWMgInZhY3V1bSBmcmVlemUgdCIgJGRibmFtZQpkb25lCgoK


--nextPart10293421.nUPlyArG6x--