Subject: Re: slab allocator performance issues
From: Tomas Vondra <tomas.vondra@enterprisedb.com>
To: Andres Freund <andres@anarazel.de>
Cc: pgsql-hackers@postgresql.org, Tomas Vondra <tv@fuzzy.cz>
References: <20210717194333.mr5io3zup3kxahfm@alap3.anarazel.de>
 <c3bf99d3-0dec-828e-e3b5-ed2d2922bdd8@enterprisedb.com>
 <20210717211405.3ocp2rpxlnozg3tq@alap3.anarazel.de>
 <02ff3ced-3212-aea1-c7a0-4829f43e58d2@enterprisedb.com>
 <20210717231019.blfhyqniu6jed3ym@alap3.anarazel.de>
 <20210718010600.fat2f5j4gvqig56j@alap3.anarazel.de>
 <37dc9b41-dd75-251d-a77e-16c72c794b00@enterprisedb.com>
 <20210719205603.lzh5rkuv7lsoq4mp@alap3.anarazel.de>
 <6717ca90-a2ea-b17d-f544-37f5181b1175@enterprisedb.com>
 <20210801210722.kctrnyxwwu4mzail@alap3.anarazel.de>
 <76ebb476-4060-393b-5bf1-0287ae1464bd@enterprisedb.com>
 <5d83ac71-9054-208d-a48a-24d9d6b61f55@enterprisedb.com>
Message-ID: <a5ccda91-d9fc-49c5-b3c7-c81528b938c5@enterprisedb.com>
Date: Fri, 10 Sep 2021 23:06:54 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.11.0
MIME-Version: 1.0
In-Reply-To: <5d83ac71-9054-208d-a48a-24d9d6b61f55@enterprisedb.com>
Content-Type: multipart/mixed;
 boundary="------------EB9D15C84ED65C09FD220441"
Content-Language: en-US
Archived-At: 
 <https://www.postgresql.org/message-id/a5ccda91-d9fc-49c5-b3c7-c81528b938c5%40enterprisedb.com>
Precedence: bulk

This is a multi-part message in MIME format.
--------------EB9D15C84ED65C09FD220441
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

Hi,

I've been investigating the regressions in some of the benchmark 
results, together with the generation context benchmarks [1].

Turns out it's pretty difficult to benchmark this, because the results 
strongly depend on what the backend did before. For example if I run 
slab_bench_fifo with the "decreasing" test for 32kB blocks and 512B 
chunks, I get this:

   select * from slab_bench_fifo(1000000, 32768, 512, 100, 10000, 5000);

    mem_allocated | alloc_ms | free_ms
   ---------------+----------+---------
        528547840 |   155394 |   87440


i.e. palloc() takes ~155ms and pfree() ~87ms (and these result are 
stable, the numbers don't change much with more runs).

But if I run a set of "lifo" tests in the backend first, the results 
look like this:

    mem_allocated | alloc_ms | free_ms
   ---------------+----------+---------
        528547840 |    41728 |   71524
   (1 row)

so the pallocs are suddenly about ~4x faster. Clearly, what the backend 
did before may have pretty dramatic impact on results, even for simple 
benchmarks like this.

Note: The benchmark was a single SQL script, running all the different 
workloads in the same backend.

I did a fair amount of perf profiling, and the main difference between 
the slow and fast runs seems to be this:

                  0      page-faults:u 

                  0      minor-faults:u 

                  0      major-faults:u 


vs

         20,634,153      page-faults:u 

         20,634,153      minor-faults:u 

                  0      major-faults:u 


Attached is a more complete perf stat output, but the page faults seem 
to be the main issue. My theory is that in the "fast" case, the past 
backend activity puts the glibc memory management into a state that 
prevents page faults in the benchmark.

But of course, this theory may be incomplete - for example it's not 
clear why running the benchmark repeatedly would not "condition" the 
backend the same way. But it doesn't - it's ~150ms even for repeated runs.

Secondly, I'm not sure this explains why some of the timings actually 
got much slower with the 0003 patch, when the sequence of the steps is 
still the same. Of course, it's possible 0003 changes the allocation 
pattern a bit, interfering with glibc memory management.

This leads to a couple of interesting questions, I think:

1) I've only tested this on Linux, with glibc. I wonder how it'd behave 
on other platforms, or with other allocators.

2) Which cases are more important? When the backend was warmed up, or 
when each benchmark runs in a new backend? It seems the "new backend" is 
something like a "worst case" leading to more page faults, so maybe 
that's the thing to watch. OTOH it's unlikely to have a completely new 
backend, so maybe not.

3) Can this teach us something about how to allocate stuff, to better 
"prepare" the backend for future allocations? For example, it's a bit 
strange that repeated runs of the same benchmark don't do the trick, for 
some reason.


regards


[1] 
https://www.postgresql.org/message-id/bcdd4e3e-c12d-cd2b-7ead-a91ad416100a%40enterprisedb.com

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--------------EB9D15C84ED65C09FD220441
Content-Type: text/plain; charset=UTF-8;
 name="perf.stat.txt"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="perf.stat.txt"

IFBlcmZvcm1hbmNlIGNvdW50ZXIgc3RhdHMgZm9yIHByb2Nlc3MgaWQgJzExODI5JzoKCiAg
IDIxOSw4NjksNzM5LDczNyAgICAgIGN5Y2xlczp1ICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAxMTksNzIyLDI4MCw0MzYgICAgICBp
bnN0cnVjdGlvbnM6dSAgICAgICAgICAgICMgICAgMC41NCAgaW5zbiBwZXIgY3ljbGUgICAg
ICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAjICAgIDEuNTYgIHN0YWxsZWQgY3ljbGVzIHBlciBpbnNuCiAgICAgNyw3ODQsNTY2LDg1
OCAgICAgIGNhY2hlLXJlZmVyZW5jZXM6dSAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgIAogICAgIDIsNDg3LDI1NywyODcgICAgICBjYWNoZS1taXNzZXM6dSAg
ICAgICAgICAgICMgICAzMS45NTEgJSBvZiBhbGwgY2FjaGUgcmVmcyAgICAKICAgICA1LDk0
MiwwNTQsNTIwICAgICAgYnVzLWN5Y2xlczp1ICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgMCAgICAgIHBhZ2UtZmF1
bHRzOnUgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAog
ICAgICAgICAgICAgICAgIDAgICAgICBtaW5vci1mYXVsdHM6dSAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAwICAgICAg
bWFqb3ItZmF1bHRzOnUgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgCiAgIDE4NywxODEsNjYxLDcxOSAgICAgIHN0YWxsZWQtY3ljbGVzLWZyb250ZW5k
OnUgIyAgIDg1LjEzJSBmcm9udGVuZCBjeWNsZXMgaWRsZSAgIAogICAxNDQsMjc0LDAxNyww
NzEgICAgICBzdGFsbGVkLWN5Y2xlcy1iYWNrZW5kOnUgICMgICA2NS42MiUgYmFja2VuZCBj
eWNsZXMgaWRsZSAgICAKCiAgICAgIDYwLjAwMDg3NjI0OCBzZWNvbmRzIHRpbWUgZWxhcHNl
ZAoKCgogUGVyZm9ybWFuY2UgY291bnRlciBzdGF0cyBmb3IgcHJvY2VzcyBpZCAnMTE4ODYn
OgoKICAgMTQ1LDA5MywwOTAsNjkyICAgICAgY3ljbGVzOnUgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICA3NCw5ODYsNTQzLDIxMiAg
ICAgIGluc3RydWN0aW9uczp1ICAgICAgICAgICAgIyAgICAwLjUyICBpbnNuIHBlciBjeWNs
ZSAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICMgICAgMi42MyAgc3RhbGxlZCBjeWNsZXMgcGVyIGluc24KICAgICA0LDc1Myw3
NjQsNzgxICAgICAgY2FjaGUtcmVmZXJlbmNlczp1ICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgCiAgICAgMSwzNDIsNjUzLDU0OSAgICAgIGNhY2hlLW1pc3Nl
czp1ICAgICAgICAgICAgIyAgIDI4LjI0NCAlIG9mIGFsbCBjYWNoZSByZWZzICAgIAogICAg
IDMsOTI1LDE3NSw1MTUgICAgICBidXMtY3ljbGVzOnUgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAyMCw2MzQsMTUzICAgICAgcGFn
ZS1mYXVsdHM6dSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgCiAgICAgICAgMjAsNjM0LDE1MyAgICAgIG1pbm9yLWZhdWx0czp1ICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgIDAg
ICAgICBtYWpvci1mYXVsdHM6dSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAKICAgMTk3LDEzMCw0NjEsNjMyICAgICAgc3RhbGxlZC1jeWNsZXMtZnJv
bnRlbmQ6dSAjICAxMzUuODYlIGZyb250ZW5kIGN5Y2xlcyBpZGxlICAgCiAgIDE2OCw0MzQs
MzQzLDIxMyAgICAgIHN0YWxsZWQtY3ljbGVzLWJhY2tlbmQ6dSAgIyAgMTE2LjA5JSBiYWNr
ZW5kIGN5Y2xlcyBpZGxlICAgIAoKICAgICAgNjAuMDAwODkxODY3IHNlY29uZHMgdGltZSBl
bGFwc2VkCg==
--------------EB9D15C84ED65C09FD220441--