From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Petr Jelinek <petr@2ndquadrant.com>
To: Robert Haas <robertmhaas@gmail.com>
Cc: John Gorman <johngorman2@gmail.com>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: PATCH: two slab-like memory allocators
Date: Thu, 20 Oct 2016 15:35:22 +0200
Message-ID: <ae88ac6a-526d-d7c2-2304-375d74d5696f@2ndquadrant.com> (raw)
In-Reply-To: <e88baf4f-afcc-3b48-0524-89a71527f6e3@2ndquadrant.com>
References: <d15dff83-0b37-28ed-0809-95a5cc7292ad@2ndquadrant.com>
<c5653004-fe91-5a13-409c-db9f34642b0c@2ndquadrant.com>
<68b837a3-ce55-f9a2-f685-9eafac536f21@2ndquadrant.com>
<885540a7-ac72-5a21-6dae-a4d5b50672da@2ndquadrant.com>
<03110c11-ea72-7b4d-027d-6f19eb300b2a@2ndquadrant.com>
<CALkS6B9W0xHS25OOq2sWhUWtS79XuPiiAOFpsh6F8v56uAvy_Q@mail.gmail.com>
<e342a3f5-6fd2-0c1f-ca12-f6b1dd7ad88f@2ndquadrant.com>
<CALkS6B8hisTEJJ7ZCQ_GiBLNVzKfsevAD+92rYN6myyFK=UQtw@mail.gmail.com>
<11821eb9-95a5-353e-ccba-13e8e9351e62@2ndquadrant.com>
<b8ded0aa-2515-7ffd-162b-8f515df5eca1@2ndquadrant.com>
<CA+Tgmob7u4thiBbdvyseQk1aVApJeG10C_rFFt8LPV03-UzD7g@mail.gmail.com>
<a4593432-c989-8ec4-2320-943e3f97ce6d@2ndquadrant.com>
<e88baf4f-afcc-3b48-0524-89a71527f6e3@2ndquadrant.com>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
On 10/19/2016 02:51 PM, Tomas Vondra wrote:
> ...
>
> Yeah. There are three contexts in reorder buffers:
>
> - changes (fixed size)
> - txns (fixed size)
> - tuples (variable size)
>
> The first two work perfectly fine with Slab.
>
> The last one (tuples) is used to allocate variable-sized bits, so I've
> tried to come up with something smart - a sequence of Slabs + overflow
> AllocSet. I agree that in hindsight it's a bit strange, and that the
> "generational" aspect is the key aspect here - i.e. it might be possible
> to implement a memory context that allocates variable-length chunks but
> still segregates them into generations. That is, don't build this on top
> of Slab. That would also fix the issue with two allocators in GenSlab.
> I'll think about this.
And here is a fairly complete prototype of this idea, adding "Gen"
generational memory context based only on the "similar lifespan"
assumption (and abandoning the fixed-size assumption). It's much simpler
than GenSlab (which it's supposed to replace), and abandoning the idea
of composing two memory contexts also fixed the warts with some of the
API methods (e.g. repalloc).
I've also been thinking that perhaps "Gen" would be useful for all three
contexts in ReorderBuffer - so I've done a quick test comparing the
various combinations (using the test1() function used before).
master slabs slabs+genslab slabs+gen gens
----------------------------------------------------------------
50k 18700 210 220 190 190
100k 160000 380 470 350 350
200k N/A 750 920 740 679
500k N/A 2250 2240 1790 1740
1000k N/A 4600 5000 3910 3700
Where:
* master - 23ed2ba812117
* slabs - all three contexts use Slab (patch 0001)
* slabs+genslab - third context is GenSlab (patch 0002)
* slabs+gen - third context is the new Gen (patch 0003)
* gens - all three contexts use Gen
The results are a bit noisy, but I think it's clear the new Gen context
performs well - it actually seems a bit faster than GenSlab, and using
only Gen for all three contexts does not hurt peformance.
This is most likely due to the trivial (practically absent) freespace
management in Gen context, compared to both Slab and GenSlab. So the
speed is not the only criteria - I haven't measured memory consumption,
but I'm pretty sure there are cases where Slab consumes much less memory
than Gen, thanks to reusing free space.
I'd say throwing away GenSlab and keeping Slab+Gen is the way to go.
There's still a fair bit of work on this, particularly implementing the
missing API methods in Gen - GenCheck() and GenStats(). As Robert
pointed out, there's also quite a bit of duplicated code between the
different memory contexts (randomization and valgrind-related), so this
needs to be moved to a shared place.
I'm also thinking that we need better names, particularly for the Gen
allocator. It's supposed to mean Generational, although there are no
explicit generations anymore. Slab is probably OK - it does not match
any of the existing kernel slab allocators exactly, but it follows the
same principles, which is what matters.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackersAttachments:
[application/x-compressed-tar] slab-allocators-v4.tgz (21.5K, ../ae88ac6a-526d-d7c2-2304-375d74d5696f@2ndquadrant.com/2-slab-allocators-v4.tgz)
download
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: tomas.vondra@2ndquadrant.com, petr@2ndquadrant.com, robertmhaas@gmail.com, johngorman2@gmail.com
Subject: Re: PATCH: two slab-like memory allocators
In-Reply-To: <ae88ac6a-526d-d7c2-2304-375d74d5696f@2ndquadrant.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox