Message-ID: <70128d4f-e93a-4795-a11c-e1b94d7aca73@vondra.me>
Date: Wed, 15 Oct 2025 19:15:47 +0200
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Adding basic NUMA awareness
From: Tomas Vondra <tomas@vondra.me>
To: Alexey Makhmutov <a.makhmutov@postgrespro.ru>,
 PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Cc: Jakub Wartak <jakub.wartak@enterprisedb.com>,
 Andres Freund <andres@anarazel.de>
References: 
 <CAKZiRmy7VW5b0OKwHXtRSudCBFB48-kYnOEXeqrLfbTQ=5qQrA@mail.gmail.com>
 <c6493e5b-76c6-45a5-8067-3a36f4df4fe7@vondra.me>
 <b877e417-32c8-4864-b89e-cf6c66c1196d@vondra.me>
 <CAKZiRmxemXXVAouzM4Ls7pA7e0u6CVSLJeL_phKkmGPOzvUv_g@mail.gmail.com>
 <5a2410f8-62a9-4483-bf0a-3a8331fb0808@vondra.me>
 <71a46484-053c-4b81-ba32-ddac050a8b5d@vondra.me>
 <CAKZiRmx3+GwaP3oiRVHavxCJh6KxhZVZp86kj50ZJAv51h2-gQ@mail.gmail.com>
 <1b5bf372-6d40-4022-bf4f-367ffb61ee51@vondra.me>
 <51e51832-7f47-412a-a1a6-b972101cc8cb@vondra.me>
 <cffcaf59-2c33-476b-a8ff-dbb8d0a8b1d5@vondra.me>
 <mntwceou3ouc4usvktwutlbt6p3bqrzy73dw5nockzodhkud4g@7bchfsl3qpth>
 <874435bd-5e25-4a7c-a5d5-8ef0c262d788@vondra.me>
 <6636d842-e8a5-44bc-a206-6df415380a7b@vondra.me>
 <659c44a5-f616-492c-ab81-60273d2fe7f6@vondra.me>
 <bf95094a-77c2-46cf-913a-443f7419bc79@postgrespro.ru>
 <9c31717c-fe38-4824-9bb7-e4f0ac4f1991@vondra.me>
Content-Language: en-US
In-Reply-To: <9c31717c-fe38-4824-9bb7-e4f0ac4f1991@vondra.me>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Archived-At: 
 <https://www.postgresql.org/message-id/70128d4f-e93a-4795-a11c-e1b94d7aca73%40vondra.me>
Precedence: bulk


On 10/13/25 13:09, Tomas Vondra wrote:
> On 10/13/25 01:58, Alexey Makhmutov wrote:
>> Hi Tomas,
>>
>> Thank you very much for working on this problem and the entire line of
>> patches prepared! I've tried to play with these patches a little and
>> here are some my observations and suggestions.
>>
>> In the current implementation we try to use all available NUMA nodes on
>> the machine, however it's often useful to limit the database only to a
>> set of specific nodes, so that other nodes can be used for other
>> processes. In my testing I was trying to use one node out of four for
>> the client program, so I'd liked to limit the database to the remaining
>> nodes. I use a systemd service with AllowedMemoryNodes/AllowedCPUs to
>> start the cluster, so the obvious choice for me was to use the
>> 'numa_get_membind' function instead of 'numa_num_configured_nodes' to
>> get the list of usable nodes. However, it is much easier to work with
>> logical nodes in the [0; n] range inside the PG code, so I've decided to
>> add mapping between 'logical nodes' (0-n in PG) to a set of physical
>> nodes actually returned by 'numa_get_membind'. We may need to map number
>> in both directions, so two translation tables are allocated and filled
>> at the first usage of 'pg_numa' functions. It also seems to be a good
>> idea to isolate all 'libnuma' calls inside 'pg_numa.c', so to keep all
>> 'numa_...' calls in it and this also allows us to hide this mapping in
>> static functions. Here is the patch, which I've used to test this idea:
>> https://github.com/Lerm/postgres/
>> commit/9ec625c2bf564f5432375ec1d7ad02e4b2559161. This idea probably
>> could be extended by adding some view to expose this mapping to the user
>> (at least for testing purposes) and allow to explicitly override this
>> mapping with a GUC setting. With such GUC setting we would be able to
>> control PG memory usage on NUMA nodes without the need for systemd
>> resource control or numactl parameters.
>>
> 
> I've argued to keep this out of scope for v1, to keep it smaller and
> simpler. I'm not against adding that feature, though. If someone writes
> a patch to support this. I suppose the commit you linked is a step in
> that direction.
> 
On second thought, I probably spoke too soon ...

What I wanted to keep out of scope for v1 is ability to pick NUMA nodes
from Postgres, e.g. setting a GUC to limit which NUMA nodes to use, etc.

But that's not what you proposed here, clearly. You're saying we should
find which NUMA nodes the process is allowed to run, and use those.
Instead of just using all *configured* nodes. And I agree with that.

I'll take a look at your commit 9ec625c. I'm not sure it's a good idea
to have our internal "logical" node ID, and a mapping to external node
ID values (exposed by the libnuma). I was thinking maybe we should use
just the external IDs, but it's true that'd be tricky when iterating
through nodes, etc. So maybe having such mapping is a good approach.

Another thing I wasn't sure about is checking for memory-only nodes. For
example rpi5 has a NUMA node for each 1GB of memory, and each CPU is
mapped to all those nodes. For buffers this probably does not matter,
but we probably should not use those NUMA nodes for PGPROC partitioning.


regards

-- 
Tomas Vondra