Subject: Re: NUMA shared memory interleaving

From: Bertrand Drouvot <[email-masked]>
Date: Fri, 18 Apr 2025 17:43:14 +0000

Hi,

On Wed, Apr 16, 2025 at 10:05:04AM -0400, Robert Haas wrote:
> On Wed, Apr 16, 2025 at 5:14 AM Jakub Wartak
> <[email-masked]> wrote:
> > Normal pgbench workloads tend to be not affected, as each backend
> > tends to touch just a small partition of shm (thanks to BAS
> > strategies). Some remaining questions are:
> > 1. How to name this GUC (numa or numa_shm_interleave) ? I prefer the
> > first option, as we could potentially in future add more optimizations
> > behind that GUC.
> 
> I wonder whether the GUC needs to support interleaving between a
> designated set of nodes rather than only being able to do all nodes.
> For example, suppose someone is pinning the processes to a certain set
> of NUMA nodes; perhaps then they wouldn't want to use memory from
> other nodes.

+1. That could be used for instances consolidation on the same host. One could
ensure that numa nodes are not shared across instances (cpu and memory resource
isolation per instance). Bonus point, adding Direct I/O into the game would
ensure that the OS page cache is not shared too.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com

---

From: Jakub Wartak <[email-masked]>
Date: Thu, 26 Jun 2025 13:31:39 +0200

> ensure that the OS page cache is not shared too.

Hi, the attached patch has two changes:
1. It adds more modes and supports this 'consolidation' and
'isolation' scenario from above. Doc in patch briefly explains the
merit.
2. it adds trivial NUMA for PQ

The original initial test expanded on the very same machine
(4s32c128t, QPI interconnect):

numa='off'
    latency average = 1271.019 ms
    latency stddev = 245.061 ms
    tps = 49.683923 (without initial connection time)
    explanation(pcm-memory): 3 sockets doing ~46MB/s on RAM (almost
idle), 1 socket doing ~17GB/s (fully saturated because s_b ended up in
this scenario only on NUMA node)

numa='all'
    latency average = 702.622 ms
    latency stddev = 13.259 ms
    tps = 90.026526 (without initial connection time)
    explanation(pcm-memory): this forced to interleave s_b across 4
NUMA nodes and each socket gets equal part of workload (9.2 - 10GB/s)
totalling ~37GB/s of memory bandwidth

This gives a boost: 90/49.6=1.8x. The values for memory bandwidth are
combined read+write.

NUMA impact on the Parallel Query:
----------------------------------
with:
    with the most simplistic interleaving of s_b +
dynamic_shared_memory for PQ interleaved too :
    max_worker_processes=max_parallel_workers=max_parallel_workers_per_gather=64
    alter on 1 partition to force real 64 parallel seq scans
The query:
    select sum(octet_length(filler)) from pgbench_accounts;
launched 64 effective parallel workes launched for 64 partitions each
of 400MB  (25600MBs), All of that was fitting in the s_b (32GB), so
all fetched from s_b. All was hot, several first runs are not shown.

select sum(octet_length(filler)) from pgbench_accounts;

numa='off'
    Time: 1108.178 ms (00:01.108)
    Time: 1118.494 ms (00:01.118)
    Time: 1104.491 ms (00:01.104)
    Time: 1112.221 ms (00:01.112)
    Time: 1105.501 ms (00:01.106)
    avg: 1109 ms

    -- not interleaved, more like appended:
    postgres=# select * from pg_shmem_allocations_numa where name =
'Buffer Blocks';
         name      | numa_node |    size
    ---------------+-----------+------------
     Buffer Blocks |         0 | 9277800448
     Buffer Blocks |         1 | 7044333568
     Buffer Blocks |         2 | 9097445376
     Buffer Blocks |         3 | 8942256128

numa='all'
    Time: 1026.747 ms (00:01.027)
    Time: 1024.087 ms (00:01.024)
    Time: 1024.179 ms (00:01.024)
    Time: 1037.026 ms (00:01.037)
    avg: 1027 ms

        postgres=# select * from pg_shmem_allocations_numa where name
= 'Buffer Blocks';
         name      | numa_node |    size
    ---------------+-----------+------------
     Buffer Blocks |         0 | 8589934592
     Buffer Blocks |         1 | 8592031744
     Buffer Blocks |         2 | 8589934592
     Buffer Blocks |         3 | 8589934592

1109/1027=1.079x, not bad for such trivial change and the paper
referenced by Thomas also stated (`We can see an improvement by a
factor of more than three by just running
the non-NUMA-aware implementation on interleaved memor`), probably it
could be improved much further, but I'm not planning to work on this
more. So if anything:
- latency-wise: it would be best to place leader+all PQ workers close
to s_b, provided s_b fits NUMA shared/huge page memory there and you
won't need more CPU than there's on that NUMA node... (assuming e.g.
hosting 4 DBs on 4-sockets each on it's own, it would be best to pin
everything including shm, but PQ workers too)
- capacity/TPS-wise or s_b > NUMA: just interleave to maximize
bandwidth and get uniform CPU performance out of this

The patch supports e.g. numa='@1' which should fully isolate the
workload to just memory and CPUs on node #1.
As for the patch: I'm lost with our C headers policy :)

One of less obvious reasons (outside of better efficiency of
consolidation of multiple PostgreSQL cluster on single NUMA server),
why I've implemented '=' and '@' is that seems that CXL memory can be
attached as a CPU-less(!) NUMA node, thus Linux - depending on
sysctls/sysfs setup - could use it for automatic memory tiering and
the above provides configurable way to prevent allocation on such
(potential) systems - simply exclude such NUMA node via config for now
and we are covered I think. I have no access to real hardware, so I
haven't researched it further, but it looks like in the far future we
could probably indicate preferred NUMA memory nodes (think big s_b,
bigger than "CPU" RAM), and that kernel could transparently do NUMA
auto balancing/demotion for us (AKA Transparent Page Placement AKA
memory) or vice versa: use small s_b and do not use CXL node at all
and expect that VFS cache could be spilled there.
numa_weighted_interleave_memory() / MPOL_WEIGHTED_INTERLEAVE is not
yet supported in distros (although new libnuma has support for it), so
I have not included it in the patch, as it was too early.

BTW: DO NOT USE meson's --buildtype=debug as it somehow disables the
NUMA optimizations benefit, I've lost hours on it (probably -O0 is so
slow that it wasn't stressing interconnects enough). Default is
--buildtype=debugoptimized which is good. Also if testing performance,
check that HW that has proper realistic NUMA remote access distances
first, e.g. here my remote had remote access 2x or even 3x. Probably
this is worth only testing on multi-sockets which have really higher
latencies/throughput limitations, but reports from 1 socket MCMs CPUs
(with various Node-per-Socket BIOS settings) are welcome too.

kernel 6.14.7 was used with full isolation:
    cpupower frequency-set --governor performance
    cpupower idle-set -D0
    echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo
    echo never > /sys/kernel/mm/transparent_hugepage/enabled
    echo never > /sys/kernel/mm/transparent_hugepage/defrag

max_connections = '10000'
huge_pages = 'on'
wal_level = 'minimal'
wal_buffers = '1024MB'
max_wal_senders = '0'
shared_buffers = '4 GB'
autovacuum = 'off'
max_parallel_workers_per_gather = '0'
numa = 'all'
#numa = 'off'

[1] - https://lwn.net/Articles/897536/

---

From: Tomas Vondra <[email-masked]>
Date: Fri, 27 Jun 2025 18:41:30 +0200

Hi,

I agree we should improve the behavior on NUMA systems. But I'm not sure
this patch does far enough, or perhaps the approach seems a bit too
blunt, ignoring some interesting stuff.

AFAICS the patch essentially does the same thing as

   numactl --interleave=all

except that it only does that to shared memory, not to process private
memory (as if we called numa_set_localalloc). Which means it has some of
the problems people observe with --interleave=all.

In particular, this practically guarantees that (with 4K memory pages),
each buffer hits multiple NUMA nodes. Because with the first half will
do to node N, while the second half goes to node (N+1).

That doesn't seem great. It's likely better than a misbalanced system
with everything allocated on a single NUMA node, but I don't see how it
could be better than "balanced" properly warmed up system where the
buffers are not split like this.

But OK, admittedly this only happens for 4K memory pages, and serious
systems with a lot of memory are likely to use huge pages, which makes
this less of an issue (only the buffers crossing the page boundaries
might get split).

My bigger comment however is that the approach focuses on balancing the
nodes (i.e. ensuring each node gets a fair share of shared memory), and
is entirely oblivious to the internal structure of the shared memory.

* It interleaves the shared segment, but it has many pieces - shared
buffers are the largest but not the only one. Does it make sense to
interleave all the other pieces?

* Some of the pieces are tightly related. For example, we talk about
shared buffers as if it was one big array, but it actually is two arrays
- blocks and descriptors. Even if buffers don't get split between nodes
(thanks to huge pages), there's no guarantee the descriptor for the
buffer does not end on a different node.

* In fact, the descriptors are so much smaller that blocks that it's
practically guaranteed all descriptors will end up on a single node.

I could probably come up with a couple more similar items, but I think
you get the idea. I do think making Postgres NUMA-aware will require
figuring out how to distribute (or not distribute) different parts of
the shared memory, and do that explicitly. And do that in a way that
allows us to do other stuff in NUMA-aware way, e.g. have a separate
freelists and clocksweep for each NUMA node, etc.

That's something numa_interleave_memory simply can't do for us, and I
suppose it might also have other downsides on large instances. I mean,
doesn't it have to create a separate mapping for each memory page?
Wouldn't that be a bit inefficient/costly for big instances?

Of course, I'm not saying all this as a random passerby - I've been
working on a similar patch for a while, based on Andres' experimental
NUMA branch. It's far from complete/perfect, more of a PoC quality, but
I hope to share it on the mailing list sometime soon.

FWIW while I think the patch doesn't go far enough, there's one area
where I think it probably goes way too far - configurability. I agree
it's reasonable to allow running on a subset of nodes, e.g. to split the
system between multiple instances etc. But do we need to configure that
from Postgres? Aren't people likely to already use something like
containers or k8 anyway? I think we should just try to inherit this from
the environment, i.e. determine which nodes we're allowed to run, and
use that. Maybe we'll find we need to be smarter, but I think we caan
leave that for later.

regards

-- 
Tomas Vondra

---

From: Jakub Wartak <[email-masked]>
Date: Mon, 30 Jun 2025 12:55:13 +0200

Hi Tomas!

> freelists and clocksweep for each NUMA node, etc.

I do understand what you mean, but I'm *NOT* stating here that it
makes PG fully "NUMA-aware". I actually try to avoid doing so with
each sentence. This is only about the imbalance problem specifically.
I think we could build those follow-up optimizations as separate
patches in this or follow-up threads. If we would do it all in one
giant 0001 (without split) the very first question would be to
quantify the impact of each of those optimizations (for which we would
probably need more GUCs?). Here I'm just showing that the very first
baby step - interleaving - helps avoid interconnect saturation in some
cases too.

Anyway, even putting the fact that local mallocs() would be
interleaved, adjusting systemd startup scripts to just include
`numactl --interleave=all` sounds like some dirty hack not like proper
UX.

Also please note that:
* I do not have lot of time to dedicate towards it, yet I was kind of
always interested in researching that and wondering why we couldn't it
for such long time, therefore the previous observability work and now
$subject (note it is not claiming to be full blown NUMA awareness,
just some basic NUMA interleave as first [well, second?] step).
* I've raised this question in the first post "How to name this GUC
(numa or numa_shm_interleave) ?" I still have no idea, but `numa`,
simply looks better, and we could just add way more stuff to it over
time (in PG19 or future versions?). Does that sound good?

> Wouldn't that be a bit inefficient/costly for big instances?

No? Or what kind of mapping do you have in mind? I think our shared
memory on the kernel side is just a single VMA (contiguous memory
region), on which technically we execute mbind() (libnuma is just a
wrapper around it). I have not observed any kind of regressions,
actually quite the opposite. Not sure what you also mean by 'big
instances' (AFAIK 1-2TB shared_buffers might even fail to start).

> I hope to share it on the mailing list sometime soon.

Cool, I didn't know Andres's branch was public till now, I know he
referenced multiple issues in presentation (and hackathon!), but I
wanted to divide it and try to get something in at least partially,
step by step, to have at least something. I think we should
collaborate (not a lot of people interested in this?) and I can try to
offer my limited help if you attack those more advanced problems. I
think we could improve this by properly ensuring that by
over(allocating)/spreading/padding certain special regions (e.g.
better distribute ProcArray, but what about cache hits?) - we get more
juice, or do you want to start from scratch and re-design/re-think all
shm allocations case by case?

> containers or k8 anyway?
> I think we should just try to inherit this from

That's what "numa=all" is all about (take whatever is there in the
OS/namespace), but I do not know a better way than just let's say
numa_get_mems_allowed() being altered somehow by namespace/cgroups. I
think if one runs on k8/containers then it's quite limited/small
deployment and he wouldn't benefit from this at all (I struggle to
imagine the point of k8 pod using 2+ sockets), quite contrary: my
experience indicates that the biggest deployments are usually almost
baremetal? And it's way easier to get consistent results. Anyway as
You say, let's leave it for later. PG currently often is not CPU-aware
(i.e. is not even adjusting sizing of certain structs based on CPU
count), so making it NUMA-aware or cgroup/namespace-aware sounds
already like taking 2-3 steps ahead into future [I think we had
discussion at least one in LWLock partitionmanager /
FP_LOCK_SLOTS_PER_BACKEND where I've proposed to size certain
structures based on $VCPUs or I am misremembering this]

-J.

---

From: Jakub Wartak <[email-masked]>
Date: Mon, 30 Jun 2025 14:02:09 +0200

On Mon, Jun 30, 2025 at 12:55 PM Jakub Wartak
<[email-masked]> wrote:
[..]
> OS/namespace)

My error, that should be: that's what "numa=AUTO" is all about (..)

-J.

---

From: Tomas Vondra <[email-masked]>
Date: Mon, 30 Jun 2025 21:23:43 +0200

> `numactl --interleave=all` sounds like some dirty hack not like proper

I wasn't suggesting to do "numactl --interleave=all". My argument was
simply that doing numa_interleave_memory() has most of the same issues,
because it's oblivious to what's stored in the shared memory. Sure, the
fact that local memory is not interleaved too is an improvement.

But I just don't see how this could be 0001, followed by some later
improvements. ISTM the improvements would have to largely undo 0001
first, and it would be nontrivial if an optimization needs to do that
only for some part of the shared memory.

> just some basic NUMA interleave as first [well, second?] step).

Sorry, I appreciate the time you spent working on these features. It
wasn't my intention to dunk on your patch. I'm afraid this is an example
of how reactions on -hackers are often focused on pointing out issues. I
apologize for that, I should have realized it earlier.

I certainly agree it'd be good to improve the NUMA support, otherwise I
wouldn't be messing with Andres' PoC patches myself.

> time (in PG19 or future versions?). Does that sound good?

I'm not sure. In my WIP patch I have a bunch of numa_ GUCs, for
different parts of the shared memory. But that's mostly for development,
to allow easy experimentation. I don't have a clear idea what UX should
look like.

> instances' (AFAIK 1-2TB shared_buffers might even fail to start).

Something as simple as giving a contiguous chunk of to each NUMA node.
Essentially 1/nodes goes to the first NUMA node, and so on. I haven't
looked into the details of how NUMA interleaving works, but from the
discussions I had about it, I understood it might be expensive. Not
sure, maybe that's wrong.

But the other reason for a simpler mapping is that it seems useful to be
able to easily calculate which NUMA node a buffer belongs to. Because
then you can do NUMA-aware freelists, clocksweep, etc.

> shm allocations case by case?

+1 to collaboration, absolutely. I was actually planning to ping you
once I have something workable. I hope I'll be able to polish the WIP
patches a little bit and post them sometime this week.

> structures based on $VCPUs or I am misremembering this]

+1 to leave this for later, we can worry about this once we have it
working with the basic whole-system NUMA setups. I hope people doing
some of this would give us feedback what config they actually need.

regards

-- 
Tomas Vondra

---

From: Jakub Wartak <[email-masked]>
Date: Tue, 1 Jul 2025 11:04:46 +0200

> fact that local memory is not interleaved too is an improvement.

... and that's enough for me to start this ;)

> only for some part of the shared memory.

OK, maybe I'll back-off a bit to see Your ideas first. It seems you
are thinking about having multiple separate shared memory segments.

> wouldn't be messing with Andres' PoC patches myself.

Yup, cool, let's stick to that.

> to allow easy experimentation.
[..]
> I don't have a clear idea what UX should look like.

Later (after research/experiments), I could still imagine sticking to
one big `numa` switch like it's today in v4-0001, but maybe with
additional 1-2 more `really_advanced_numa=stuff` (but not lots of
them, I would imagine e.g. that NUMA for analytics could be different
setup that NUMA for OLTP -- AKA do we want to optimize for
interconnect bandwidth or latency?).

> Something as simple as giving a contiguous chunk of to each NUMA node.

That would actually be multiple separate VMAs/shared memory regions
(main, and specific ones for in case of NUMA - per structure) and
potentially - speculating here - slower fork()?

Related, the only complaint about memory allocated via
mmap(MAP_SHARED|MAP_HUGETLB) with NUMA, I have so far is that if the
per-zone HP free memory is too small (especially with HP=on), it
starts to spill over to the others nodes without interleaving and
without notification, you may have the same problem here unless it is
strict allocation.

> sure, maybe that's wrong.

I would really like to hear the argument how NUMA interleaving is
expensive on the kernel side. It's literally bandwidth over latency. I
can buy the argument that e.g. by having dedicated mmap(MAP_SHARED)
for ProcArray[] (assuming we are HP=on/2MB), and by having a smaller
page size just for this stuct (sizeof() =~ 832b? so let's assume even
wasting 4kB per entry), could better enable kernel's NUMA
autobalancing to better relocate those necessary pages closer to the
active processes (warning: I'm making lots of assumptions, haven't
really checked memory access patterns for this struct). No idea how
bad it would be on the CPU caches too, though by making it so big here
in this theoretical context. But the easy counter argument also could
be: smaller page size = no HP available --> potentially making it
__swap-able__ and potentially causing worse dTLB hit-rates? ... and we
are just discussing a single shared memory entry and there are 73 :)

My take is that by doing it is - as an opt-in - basic interleaving is
safe, proven and and gives *more* predictable latency that without it
(of course as You mention we could do better with some allocation for
specific structures, but how do You know where CPU scheduler puts
stuff?) I think we would need to limit ourselves to just optimizing
the most crucial (hot) stuff, like ProcArray[], but probably doesn't
make sense for investigating structures like multixacts/substractions
in this attempt.

E.g. I could even imagine that we could boost standby's NUMA-awareness
too, just by putting most used memory (eg.g. XLOG) to the same NODE
that is used by startup/recovery and walreciever (by CPU pinning), not
sure is it worth the effort though in this attempt and the problem
would be: what to do with those low-level/optimized allocations after
pg_promote() to primary? In theory this quickly escalates to calling
interleave on that struct again, so maybe let's put it aside.

To sum up, my problem is that optimization possibilities are quite
endless, so we need to settle on something realistic, right?

> then you can do NUMA-aware freelists, clocksweep, etc.

Yay, sounds pretty advanced!

> patches a little bit and post them sometime this week.

Cool.

-J.

---

From: Tomas Vondra <[email-masked]>
Date: Tue, 1 Jul 2025 16:02:39 +0200

> interconnect bandwidth or latency?).

Maybe. I have no clear idea yet, but I'd like to keep the number of new
GUCs as low as possible.

> potentially - speculating here - slower fork()?

I may be confused about what you mean by VMA, but it certainly does not
require creating separate shared memory segments. Interleaving also does
not require that. You can move a certain range of memory to a particular
NUMA node, and that's it.

We may end up with separate shared memory segments for different parts
of the shared memory (instead of having a single segment like now), e.g.
to support dynamic shared_buffers resizing. But even with that we'd have
a shared memory segment for each part, split between NUMA nodes.

Well, we'd probably want separate segments, because for some parts it's
not great to have 2MB pages, because it's too coarse. And you can only
use huge pages for the whole segment.

> are just discussing a single shared memory entry and there are 73 :)

I admit I don't recall the exact details of why exactly interleaving
would be expensive on the kernel side. I've been told by smarter people
that might be the case, but I don't remember the exact explanation. And
maybe it isn't measurably more expensive ...

I've been focusing on the aspect that it makes certain things more
difficult, or even impossible ...

> make sense for investigating structures like multixacts/substractions

My argument is that if we allocate the structs "well" then we can do
something smart later, like pick a PGPROC placed on the current NUMA
node (at connection time), and possibly even pin it to that NUMA node so
that it doesn't get migrated.

This means not just the PROPROC itself, but also stuff like fast-path
locking arrays (which are stored separately). And interleaving could
easily place them on a different NUMA node.

> endless, so we need to settle on something realistic, right?

Perhaps. But maybe we should explore the possibilities first, before
just settling out on something at the very beginning. To make an
informed decision we need to know what the costs/benefits are, and we
need to understand what the "advanced" solution might look like, so that
we don't pick a design that makes that impossible.

regards

-- 
Tomas Vondra

---

From: Tomas Vondra <[email-masked]>
Date: Tue, 1 Jul 2025 21:09:16 +0200

Hi Jakub,

FYI I've posted my experimental NUMA patch series here:

https://www.postgresql.org/message-id/099b9433-2855-4f1b-b421-d078a5d82017%40vondra.me

I've considered posting it to this thread, but it seemed sufficiently
different to start a new thread.

regards

-- 
Tomas Vondra