public inbox for [email protected]
help / color / mirror / Atom feedFrom: Tomas Vondra <[email protected]>
To: Andres Freund <[email protected]>
Cc: Jakub Wartak <[email protected]>
Cc: Alexey Makhmutov <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Subject: Re: Adding basic NUMA awareness
Date: Thu, 15 Jan 2026 00:26:47 +0100
Message-ID: <[email protected]> (raw)
In-Reply-To: <clx4zzd7kau4vvh5ynu5ssxg3jqfqzurgcbtotytzgzkhb3nis@qfl5xwv44yad>
References: <[email protected]>
<CAKZiRmwPVxi1H23pNZ4_Vc=mtMaNgY1z79s6SwjuUZD3EaOPeA@mail.gmail.com>
<[email protected]>
<CAKZiRmxwN+qMpbijCLPix_y6mwSjgus2CPPj=1+uFo9fQG-Knw@mail.gmail.com>
<[email protected]>
<uezi46xhhbvdjgdi6wl7iqgfcdh4jmnnyzbfovdcrck6ywqa7j@fj3yimxvekk6>
<[email protected]>
<zndz3dlmp7xlypczxjbwdfrey3masto6vuwpnzjvgunslprv25@rucfkdgfpb4p>
<rsjxzhnxxfq5i5yxv66mhinb42o3vzmqpkbfpexpkk5prreh2l@jyp73gsyyfzn>
<[email protected]>
<clx4zzd7kau4vvh5ynu5ssxg3jqfqzurgcbtotytzgzkhb3nis@qfl5xwv44yad>
On 1/13/26 15:14, Andres Freund wrote:
> Hi,
>
> On 2026-01-13 02:13:40 +0100, Tomas Vondra wrote:
>> On the azure VM (scale 200, 32GB sb), there's still no difference:
>
> One possibility is that the host is configured with memory interleaving. That
> configures the memory so that physical memory addresses interleave between the
> different NUMA nodes, instead of really being node local. That can help avoid
> bad performance characteristics for NUMA naive applications.
>
> I don't quite know how to figure that out though, particularly from within a
> VM :(. Even something like https://github.com/nviennot/core-to-core-latency
> or intel's mlc will not necessarily be helpful, because it depends on which
> node the measured cacheline ends up on.
>
> But I'd probably still test it, just to see whether you're observing very
> different latencies between the systems.
>
I did this on the two Azure instances I've been using for testing (D96
and HB176), and I got this:
D96 (v6):
Numa node
Numa node 0 1
0 129.9 129.9
1 128.3 128.1
HB176 (v4):
Numa node
Numa node 0 1 2 3
0 107.3 116.8 207.3 207.0
1 120.5 110.6 207.5 207.1
2 207.0 207.2 107.8 116.8
3 204.4 204.7 117.7 107.9
I guess this confirms that D96 is mostly useless for evaluation of the
NUMA patches. This is a single-socket machine, with one NUMA node per
chiplet (I assume), and there's about no difference in latency.
For HB176 there clearly seems to be a difference of ~90ns between the
sockets, i.e. the latency about doubles in some cases. Each socket has
two chiplets - and there the story is about the same as on D96.
I did this on my old-ish Xeon too, and it's somewhere in between. There
clearly is difference between the sockets, but it's smaller than on
HB176. Which matches with your observation that the latency is really
increasing over time.
I doubt the interleaving mode is enabled. It clearly is not enabled on
the HB176 machine (otherwise we wouldn't see the difference, I think),
and the smaller instance can be explained by having a single socket.
I've attached the complete mlc results, for completeness.
I've also done bigger SQL test with pinning the memory/backends to
different nodes, for a range of scales and the two queries (agg and
offset). I'm attaching results for scale 100 and 10000 from D96 and
HB176 instances.
The numbers are timings per query (avg latency reported by pgbench). I
think this mostly aligns with the mlc results - the D96 shows no
difference, while HB176 shows clear differences when memory/cpu get
pinned to different sockets (but not chiplets in the same socket).
But there are some interesting details too, particularly when it comes
to behavior of the two queries. The "offset" query is affected by
latency even with no parallelism (max_parallel_workers_per_gather=0),
and it shows ~30% hit for cross-socket runs. But for "agg" there's no
difference in that case, and the hit is visible only with 4 or 8
workers. That's interesting.
Anyway, my plan at this point is to revive the old patch (before
changing direction to the simple patch), and see if we can observe a
difference on the "right" hardware. Maybe some of the results with no
improvements were due to this. This workload seems much more realistic.
regards
--
Tomas Vondra
Intel(R) Memory Latency Checker - v3.12
Measuring idle latencies for sequential access (in ns)...
Numa node
Numa node 0 1
0 81.9 129.0
1 122.5 80.8
Measuring Peak Injection Memory Bandwidths for the system
Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec)
Using all the threads from each core if Hyper-threading is enabled
Using traffic with the following read-write ratios
ALL Reads : 126912.2
3:1 Reads-Writes : 66042.0
2:1 Reads-Writes : 102700.1
1:1 Reads-Writes : 88529.7
Stream-triad like: 93077.4
All NT writes : 78795.6
1:1 Read-NT write: 82886.5
Measuring Memory Bandwidths between nodes within system
Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec)
Using all the threads from each core if Hyper-threading is enabled
Using Read-only traffic type
Numa node
Numa node 0 1
0 63509.3 30463.7
1 30472.2 63792.0
Measuring Loaded Latencies for the system
Using all the threads from each core if Hyper-threading is enabled
Using Read-only traffic type
Inject Latency Bandwidth
Delay (ns) MB/sec
==========================
00000 292.72 126299.0
00002 292.37 126289.0
00008 288.53 126332.1
00015 285.61 126249.0
00050 269.54 125687.3
00100 257.42 124665.4
00200 198.50 119536.1
00300 135.80 95900.8
00400 111.41 74369.1
00500 100.35 60355.2
00700 94.54 43772.0
01000 88.26 31124.7
01300 85.85 24206.5
01700 85.15 18732.8
02500 83.64 13021.2
03500 83.44 9540.3
05000 82.52 6917.7
09000 81.10 4204.8
20000 80.08 2337.2
Measuring cache-to-cache transfer latency (in ns)...
Local Socket L2->L2 HIT latency 37.9
Local Socket L2->L2 HITM latency 41.5
Remote Socket L2->L2 HITM latency (data address homed in writer socket)
Reader Numa Node
Writer Numa Node 0 1
0 - 100.5
1 100.2 -
Remote Socket L2->L2 HITM latency (data address homed in reader socket)
Reader Numa Node
Writer Numa Node 0 1
0 - 102.1
1 101.6 -
Intel(R) Memory Latency Checker - v3.12
*** Unable to modify prefetchers (try executing 'modprobe msr')
*** So, enabling random access for latency measurements
Measuring idle latencies for random access (in ns)...
Numa node
Numa node 0 1 2 3
0 107.3 116.8 207.3 207.0
1 120.5 110.6 207.5 207.1
2 207.0 207.2 107.8 116.8
3 204.4 204.7 117.7 107.9
Measuring Peak Injection Memory Bandwidths for the system
Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec)
Using all the threads from each core if Hyper-threading is enabled
Using traffic with the following read-write ratios
ALL Reads : 637545.8
3:1 Reads-Writes : 577715.7
2:1 Reads-Writes : 486147.5
1:1 Reads-Writes : 429828.1
Stream-triad like: 588366.2
All NT writes : 505637.7
1:1 Read-NT write: 450882.6
Measuring Memory Bandwidths between nodes within system
Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec)
Using all the threads from each core if Hyper-threading is enabled
Using Read-only traffic type
Numa node
Numa node 0 1 2 3
0 211504.2 203084.9 91915.5 92326.9
1 202149.1 260100.9 91868.6 92133.3
2 91912.9 112167.5 211826.4 202075.9
3 92147.2 112663.6 202427.9 211755.0
Measuring Loaded Latencies for the system
Using all the threads from each core if Hyper-threading is enabled
Using Read-only traffic type
Inject Latency Bandwidth
Delay (ns) MB/sec
==========================
00000 632.81 846408.6
00002 631.00 846629.6
00008 632.98 845822.7
00015 640.31 845685.8
00050 662.04 839876.2
00100 659.75 821062.7
00200 160.90 791997.3
00300 123.19 539324.2
00400 120.81 389993.4
00500 118.85 315980.0
00700 117.53 228791.5
01000 116.66 161892.1
01300 115.78 125314.0
01700 110.37 96388.6
02500 108.80 66030.3
03500 108.52 47464.2
05000 108.29 33474.1
09000 107.96 18904.4
20000 107.76 8844.2
Measuring cache-to-cache transfer latency (in ns)...
Local Socket L2->L2 HIT latency 27.5
Local Socket L2->L2 HITM latency 27.6
Remote Socket L2->L2 HITM latency (data address homed in writer socket)
Reader Numa Node
Writer Numa Node 0 1 2 3
0 - 125.8 213.3 211.7
1 125.7 - 213.6 212.2
2 214.7 212.3 - 124.6
3 214.3 212.4 124.7 -
Remote Socket L2->L2 HITM latency (data address homed in reader socket)
Reader Numa Node
Writer Numa Node 0 1 2 3
0 - 125.8 217.8 216.8
1 125.8 - 214.5 213.1
2 219.1 217.2 - 124.7
3 216.8 215.0 124.7 -
Intel(R) Memory Latency Checker - v3.12
*** Unable to modify prefetchers (try executing 'modprobe msr')
*** So, enabling random access for latency measurements
Measuring idle latencies for random access (in ns)...
Numa node
Numa node 0 1
0 129.9 129.9
1 128.3 128.1
Measuring Peak Injection Memory Bandwidths for the system
Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec)
Using all the threads from each core if Hyper-threading is enabled
Using traffic with the following read-write ratios
ALL Reads : 233023.4
3:1 Reads-Writes : 267684.9
2:1 Reads-Writes : 275930.3
1:1 Reads-Writes : 292524.8
Stream-triad like: 273671.9
All NT writes : 170797.1
1:1 Read-NT write: 292916.6
Measuring Memory Bandwidths between nodes within system
Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec)
Using all the threads from each core if Hyper-threading is enabled
Using Read-only traffic type
Numa node
Numa node 0 1
0 123198.1 117550.5
1 119335.1 116650.4
Measuring Loaded Latencies for the system
Using all the threads from each core if Hyper-threading is enabled
Using Read-only traffic type
Inject Latency Bandwidth
Delay (ns) MB/sec
==========================
00000 721.29 232845.2
00002 664.14 232818.5
00008 664.32 232741.6
00015 662.87 232813.0
00050 658.94 232603.5
00100 654.70 232538.6
00200 175.42 211784.1
00300 158.98 143574.0
00400 154.97 105691.8
00500 152.84 85336.8
00700 151.05 61592.4
01000 149.91 43717.3
01300 150.28 33896.4
01700 142.11 26119.2
02500 142.59 17949.0
03500 152.47 12940.1
05000 140.65 9230.0
09000 141.48 5332.5
20000 140.18 2654.3
Measuring cache-to-cache transfer latency (in ns)...
Local Socket L2->L2 HIT latency 25.3
Local Socket L2->L2 HITM latency 25.3
Attachments:
[text/plain] numa-xeon.txt (2.0K, ../[email protected]/2-numa-xeon.txt)
download | inline:
Intel(R) Memory Latency Checker - v3.12
Measuring idle latencies for sequential access (in ns)...
Numa node
Numa node 0 1
0 81.9 129.0
1 122.5 80.8
Measuring Peak Injection Memory Bandwidths for the system
Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec)
Using all the threads from each core if Hyper-threading is enabled
Using traffic with the following read-write ratios
ALL Reads : 126912.2
3:1 Reads-Writes : 66042.0
2:1 Reads-Writes : 102700.1
1:1 Reads-Writes : 88529.7
Stream-triad like: 93077.4
All NT writes : 78795.6
1:1 Read-NT write: 82886.5
Measuring Memory Bandwidths between nodes within system
Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec)
Using all the threads from each core if Hyper-threading is enabled
Using Read-only traffic type
Numa node
Numa node 0 1
0 63509.3 30463.7
1 30472.2 63792.0
Measuring Loaded Latencies for the system
Using all the threads from each core if Hyper-threading is enabled
Using Read-only traffic type
Inject Latency Bandwidth
Delay (ns) MB/sec
==========================
00000 292.72 126299.0
00002 292.37 126289.0
00008 288.53 126332.1
00015 285.61 126249.0
00050 269.54 125687.3
00100 257.42 124665.4
00200 198.50 119536.1
00300 135.80 95900.8
00400 111.41 74369.1
00500 100.35 60355.2
00700 94.54 43772.0
01000 88.26 31124.7
01300 85.85 24206.5
01700 85.15 18732.8
02500 83.64 13021.2
03500 83.44 9540.3
05000 82.52 6917.7
09000 81.10 4204.8
20000 80.08 2337.2
Measuring cache-to-cache transfer latency (in ns)...
Local Socket L2->L2 HIT latency 37.9
Local Socket L2->L2 HITM latency 41.5
Remote Socket L2->L2 HITM latency (data address homed in writer socket)
Reader Numa Node
Writer Numa Node 0 1
0 - 100.5
1 100.2 -
Remote Socket L2->L2 HITM latency (data address homed in reader socket)
Reader Numa Node
Writer Numa Node 0 1
0 - 102.1
1 101.6 -
[text/plain] numa-hb176-epyc-9v33x.txt (2.6K, ../[email protected]/3-numa-hb176-epyc-9v33x.txt)
download | inline:
Intel(R) Memory Latency Checker - v3.12
*** Unable to modify prefetchers (try executing 'modprobe msr')
*** So, enabling random access for latency measurements
Measuring idle latencies for random access (in ns)...
Numa node
Numa node 0 1 2 3
0 107.3 116.8 207.3 207.0
1 120.5 110.6 207.5 207.1
2 207.0 207.2 107.8 116.8
3 204.4 204.7 117.7 107.9
Measuring Peak Injection Memory Bandwidths for the system
Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec)
Using all the threads from each core if Hyper-threading is enabled
Using traffic with the following read-write ratios
ALL Reads : 637545.8
3:1 Reads-Writes : 577715.7
2:1 Reads-Writes : 486147.5
1:1 Reads-Writes : 429828.1
Stream-triad like: 588366.2
All NT writes : 505637.7
1:1 Read-NT write: 450882.6
Measuring Memory Bandwidths between nodes within system
Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec)
Using all the threads from each core if Hyper-threading is enabled
Using Read-only traffic type
Numa node
Numa node 0 1 2 3
0 211504.2 203084.9 91915.5 92326.9
1 202149.1 260100.9 91868.6 92133.3
2 91912.9 112167.5 211826.4 202075.9
3 92147.2 112663.6 202427.9 211755.0
Measuring Loaded Latencies for the system
Using all the threads from each core if Hyper-threading is enabled
Using Read-only traffic type
Inject Latency Bandwidth
Delay (ns) MB/sec
==========================
00000 632.81 846408.6
00002 631.00 846629.6
00008 632.98 845822.7
00015 640.31 845685.8
00050 662.04 839876.2
00100 659.75 821062.7
00200 160.90 791997.3
00300 123.19 539324.2
00400 120.81 389993.4
00500 118.85 315980.0
00700 117.53 228791.5
01000 116.66 161892.1
01300 115.78 125314.0
01700 110.37 96388.6
02500 108.80 66030.3
03500 108.52 47464.2
05000 108.29 33474.1
09000 107.96 18904.4
20000 107.76 8844.2
Measuring cache-to-cache transfer latency (in ns)...
Local Socket L2->L2 HIT latency 27.5
Local Socket L2->L2 HITM latency 27.6
Remote Socket L2->L2 HITM latency (data address homed in writer socket)
Reader Numa Node
Writer Numa Node 0 1 2 3
0 - 125.8 213.3 211.7
1 125.7 - 213.6 212.2
2 214.7 212.3 - 124.6
3 214.3 212.4 124.7 -
Remote Socket L2->L2 HITM latency (data address homed in reader socket)
Reader Numa Node
Writer Numa Node 0 1 2 3
0 - 125.8 217.8 216.8
1 125.8 - 214.5 213.1
2 219.1 217.2 - 124.7
3 216.8 215.0 124.7 -
[text/plain] numa-d96-epyc-9v74.txt (1.8K, ../[email protected]/4-numa-d96-epyc-9v74.txt)
download | inline:
Intel(R) Memory Latency Checker - v3.12
*** Unable to modify prefetchers (try executing 'modprobe msr')
*** So, enabling random access for latency measurements
Measuring idle latencies for random access (in ns)...
Numa node
Numa node 0 1
0 129.9 129.9
1 128.3 128.1
Measuring Peak Injection Memory Bandwidths for the system
Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec)
Using all the threads from each core if Hyper-threading is enabled
Using traffic with the following read-write ratios
ALL Reads : 233023.4
3:1 Reads-Writes : 267684.9
2:1 Reads-Writes : 275930.3
1:1 Reads-Writes : 292524.8
Stream-triad like: 273671.9
All NT writes : 170797.1
1:1 Read-NT write: 292916.6
Measuring Memory Bandwidths between nodes within system
Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec)
Using all the threads from each core if Hyper-threading is enabled
Using Read-only traffic type
Numa node
Numa node 0 1
0 123198.1 117550.5
1 119335.1 116650.4
Measuring Loaded Latencies for the system
Using all the threads from each core if Hyper-threading is enabled
Using Read-only traffic type
Inject Latency Bandwidth
Delay (ns) MB/sec
==========================
00000 721.29 232845.2
00002 664.14 232818.5
00008 664.32 232741.6
00015 662.87 232813.0
00050 658.94 232603.5
00100 654.70 232538.6
00200 175.42 211784.1
00300 158.98 143574.0
00400 154.97 105691.8
00500 152.84 85336.8
00700 151.05 61592.4
01000 149.91 43717.3
01300 150.28 33896.4
01700 142.11 26119.2
02500 142.59 17949.0
03500 152.47 12940.1
05000 140.65 9230.0
09000 141.48 5332.5
20000 140.18 2654.3
Measuring cache-to-cache transfer latency (in ns)...
Local Socket L2->L2 HIT latency 25.3
Local Socket L2->L2 HITM latency 25.3
[application/pdf] d96-100.pdf (36.1K, ../[email protected]/5-d96-100.pdf)
download
[application/pdf] d96-10000.pdf (36.1K, ../[email protected]/6-d96-10000.pdf)
download
[application/pdf] hb176-10000.pdf (38.8K, ../[email protected]/7-hb176-10000.pdf)
download
[application/pdf] hb176-100.pdf (38.5K, ../[email protected]/8-hb176-100.pdf)
download
view thread (23+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: Adding basic NUMA awareness
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox