NUMA (Non-Uniform Memory Access)

Not All Memory Is Created Equal

On a laptop or small server with a single CPU socket, all RAM is approximately the same distance from the processor. Access time to any byte in RAM is roughly uniform. This is called UMA (Uniform Memory Access).

But on large servers—the kind that run databases, Kubernetes control planes, and big data workloads—there are often two or more physical CPU sockets, each with its own set of RAM DIMMs wired directly to it. This architecture is called NUMA (Non-Uniform Memory Access).

How it works:
- A server with 2 CPU sockets and 128 GB of RAM typically has 64 GB physically wired to Socket 0 and 64 GB wired to Socket 1.
- For a thread running on Socket 0, accessing its local RAM takes ~80 ns.
- For the same thread to access Socket 1's remote RAM, the request must travel over an interconnect (Intel's QPI/UPI or AMD's Infinity Fabric), adding latency. Remote access takes ~130–180 ns—roughly 1.5–2x slower.
- This penalty applies to every single remote memory access, and a workload that makes billions of memory accesses per second will feel it acutely.

Why this matters for platform engineers:

1. VM Placement: If a VM is allocated 32 GB of RAM but the hypervisor spreads that memory across both NUMA nodes, every other memory access pays the remote penalty. This is why numactl --hardware and NUMA-aware VM placement matter.

2. Database Tuning: Databases like PostgreSQL and MySQL are highly sensitive to NUMA topology. A common anti-pattern is running a database on a NUMA system with the default "interleave" memory policy, which spreads allocations evenly across nodes. For single-instance databases, binding the process to a single NUMA node often yields 10–30% better performance.

3. Kubernetes: The kubelet has a NUMA-aware topology manager (enabled via --topology-manager-policy=single-numa-node) that ensures a pod's CPU and memory allocations come from the same NUMA node. This is critical for latency-sensitive workloads like real-time data processing or low-latency APIs.

4. Cloud Instance Types: Cloud providers map their instance types to physical NUMA topology. An instance type that spans two NUMA nodes (common with very large instances) may exhibit higher memory latency variance than a smaller instance that fits within a single node.

Checking NUMA Topology

The numactl package is the standard toolkit for inspecting and controlling NUMA behavior on Linux. Its --hardware report shows the topology: how many nodes exist, which CPUs and how much RAM belong to each, and a node distances table—typically a normalized 10 for local access and roughly 21 for remote, a penalty of about 2x.

numastat complements this with allocation statistics, either system-wide or scoped to a single process. Its most revealing counter is numa_miss: how many times a memory allocation intended for the local node had to be satisfied by a remote node instead. High numa_miss values are the signature of a NUMA-unaware workload paying the remote access penalty. Once diagnosed, a workload can be pinned so both its CPU threads and its memory allocations stay on one node—eliminating remote access entirely—or, for workloads that genuinely touch all memory evenly, deliberately spread across nodes with an interleaved memory policy.

Key Point: NUMA is not an exotic edge case. Any server with 2+ CPU sockets is a NUMA system, and that includes most production database servers, large Kubernetes nodes, and bare-metal hosts. Ignoring NUMA topology is one of the most common causes of unexplained performance variability in production systems. A workload that benchmarks beautifully on a single-socket dev machine can perform 30% worse in production purely due to NUMA effects—and the symptom often looks like random latency spikes rather than a consistent slowdown, making it especially hard to diagnose without knowing to look for it.