Prompt and context
A memory-intensive service has a higher p99 after an upgrade while CPU utilization is normal. The host has multiple NUMA nodes and a thread may run on a different node from its pages. Explain NUMA and give a diagnosis from topology, affinity, allocation, and controlled benchmarks. The question tests operating systems, performance analysis, and engineering reasoning.
What the interviewer is testing
Understand locality
NUMA exposes multiple CPU and memory nodes in one addressable system. Local memory is generally faster and offers more nearby bandwidth, so performance depends on keeping most cache-miss accesses local.
Separate symptoms from evidence
Normal CPU utilization does not rule out remote-memory stalls. Use topology, process placement, numastat, hardware counters, and repeatable baselines instead of one aggregate metric.
Propose safe experiments
Compare fixed CPU placement, fixed memory placement, interleaving, and the default policy with identical load. Use p50/p99, bandwidth, and miss counters to connect a change to a cause.
Questions to clarify first
- How many NUMA nodes, CPUs, memory banks, and devices exist on the host?
- Is the regression observed in a single thread, thread pool, container, or VM?
- Does the service use huge pages, shared memory, memory mapping, or GPU/NIC DMA?
- What are the process/thread affinity, cpuset, and memory policy settings?
- Is automatic NUMA balancing enabled, and did the kernel or runtime change?
- What baseline separates memory latency, bandwidth saturation, and lock contention?
A 30-second answer
“NUMA groups CPUs, memory, and interconnects into nodes; local memory is usually faster, while remote access adds latency and consumes interconnect bandwidth. I would inspect numactl --hardware, affinity, numastat -p, and /proc/<pid>/numa_maps, then compare default, local binding, and interleaving on the same workload. I would correlate p99 with numa_miss, bandwidth, remote-access counters, and migrations. The fix might align threads and data, shard the workload, tune page placement, or roll back automatic balancing, with a reversible rollout.”
Step-by-step deep answer
Map the hardware and software topology
Record nodes, CPUs, memory, PCIe devices, and distance information. The Linux kernel describes NUMA cells connected by an interconnect: every CPU can address global memory, but distance changes latency and bandwidth.
Check process and thread placement
Inspect CPU affinity, cpusets, container limits, and thread migration. A thread running on node 0 while repeatedly reading node 1 pages has lost locality; thread-pool scaling can also change first-touch placement.
Check page distribution and hit statistics
numastat reports numa_hit for allocations satisfied on the preferred node, numa_miss for allocations that could not use that preference, and local_node/other_node from the executing CPU's locality. Inspect per-process data and /proc/<pid>/numa_maps instead of treating system totals as service evidence.
Run repeatable A/B policies
Compare the default policy, --cpunodebind and --membind fixed policies, and --interleave while keeping input, thread count, and load constant. Improvement with local binding supports a locality hypothesis; improvement with interleaving can indicate a single-node bandwidth hotspot.
Evaluate automatic NUMA balancing
Automatic balancing scans access patterns and may migrate pages. It can improve long-lived locality, but scanning and migration can add jitter for short requests or high churn. Measure migrations, faults, scans, and request tails under a controlled toggle.
Check shared data and locks
Shared queues, allocator metadata, and cross-node locks can create both remote access and cache-line contention. CPU binding alone may not help, so correlate lock wait, bandwidth, and cache-miss counters to rule out false sharing or lock contention.
Diagnostic pseudocode
~~~text record topology, affinity, numa_maps, numastat, p99 run baseline with fixed workload for policy in [default, local_bind, interleave]: run same workload and collect latency, bandwidth, misses, migrations compare deltas and check confidence intervals apply the least invasive policy; keep rollback switch ~~~
Complexity, risk, and verification
Binding is not a complexity problem; the risks are reduced scheduling flexibility, node-local memory exhaustion, and cross-node device DMA. Test cold start, steady state, scaling, container migration, and node failures. Keep p50/p99, throughput, bandwidth, misses, migrations, and OOM signals for every change.
| Evidence | Meaning | Likely next step |
|---|---|---|
Rising numa_miss / other_node | Placement and preference diverge | Check affinity and memory policy |
| Rising remote accesses near bandwidth limit | Interconnect is a bottleneck | Shard data or change placement |
| Rising migrations and faults | Balancing or first-touch behavior changed | Tune warm-up, policy, or layout |
| Local binding improves p99 | Locality has causal evidence | Canary the binding and observe |
Model answer
“NUMA is shared addressability with non-uniform distance: CPUs, memory, and devices are grouped into nodes, and local access is generally faster. I would first record topology, process/thread affinity, container cpusets, numastat -p, and /proc/<pid>/numa_maps. Then I would run the same workload under default, CPU/memory-local, and interleaved policies, collecting p99, bandwidth, remote accesses, numa_miss, migrations, and faults. If local binding improves the tail, align the pool and data shards by node; if one node saturates, consider interleaving or sharding. Use experiments to separate automatic balancing, shared queues, and lock contention, then ship a reversible policy with rollback thresholds.”
Common mistakes
Treating NUMA as low total memory
NUMA is about distance and bandwidth, not just total capacity. An imbalanced node can hit local OOM while the host still has free memory elsewhere.
Looking only at CPU utilization
Memory latency, interconnect bandwidth, and stalls do not map cleanly to one CPU percentage. Collect tail latency, bandwidth, and memory counters.
Binding CPUs without memory
First-touch and allocation policy can place pages elsewhere after a thread moves. Validate CPU affinity and memory policy together.
Disabling balancing on one miss counter
Misses can be expected for shared data or pressure. Disabling balancing may worsen long-term locality; run a controlled A/B and measure migration cost.
Ignoring containers and VMs
Host topology, virtual NUMA, cpusets, and device placement may differ from the process view. Verify the topology at the deployment layer.
Concluding from one short benchmark
Warm-up, migration, caches, and load shape affect NUMA. Cover steady state, peaks, and reclamation, with repeated trials.
Follow-up questions and responses
Why can NUMA improve scalability?
Each node contributes local memory bandwidth, allowing aggregate bandwidth to scale across nodes. The trade-off is that software must preserve locality.
What is the difference between numa_hit and local_node?
numa_hit is based on the process's preferred node; local_node is based on the CPU's local node. A memory policy can make their signals differ.
When would you use interleaving?
Use it when the working set is broadly shared, one node's bandwidth is insufficient, or pairing threads with pages is impractical. It may not reduce single-access latency.
How do you handle first touch?
Warm pages from the threads that will use them, or apply an explicit memory policy. Otherwise the initializer can determine placement.
Is page migration always good?
Migration can improve locality but consumes bandwidth and causes pauses. Compare migration cost, access benefit, and tail latency rather than maximizing migrations.
How do you ship the diagnosis?
Canary a reversible startup or policy switch. Set thresholds for p99, node headroom, remote access, migrations, and OOM, and expand only after the evidence remains healthy.