The 8-GPU NUMA Trap: CPU-GPU Affinity and Multi-Node Inference

When you bolt eight GPUs into a single server, you expect linear scaling. The reality is messier: the box is not one machine but a patchwork of NUMA domains, PCIe switches, and memory controllers that silently decide where data travels. The 8-GPU NUMA trap is the name for the performance cliff that appears when your inference workload's threads and tensors land on the wrong side of those boundaries. It is not a hardware failure; it is a topology mismatch. And it is the difference between a cluster that hums and one that hiccups under load.

This article walks through how NUMA and PCIe topology shape multi-GPU inference, why CPU-GPU affinity matters more than raw core count, and how to diagnose and fix the problem in practice—using tools you already have. No magic numbers, just the mechanics.

The Anatomy of a NUMA Trap

A modern dual-socket server is two computers glued together by a coherence fabric. Each CPU socket is a NUMA node: it owns a slice of DRAM and a set of PCIe lanes. When a process running on socket 0 allocates memory, it gets pages from socket 0's DRAM—fast. When it touches memory on socket 1, it pays a penalty: the request hops over the inter-socket link (e.g., UPI or Infinity Fabric) and back. The latency is higher, and bandwidth is shared.

GPUs are attached to PCIe roots, and those roots are distributed across sockets. On a typical 8-GPU board, four GPUs hang off socket 0's PCIe root, four off socket 1. If your inference process runs on socket 0 but the GPU you're using is on socket 1, every tensor copy traverses the inter-socket link. For a model that streams activations and KV-cache entries, that overhead compounds.

The trap is subtle because the OS scheduler does not care about PCIe topology. It sees 64 logical CPUs and balances threads across them. Your process might run on socket 0, allocate memory on socket 0, then issue a CUDA call to a GPU on socket 1. The data path is: DRAM (socket 0) → CPU cache (socket 0) → inter-socket link → PCIe root (socket 1) → GPU. That is three extra hops compared to the ideal path.

Why Memory Bandwidth Matters for Inference

Inference is not compute-bound in the traditional sense. A transformer forward pass spends a large fraction of time moving weights and KV-cache entries from HBM to the GPU's compute units. The GPU's memory bandwidth is the bottleneck, not the FLOPs. But that is the GPU's internal bandwidth. The problem is the data that must cross the PCIe bus: input tokens, intermediate activations for tensor parallelism, and the output logits.

When you run multi-GPU inference with tensor parallelism, the model is sharded across GPUs. Each layer's forward pass requires an all-reduce to exchange partial results. Those all-reduce messages travel over PCIe. If the GPUs are on different sockets, the messages also travel over the inter-socket link. The link's bandwidth is finite—typically around 100 GB/s for modern CPUs—and it is shared with DRAM traffic. Under load, you get contention: the all-reduce stalls while waiting for the link to free up.

Teams that benchmark multi-GPU inference often see latency spikes that correlate with NUMA placement. A common pattern is that a model runs fine on a single GPU, but when scaled to eight, the per-token latency grows faster than expected. The usual suspect is the communication overhead, but the root cause is often that the CPU threads handling the data movement are pinned to the wrong socket.

Diagnosing the Trap

Before you change anything, you need to see the topology. The lstopo tool (from the hwloc package) gives a visual map of your sockets, cores, memory, and PCIe devices. Run it and look for the GPU PCIe addresses. Each GPU has a BDF (bus:device.function) like 0000:3b:00.0. The first two digits after the colon indicate the PCIe bus, which correlates with the socket.

On a dual-socket system, the PCIe buses are often split: sockets 0 and 1 each own a range. For example, bus 3b might be on socket 0, and bus 86 on socket 1. You can also check with lspci -tv to see the tree.

To find which NUMA node a GPU is on, use nvidia-smi topo -m. That command prints a matrix of GPU-to-GPU affinity, showing NV# for NVLink connections and PIX for PCIe switches. It also shows the NUMA node each GPU belongs to in the NUMANode column. If the output shows GPUs spread across nodes 0 and 1, you have a heterogeneous placement.

Another diagnostic is to run a simple memory copy test. Use numactl --cpunodebind=0 --membind=0 to run a bandwidth benchmark (like mbw or stream) and compare it to a run with --cpunodebind=1. The difference tells you the inter-socket penalty on your system. But that only measures DRAM, not GPU traffic. For GPU traffic, you can use nvidia-smi pmon to watch PCIe transfers during inference and see if any GPU's tx and rx counters spike when you expect them to be idle.

Fixing Affinity: The Practical Steps

Once you know the topology, the fix is to pin your inference process to the CPU cores and memory node that match the GPU you're using. The tool is numactl. For example, if GPU 0 is on NUMA node 0, run:

numactl --cpunodebind=0 --membind=0 python serve.py

That binds the process to socket 0's CPUs and memory. But if your workload uses multiple GPUs (e.g., tensor parallelism across 4 GPUs on node 0), you need to bind to all the cores on that node. Use --physcpubind to specify a core range:

numactl --physcpubind=0-15 --membind=0 python serve.py

That ensures the process's threads run on the cores that have the shortest path to the GPUs.

For PyTorch, you can also set the device affinity programmatically. The torch.cuda module lets you check the current device and set it, but the CPU affinity is outside PyTorch's control. You need to combine numactl with taskset if you want finer control over which specific cores.

When using vLLM, the server spawns multiple worker processes for tensor parallelism. Each worker should be pinned to the appropriate NUMA node. vLLM's --worker-cls and --worker-module options allow custom worker classes, but the simplest approach is to launch the entire server under numactl and let the OS scheduler keep threads local—though that is not guaranteed. For deterministic placement, you can set CUDA_VISIBLE_DEVICES to a GPU list that are all on the same socket, and then bind accordingly.

The Role of PCIe Switches and Peer-to-Peer

Modern GPUs often connect through PCIe switches to increase the number of devices per root. The switch adds latency, but it also enables peer-to-peer (P2P) transfers between GPUs without going through the CPU. NVLink is the high-bandwidth option, but not all GPUs have it. On systems without NVLink, P2P over PCIe is the only way to avoid the CPU as a relay.

To enable P2P, you need to ensure the CUDA context allows it. In PyTorch, you can check with torch.cuda.can_device_access_peer(device1, device2). If it returns False, you may need to enable it via torch.cuda.set_device and then call torch.cuda.synchronize before enabling peer access. But the catch is that P2P only works if the GPUs are on the same PCIe root or through a switch that supports it. If your GPUs are on different sockets, P2P falls back to going through the CPU, which is worse than using the inter-socket link directly.

A common pattern is to use NVLink for intra-socket pairs and PCIe for inter-socket pairs. The topology matrix from nvidia-smi topo -m shows which pairs have NVLink. When designing your tensor parallel group, try to place the shards on GPUs that are NVLink-connected. That reduces the communication overhead significantly, but it also means the CPU threads feeding those GPUs should be on the same NUMA node as the GPUs.

Real-World Configuration: A Case Study

Consider a typical 8-GPU server: two sockets, each with four GPUs. The GPUs on socket 0 are connected via NVLink in a ring, and the same for socket 1. The two sockets are connected via UPI. When you run a model with tensor parallelism across all eight GPUs, the all-reduce operations will involve both intra-socket NVLink transfers and inter-socket UPI transfers.

If you naively start the server with python serve.py, the OS may place the main thread on socket 0, and the worker threads on various cores. The memory allocation for the model weights will be on the node where the allocation happens, which might be node 0. But the GPU on socket 1 will need to access those weights via the inter-socket link, causing a slowdown.

A better approach is to split the model into two groups: one for each NUMA node. Run two vLLM instances, each bound to a different socket and using the GPUs on that socket. Then put a load balancer in front. This avoids inter-socket traffic altogether. The trade-off is that you lose the ability to run a single model larger than what fits on four GPUs, but for many workloads, four GPUs is enough.

If you must use all eight GPUs for a single model, you need to ensure that the CPU threads are distributed across both sockets, and that each thread's memory is local to its socket. You can do this by setting CUDA_VISIBLE_DEVICES to the GPU list and then using numactl --interleave=all to spread memory allocations across both nodes. That way, each GPU's access to weights is partially local. But interleaving increases the chance of remote access, so it's a trade-off.

Tooling: vLLM and PyTorch Specifics

vLLM has a --num-gpu-blocks parameter that controls the KV cache size, but it does not control NUMA placement. To pin vLLM to a specific NUMA node, you can wrap the launch command with numactl. For example:

numactl --cpunodebind=0 --membind=0 python -m vllm.entrypoints.openai.api_server --model /path/to/model --tensor-parallel-size 4

This runs the server on socket 0 with memory on node 0. If the model is sharded across the four GPUs on that socket, the communication stays within the socket.

For PyTorch, you can set the device and then use torch.cuda.set_per_process_memory_fraction to limit memory, but that doesn't help with NUMA. The key is to use os.sched_setaffinity in your code to pin threads to specific cores. Here's a snippet:

import os
import torch

def pin_to_numa_node(node_id):
    # Get the CPU list for the NUMA node (e.g., from /sys/devices/system/node/node0/cpulist)
    with open(f'/sys/devices/system/node/node{node_id}/cpulist') as f:
        cpulist = f.read().strip()
    # Parse the list (supports ranges like 0-15)
    cpus = []
    for part in cpulist.split(','):
        if '-' in part:
            start, end = map(int, part.split('-'))
            cpus.extend(range(start, end+1))
        else:
            cpus.append(int(part))
    os.sched_setaffinity(0, cpus)

# Before any CUDA calls
pin_to_numa_node(0)
torch.cuda.set_device(0)

That pins the main thread to the cores of node 0. For worker threads, you need to set affinity in each thread's start routine.

The Cost of Ignoring the Trap

The trap is not just about latency; it's about throughput and stability. When the inter-socket link saturates, you get tail latency spikes that are hard to reproduce. Some teams observe that their inference service degrades under concurrent requests because the memory bandwidth is consumed by remote accesses, leaving less for the actual compute.

A common pattern is that a team runs a benchmark with a single request and sees decent numbers, but under load, the p99 latency jumps. They blame the GPU or the model, but the real culprit is the NUMA placement. The fix is often as simple as setting numactl flags.

Conclusion: Make Topology Your Friend

NUMA is not a bug; it's a feature of large systems. The key is to align your software with the hardware. Before you optimize kernels or quantization, check your topology. Use lstopo and nvidia-smi topo -m to map the system. Then pin your processes accordingly. It's a cheap win that can save you from a whole class of performance problems.

In the RiNET stack, we run Qwen on vLLM across three Hetzner servers, each with a modest GPU count. The same principles apply at a smaller scale: we always check the PCIe topology and set numactl flags when launching the server. It's a habit that pays off in consistent latency.

Remember: the 8-GPU NUMA trap is not about having eight GPUs; it's about treating them as one machine when they're really two. Respect the boundaries, and your inference will thank you.