NUMA Affinity for vLLM: Pin Inference Threads First

Why cross-socket memory traffic quietly caps LLM throughput on multi-socket bare metal, and how to stop it.

by

A multi-socket server looks like one big machine to the OS but behaves like two machines sharing a slow wire. When vLLM threads and their KV cache land on different NUMA nodes, every attention step pays a cross-socket memory tax. Pinning inference work to a single socket is the cheapest latency win available before you touch model code.

What is NUMA and why does it matter for LLM inference?

NUMA means each CPU socket owns a slice of RAM. A core reads its local slice fast and the remote slice through an interconnect that is slower and shared. LLM inference is memory-bandwidth bound: the model weights and KV cache are streamed constantly, so remote reads hurt more than in typical web workloads.

A common pattern on dual-socket hosts is that the scheduler spreads vLLM worker threads across both sockets while the KV cache pages sit wherever the allocator first touched them. The result is a steady fraction of remote traffic that no amount of batching fixes.

How does the cross-socket tax show up in vLLM?

It shows up as latency that does not scale with batch size. Teams typically observe that time-per-output-token stays flat or rises when they add concurrency, even though GPU utilization looks healthy. The GPU is waiting on host-side work that is reading remote memory.

The tax is worst in three places:

  • Tokenization and detokenization, which run on CPU and touch request buffers.
  • Scheduler and block manager bookkeeping, which walks KV cache metadata.
  • Sampling and logits post-processing when they are not fully offloaded.

None of these are the matmul. They are the glue, and the glue is where remote memory shows up.

How do you check the current NUMA layout?

Start by mapping the hardware before changing anything. On Linux, numactl and lscpu give you the topology in seconds.

lscpu | grep -i numa
numactl --hardware

Look at the node distances. A local distance of 10 and a remote distance of 21 is the classic dual-socket signature. Then check where your process actually lives.

numastat -p $(pgrep -f vllm)

If Numa_Hit and Numa_Miss are both large, the process is straddling sockets. That is the signal to pin.

How do you pin vLLM to one NUMA node?

The simplest approach is to launch the server under numactl with CPU and memory binding set to the same node. This keeps the Python process, its worker threads, and the KV cache allocations on one socket.

numactl --cpunodebind=0 --membind=0 \
  python -m vllm.entrypoints.openai.api_server \
  --model /models/your-model \
  --tensor-parallel-size 1

The key flag is --membind. CPU binding alone does not move memory; the allocator will still pull pages from the remote node under pressure. Bind both.

For multi-GPU hosts, align each tensor-parallel rank with the socket that owns its GPU. On most dual-socket boards, GPUs are wired to a specific CPU's PCIe root complex. Pin rank N to the node that owns GPU N.

CUDA_VISIBLE_DEVICES=0 numactl --cpunodebind=0 --membind=0 \
  python -m vllm.entrypoints.openai.api_server --model /models/your-model

If you run multiple ranks, wrap each in its own numactl invocation or use a launcher that respects per-rank affinity.

What about the KV cache and huge pages?

The KV cache is the largest host-side allocation vLLM makes. If it is backed by transparent huge pages, the kernel may migrate or split pages in ways that cross sockets. Disable THP for the vLLM process or reserve huge pages per node.

echo never > /sys/kernel/mm/transparent_hugepage/enabled

On hosts where huge pages are already reserved, allocate them per NUMA node with numactl --membind so the pool does not span sockets. The goal is one node, one memory pool, one set of threads.

When should you not pin?

Pinning is not free. It reduces the CPU and memory the process can use, which matters when the model is large and the host is small. If a single socket cannot hold the KV cache plus the runtime, binding will cause OOM kills instead of speedups.

A practical rule: pin when the model fits comfortably on one node and you run one server per socket. Skip it when you are memory-constrained or when the workload is dominated by GPU compute and host-side work is negligible.

Before you change anything, run numastat on a live vLLM process and note the miss count. That single number tells you whether pinning is worth the effort on your host.

What should you measure after pinning?

Do not trust throughput alone. Watch Numa_Miss after binding; it should drop toward zero for the pinned process. Then compare time-per-output-token at the same concurrency you ran before. If the miss count falls but latency does not, the bottleneck was never the socket.

Have you checked numastat on your inference host yet, and what did the miss count tell you?

#bare-metal#llm-inference#numa#performance#vllm
Share — X / Twitter · LinkedIn · HN · Email
Damir Radulić
Founder of RiNET. On the Croatian internet since 1996 (Kvarner Net). In Amsterdam now, building autonomous AI infrastructure that runs on Monday morning when nobody's watching — sovereign stacks, agent swarms, LoRA fine-tuning, civic-intelligence platforms.