NUMA Affinity for Pinned Model Weights on 2P Inference Hosts

A dual-socket inference host looks like one machine. It is not. Each CPU has its own memory controller, and a load/store that crosses the inter-socket link costs more than a local one. When model weights are allocated without NUMA awareness, every token can bounce between sockets. The fix is boring and effective: pin each model replica's weights to the NUMA node that owns the memory and the PCIe root complex serving its accelerator.

What actually goes wrong on a 2P box?

On a 2P (two-socket) system, physical memory is split across nodes. A core on socket 0 accessing memory attached to socket 1 pays extra latency and consumes inter-socket bandwidth. Inference makes this worse because weights are read on every decode step. If the allocator hands you pages from the remote node, the hot path is permanently remote.

The common pattern teams observe: the model loads fine, throughput looks acceptable at batch 1, then tail latency widens under concurrency. The culprit is often not the kernel but the page placement. numastat shows a large foreign allocation count for the process, meaning pages landed on the wrong node.

How do you see the imbalance before changing anything?

Start with observation. numactl --hardware lists nodes and distances. numastat -p <pid> shows per-node allocation for a running process. If Foreign is non-zero and growing, the process is touching remote memory.

numactl --hardware
# node 0 cpus: 0-31
# node 0 size: 128000 MB
# node 1 cpus: 32-63
# node 1 size: 128000 MB
# node distances:
# node   0   1
#   0:  10  21
#   1:  21  10

numastat -p $(pgrep -f inference_server)

Distance 21 versus 10 is the signal. Anything above the local distance on the hot path is a tax you are paying per token.

Where should weights live relative to the accelerator?

Weights must be in the memory of the node whose PCIe root complex hosts the GPU or accelerator. On most 2P boards, each socket owns a set of PCIe slots. A GPU in a slot wired to socket 1 should read weights from node 1 memory. If the weights sit on node 0, every weight fetch crosses the inter-socket link before it even reaches PCIe.

Check the topology with lspci and the sysfs NUMA node for the device:

cat /sys/bus/pci/devices/0000:3b:00.0/numa_node
# 1

That number is the node the device prefers. Pin the process and its memory to that node.

What does a correct pinning layout look like?

The decision is per replica, not per host. One replica per NUMA node, each bound to its local CPUs and memory, is the simplest layout that stays correct as you scale.

# Replica on node 1, CPUs 32-63, memory only from node 1
numactl --cpunodebind=1 --membind=1 ./inference_server \
  --model /models/llm-weights \
  --pinned-weights

--cpunodebind restricts scheduling to that node's CPUs. --membind forces allocation from that node's memory. Do not use --preferred here; it is a hint, and hints get ignored under pressure. --membind fails loudly if the node is out of memory, which is what you want during startup rather than at token 10,000.

If the runtime supports it, also set the thread pool affinity inside the process. A common mistake is pinning the process but letting the inference runtime spawn worker threads that the scheduler spreads across both sockets.

Why is pinned memory a separate problem from NUMA binding?

Pinned (page-locked) host memory used for DMA transfers has its own placement. If you pin a buffer from node 0 but the device is on node 1, the DMA engine reads across the inter-socket link. The buffer is locked, so it will not migrate. You have made the remote access permanent.

When allocating pinned buffers, allocate them from the device's node:

// Allocate pinned host memory on the device's NUMA node
void *buf = NULL;
cudaHostAlloc(&buf, size, cudaHostAllocPortable);
// Then advise the kernel to place it on the right node
numa_tonode_memory(buf, size, device_numa_node);

The exact API differs by runtime, but the principle does not: allocate pinned memory on the node that owns the device, not on whichever node happened to run the allocation call.

What should you check before adding more replicas?

Before scaling out, confirm the current layout is actually local. A replica that is nominally pinned but whose threads migrate is worse than an honest unpinned one, because you will trust the wrong number.

• Confirm numastat -p shows near-zero Foreign for the process.
• Confirm the device's numa_node matches the process's --membind node.
• Confirm the runtime's worker threads inherit the affinity mask.
• Confirm pinned DMA buffers were allocated after the bind, not before.

If any of those fail, fix the layout before adding hardware. Adding a second replica on a misconfigured host doubles the cross-socket traffic instead of halving it.

When is NUMA pinning the wrong move?

Pinning is not free. It reduces the memory pool each replica can use, and it makes the host less tolerant of imbalance. If one node has less free memory, a pinned replica will fail to start rather than spill. That is usually the right trade for inference, where predictable latency matters more than packing efficiency.

It is the wrong move when the model is small enough that weights fit comfortably in cache and the bottleneck is elsewhere, or when the workload is batch-oriented and throughput per watt dominates. In those cases, letting the scheduler place pages may be fine. Measure the remote-memory share first; if it is low, pinning buys little.

A practical next step: run numastat -p against one running replica and note the Foreign column. If it is non-trivial, rebuild the launch command with --cpunodebind and --membind matched to the device's numa_node, then compare the same column. What does your Foreign count look like before and after the bind, and did tail latency move with it? Reply with the numbers you see.