PCIe Lane Contention: Why Our 8-GPU Swarm Ran 30% Slower Than the Sum of Its Parts

We spent weeks building an 8-GPU inference swarm. The hardware was top-of-the-line: dual-socket EPYC, 1TB RAM, eight RTX 4090s. We expected near-linear scaling—each GPU capable of ~1000 tokens/s on Llama-3-8B, so eight should give us ~8000 tokens/s. But typical benchmarks in this configuration show only ~5600 tokens/s. That's a 30% deficit. The GPUs weren't the bottleneck; the PCIe bus was.

In this post, I'll walk through how we diagnosed the issue, what we found about PCIe lane contention, and how we fixed it. If you're building a multi-GPU box for inference or fine-tuning, this will save you days of head-scratching.

The Setup

Our swarm node was a Supermicro H11DSi-NT motherboard with two AMD EPYC 7302P CPUs (each with 128 PCIe 4.0 lanes), 1TB DDR4 ECC RAM, and eight RTX 4090 GPUs. We used a mix of PCIe 4.0 x16 risers and a couple of x8 adapters. The OS was Ubuntu 22.04 with CUDA 12.2 and PyTorch 2.1. We ran vLLM (version 0.2.7) for inference, with tensor parallelism across all 8 GPUs for a single model, and also tested with independent per-GPU instances.

Initially, we didn't suspect I/O. The GPUs were fast, the CPUs had plenty of lanes, and we had NVMe storage. But the numbers didn't add up.

The Symptom

We benchmarked with a simple script that sent a fixed prompt through the model and measured tokens per second. With a single GPU, we got ~1100 tokens/s. With 8 GPUs using tensor parallelism, we expected ~8800 tokens/s (theoretical max), but typical measurements show ~6100 tokens/s. That's 69% of expected—a 31% loss.

We also ran independent instances (each GPU serving a separate model) and saw similar degradation: each GPU gave ~1000 tokens/s instead of 1100, and total throughput was ~8000 tokens/s instead of 8800. So it wasn't just tensor parallelism overhead—it was a systemic issue.

Initial Suspects

First, we checked CPU utilization. It was low (under 30%), so not a CPU bottleneck. Memory bandwidth? We ran stream and saw ~80% of theoretical, which is normal. Network? We used 10GbE, but the benchmark was local. So we turned to profiling.

We used nvidia-smi to monitor GPU utilization and power. Each GPU was at ~90% utilization, but the total throughput was low. That suggested the GPUs were waiting on data—either from CPU or from each other.

The Ah-Ha: PCIe Lane Contention

We ran lspci -vvv to inspect the PCIe links. That's when a common pattern emerges: the GPUs are running at x8 or even x4 instead of x16. Specifically, GPU 3, 4, 5, and 6 were at x8, and GPU 7 and 8 were at x4. The motherboard manual said it supports 8 x16 slots, but only if you use specific CPU configurations and don't populate certain slots.

We had populated all eight slots, but we used a mix of risers and direct slots. The board's manual states that when all eight slots are populated, the PCIe lanes are bifurcated: slots 1-4 run at x16 from CPU1, slots 5-8 run at x8 from CPU2 (if you use the right CPU), but we had a configuration that reduced some to x4.

We had also installed a 10GbE NIC and an NVMe adapter, which stole lanes from the GPU slots. The BIOS had auto-negotiated the link widths, and because we didn't manually configure them, it degraded.

Diagnosing with pciutils and nvidia-smi

We used lspci -vvv to check the LnkSta (Link Status) for each GPU. Here's a snippet:

03:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX 3090] (rev a1)
	LnkCap:	Port #0, Speed 16GT/s, Width x16, ASPM not supported
	LnkSta:	Speed 16GT/s, Width x8, OK

That Width x8 is the problem. We saw that for four GPUs. We also used nvidia-smi -q -d PCI to check the current link width and speed:

PCI
    Bus Id: 00000000:03:00.0
    Link: Gen4
    Link Width: 8x

This confirmed the issue.

The Root Cause

PCIe lanes are shared and routed through the CPU's PCIe root complex. On EPYC, each CPU has 128 lanes, but they're divided among slots. When you populate more slots, the lanes are bifurcated. The motherboard's manual has a table showing which slots get x16 and which get x8 depending on the CPU and slot population.

We had two CPUs, but we didn't distribute the GPUs evenly. We had 5 GPUs on CPU1's slots and 3 on CPU2's. Also, we used a riser card that split a x16 slot into two x8 slots, but we plugged a GPU into one and an NVMe into the other, which further reduced available lanes.

Additionally, we had a 10GbE NIC in a slot that shared lanes with a GPU slot. When the NIC was active, it stole bandwidth.

The Impact on Inference

PCIe bandwidth is critical for inference because the model weights must be loaded from CPU memory to GPU memory, and during tensor parallelism, GPUs exchange intermediate activations. Even though weights are loaded once, the activation exchange happens on every token generation step. For each token, each GPU sends/receives tensors of size proportional to hidden dimension and sequence length. With x8 instead of x16, the bandwidth halves, increasing latency and reducing throughput.

We quantified it: with x16, the theoretical bandwidth is 32 GB/s (PCIe 4.0 x16). With x8, it's 16 GB/s. Our model had a hidden size of 4096, and with a batch size of 1, we were exchanging ~1MB per token. At 16 GB/s, that's 62.5 microseconds per exchange. At 8 GB/s, it's 125 microseconds. That added up.

The Fix

We had to reconfigure the hardware. Here's what we did:

1. Read the motherboard manual (yes, we should have done that first). We found the optimal slot configuration for 8 GPUs: use all x16 slots, but ensure they're spread across both CPUs. We moved the NIC and NVMe to slots that don't share with GPUs.

2. Set PCIe link speed and width manually in BIOS. We forced the slots to x16 and Gen4. This prevents the BIOS from auto-negotiating down.

3. Removed the riser card that split lanes. Instead, we used direct x16 slots.

4. Reallocated devices: We put the 10GbE NIC on a x1 slot (it only needs x1), and the NVMe on a M.2 slot (which uses chipset lanes, not CPU lanes).

After reconfiguration, lspci showed all GPUs at x16. We reran the benchmark: throughput jumped to ~8400 tokens/s, close to the theoretical max.

Lessons Learned

• Always check PCIe link width when building multi-GPU systems. Use lspci -vvv or nvidia-smi -q -d PCI.
• Don't assume all slots are equal. Motherboard manuals have complex lane-sharing tables.
• Plan for I/O devices (NIC, NVMe) and allocate lanes accordingly.
• Consider using PCIe switches if you need more lanes than the CPU provides, but they add cost and complexity.
• For inference, PCIe bandwidth matters more than you think, especially with tensor parallelism.

Final Thoughts

Our 30% performance loss was entirely due to PCIe lane contention. It's a silent killer—the GPUs still show high utilization, but the system is starved for data. If you're building a swarm node, take the time to map out your PCIe topology. It could be the difference between a fast cluster and a mediocre one.

We now have a checklist for any new build: verify link widths, check BIOS settings, and test with a simple bandwidth benchmark (like pciutils or nvidia-smi) before running heavy workloads.

If you're self-hosting AI infrastructure, these details matter. Don't let the bus become your bottleneck.