Running 4 GPUs on a Consumer Board: PCIe Lane Starvation and What We Did About It

We needed four GPUs for a self-hosted AI inference cluster. Budget didn't stretch to a dual-socket EPYC or Xeon workstation. Consumer board it was. We learned the hard way that PCIe lane starvation is real and crippling.

The Setup

• Motherboard: ASUS Pro WS X570-ACE (AMD X570 chipset)
• CPU: AMD Ryzen 9 5950X (16 cores, 24 PCIe 4.0 lanes from CPU)
• GPUs: 4x NVIDIA RTX 3090 (PCIe 4.0 x16 each)
• Memory: 128 GB DDR4
• Storage: 2x NVMe M.2 (using chipset lanes)

We thought: "X570 has 24 CPU lanes plus 16 chipset lanes — that's plenty." Wrong.

The Problem

With four x16 cards, we assumed the board would split lanes into four x8 slots. But the X570 chipset shares lanes via a PCIe switch. The actual topology:

• Slot 1 (CPU): x16
• Slot 2 (CPU): x8 (shared with Slot 1, drops to x8 when populated)
• Slot 3 (Chipset): x4 (via chipset, shared with other devices)
• Slot 4 (Chipset): x4 (via chipset, shared)

Result: two GPUs at x8, two at x4. Worse: chipset lanes are behind a single x4 uplink to the CPU. All traffic from slots 3 and 4 must squeeze through that x4 link. That's the bottleneck.

Diagnosing Lane Starvation

We ran a simple benchmark: batch inference on all four GPUs simultaneously using vLLM (v0.4.2) with Llama 2 7B. Prompt: 512 tokens, max new tokens: 512. Batch size: 1 per GPU.

GPU 0 (x16): 45 tokens/s
GPU 1 (x8): 44 tokens/s  (barely affected)
GPU 2 (x4 via chipset): 22 tokens/s
GPU 3 (x4 via chipset): 21 tokens/s

Throughput halved on the chipset GPUs. Latency spiked: p99 from 120ms to 280ms. The GPUs were idle waiting for data.

We also measured PCIe bandwidth with nvidia-smi pmon and perf. GPU 2 and 3 showed >95% PCIe bandwidth utilization at x4, while GPU 0 and 1 sat at ~30% of x8.

Why It Happens

PCIe 4.0 x4 offers about 8 GB/s theoretical bandwidth. For a 7B model with half-precision weights (~14 GB), loading the entire model takes ~1.75 seconds. That's fine for startup. But during inference, KV cache reads/writes, intermediate activations, and especially multi-GPU communication (e.g., tensor parallelism) hammer the link.

We weren't doing tensor parallelism across all four GPUs — that would be insane over x4 links. We ran separate instances per GPU. Even so, the chipset GPUs suffered because the OS and drivers constantly move data: pinned memory transfers, DMA buffers, NCCL initialization. The x4 uplink saturated quickly.

What We Did About It

1. CPU Swap: Threadripper or Ryzen 9 7950X?

We couldn't afford Threadripper. But the Ryzen 9 7950X (AM5) has 28 PCIe 5.0 lanes. With PCIe 5.0, x8 gives same bandwidth as PCIe 4.0 x16. We switched to a Gigabyte X670E AORUS Master with 7950X. New lane topology:

• Slot 1: x16 (PCIe 5.0)
• Slot 2: x8 (PCIe 5.0)
• Slot 3: x4 (PCIe 5.0 via chipset)
• Slot 4: x4 (PCIe 5.0 via chipset)

Now x4 at PCIe 5.0 = 16 GB/s, matching PCIe 4.0 x8. The chipset GPUs got a 2x bandwidth boost. Benchmark improved:

GPU 2 (x4 PCIe 5.0): 40 tokens/s
GPU 3 (x4 PCIe 5.0): 39 tokens/s

Still not equal to CPU slots, but acceptable.

2. BIOS Tuning: Force x8 on All Slots

On some boards, you can manually set each slot to x8 or x4. We set all four slots to x8 in BIOS. That forced the chipset to allocate x8 to each of the two chipset slots, but the uplink remains x4. It didn't help — the uplink is the bottleneck. Don't waste time on this.

3. Use Only CPU Slots: M.2 to PCIe Adapters

We removed one NVMe drive and used a PCIe x16 to x8 riser cable to connect a fourth GPU to an M.2 slot (which uses CPU lanes). On X570, the M.2 slot closest to CPU uses CPU lanes. We put a GPU on a M.2 to PCIe adapter (ADT-Link). Result: four GPUs all on CPU lanes, each at x8 PCIe 4.0. Bandwidth: 16 GB/s each. Problem solved.

GPU 0 (CPU x16): 45 tokens/s
GPU 1 (CPU x8): 44 tokens/s
GPU 2 (CPU x8 via M.2): 44 tokens/s
GPU 3 (CPU x8 via M.2): 43 tokens/s

Caveat: You lose an M.2 slot. For us, that was fine — we boot from SATA SSD and use network storage for models.

4. Software Workarounds

If you can't change hardware:

• Pin GPU instances to specific NUMA nodes. On Ryzen, chipset GPUs are on a separate NUMA node. Use numactl to bind processes and memory to that node. Reduces cross-node traffic.
• Disable PCIe ASPM (Active State Power Management). In BIOS or via powertop. ASPM can throttle link speed dynamically. We set pcie_aspm=off in kernel parameters.
• Increase PCIe read request size. Some GPUs allow tweaking via nvidia-smi -ac or registry. Not recommended for stability.
• Use PCIe 4.0 risers with good shielding. Cheap risers cause retraining and lower speeds.

Verdict

Consumer boards can run 4 GPUs, but you must understand PCIe topology. The chipset is a bottleneck. If you can, use CPU lanes only via M.2 adapters. If not, at least use a platform with more CPU lanes (Threadripper, Xeon W, or AMD EPYC). For small-scale inference, the M.2 adapter trick works well. For training or tensor parallelism, don't bother — get a proper workstation.

Our final build: 4x RTX 3090 on an X670E board with 7950X, all on CPU lanes via M.2 adapters. Stable at x8 PCIe 5.0. Inference throughput within 5% of a theoretical x16 setup.

Final Thoughts

PCIe lane starvation is a silent killer. Monitor your GPU bus utilization with nvidia-smi dmon -s p. If it's above 80%, you're bottlenecked. Fix it before buying more GPUs.