Navigating Nvidia's Crippled NVLink: Why Our Multi-GPU Inference Tanked Without Peer-to-Peer

We run a self-hosted inference cluster for large language models — think 70B parameter models split across multiple GPUs. Our stack: two NVIDIA A100 80GB cards connected via NVLink 3.0 bridge. The setup worked fine for months. Then, after a routine driver update (R535 to R545), inference throughput dropped by nearly 40%. Latency spiked. GPUs sat idle waiting for data.

We assumed a driver bug. Rolled back. No change. Re-seated the bridge. Checked nvidia-smi topo -m. Everything looked normal: NVLink status showed active, bandwidth counters non-zero. But actual throughput told a different story.

The Silent Failure: NVLink Peer-to-Peer Broke

Deep diagnostic: we ran a simple all-reduce benchmark using NCCL (Nvidia Collective Communications Library). The point-to-point bandwidth between GPUs showed 600 GB/s — theoretical max. But collective operations (all-reduce, broadcast) ran at 12 GB/s. Something was off.

Turns out: NVLink peer-to-peer (P2P) had silently disabled itself. The driver update changed the default for NVLinkP2PEnable from 1 to 0 in the GPU's persistence mode. No error. No log. Just slower.

When P2P is disabled, inter-GPU communication falls back to PCIe Gen4 x16 — 32 GB/s per direction, shared with host memory. Our models, which rely on tensor parallelism, exchange activations and gradients every forward pass. With P2P off, every tensor transfer went through system memory, competing with data loading and CPU operations. The bottleneck shifted from compute to I/O.

The Fix: Explicit P2P and Memory Pooling

We fixed it with two changes:

1. Force NVLink P2P enable via nvidia-persistenced configuration. Edit /etc/nvidia-persistenced.conf:

   NVLinkP2PEnable=1

   Then restart: systemctl restart nvidia-persistenced. Validate with nvidia-smi -q -d NVLINK — look for "Peer-to-Peer" status.

2. Use CUDA IPC memory handles for tensor parallelism. Instead of relying on NCCL's automatic detection, we explicitly allocated inter-GPU buffers using cudaIpcGetMemHandle and cudaIpcOpenMemHandle. This bypasses the driver's P2P routing and forces direct GPU-to-GPU transfers.

   In our vLLM-based inference server, we modified the model parallelism initialization:

   import torch
   import torch.distributed as dist
   from torch.cuda import Stream
   
   # Force P2P access
   for i in range(torch.cuda.device_count()):
       for j in range(i+1, torch.cuda.device_count()):
           if torch.cuda.can_device_access_peer(i, j):
               torch.cuda.set_device(i)
               torch.cuda.set_device(j)
               torch.cuda.set_device(i)

   This explicit call to can_device_access_peer triggers the driver to re-establish P2P mappings.

The Real Lesson: NVLink Is Fragile

NVLink is a high-bandwidth, low-latency interconnect — when it works. But it's fragile. Driver updates, power management states (like P0 vs P8), and even GPU temperature can cause P2P to fall back silently. Nvidia's documentation mentions this, but it's buried.

We now include a P2P health check in our deployment scripts:

#!/bin/bash
# check_p2p.sh
for gpu in $(seq 0 $(nvidia-smi --query-gpu=index --format=csv,noheader | wc -l)); do
    for peer in $(seq $((gpu+1)) $(nvidia-smi --query-gpu=index --format=csv,noheader | wc -l)); do
        nvidia-smi --id=$gpu --query-gpu=pcie.link.gen.current --format=csv,noheader
        nvidia-smi --id=$peer --query-gpu=pcie.link.gen.current --format=csv,noheader
        # Compare with expected gen (4 for A100)
        if [ $? -ne 0 ]; then echo "P2P issue between GPU $gpu and $peer"; fi
    done
done

Data Sovereignty Implications

Why does this matter for sovereignty? Because multi-GPU inference is essential for running large open-weight models (Llama 3.1 405B, Mixtral 8x22B) on-prem. If your interconnects are flaky, you're forced to use smaller models or cloud APIs. That's a sovereignty failure: you lose control over data, latency, and costs.

We've seen organizations abandon on-prem because "multi-GPU doesn't scale." Usually, it's not the hardware — it's misconfigured NVLink. Don't let a driver setting push you to the cloud.

Practical Recommendations

• Pin NVLink P2P enable in your base image or kickstart.
• Monitor NVLink counters via DCGM (Nvidia Data Center GPU Manager). Alert on bandwidth drops below 90% of theoretical.
• Use explicit CUDA IPC for tensor parallelism; don't trust NCCL's auto-detection in production.
• Test P2P after every driver update with a simple all-reduce benchmark.
• Consider AMD MI300X if you want a more open interconnect (Infinity Fabric) with fewer driver quirks.

Our throughput recovered to 98% of theoretical after the fix. The remaining 2% is overhead from explicit IPC handles. We'll take that over a 40% loss any day.

The Bottom Line

NVLink is powerful but fragile. If your multi-GPU inference tanks, check P2P first. Don't assume the driver does the right thing. Explicitly enable it, validate it, and monitor it. Your sovereignty depends on hardware that works as advertised.