GPU Clock Throttling: Why Inference Latency Doubled at Hour 3

You've seen it happen: a model serves requests at a steady clip for the first few hours, then latency creeps up. By hour three, p99 responses are double what they were at the start. The model didn't change. The traffic didn't change. The GPU just got hot, and the clock controller decided to back off.

This isn't a mystery. It's a well-understood consequence of silicon physics and power management. But it's easy to miss if you're only monitoring average latency and not the GPU's internal state. This article walks through the mechanisms, the symptoms, and the practical ways to keep your inference latency predictable.

The Physics of Throttling

A GPU's clock speed is not a constant. It's a dynamic value that the driver and hardware adjust in response to two constraints: power draw and temperature. The GPU has a target power limit (TGP, or Total Graphics Power) and a thermal limit (usually around 80-90°C for most data-center GPUs).

When a workload is sustained, the GPU's power draw goes up. The cooling system—fans, heatsinks, ambient air—has to remove that heat. If the cooling can't keep up, the temperature rises. Once the temperature hits the thermal limit, the clock controller starts dropping clocks to reduce power and heat generation. This is called thermal throttling.

There's also power throttling. Even if the temperature is fine, the GPU might hit its power limit under a heavy load. The clock controller then reduces clocks to stay within the power envelope. Power throttling can happen on any GPU, but it's more common on models that are overclocked or have a narrow power budget.

Both mechanisms are invisible to the application. The GPU just runs slower. Your inference server sees higher latency, but the GPU's SM occupancy and memory bandwidth look normal. The only clue is the clock frequency reported by nvidia-smi or similar tools.

Why It Takes Hours

Thermal throttling is not instant. A GPU heats up gradually. The thermal mass of the heatsink and the time constant of the cooling system mean that temperature rises over minutes, not seconds. For a heavy inference workload, the GPU might reach its thermal limit after 30 minutes to several hours, depending on the ambient temperature, the cooling solution, and the workload intensity.

Power throttling can happen sooner, but it's often masked by the fact that inference workloads have variable power draw. A burst of large batches spikes power, but idle periods between requests allow the GPU to cool down. The problem is when the workload is uniformly heavy—like a batch inference job that runs continuously without breaks. Then the GPU stays at high power, and the temperature climbs steadily.

Another factor is the GPU's boost behavior. Modern GPUs have a base clock and a boost clock. The boost clock is only sustainable for a short period (often seconds) before the power or thermal limits force a drop. So even at the start of a run, you might see clocks boost to the maximum, but after a few seconds they settle to a lower sustainable level. This is normal. The issue is when the sustainable level itself drops over time.

Detecting Throttling

You need to monitor more than latency. The key metrics are:

• GPU temperature: The core temperature in degrees Celsius.
• GPU clock: The current SM (streaming multiprocessor) clock in MHz.
• Power draw: The current power consumption in watts.
• Throttle reasons: nvidia-smi exposes a set of flags that indicate the reason for throttling (e.g., SW_POWER_CAP, HW_SLOWDOWN, SW_THERMAL).

A common pattern is to log these metrics at regular intervals (e.g., every second) using a tool like nvidia-smi dmon or via the NVML library. If you see the clock frequency dropping while the temperature is at the limit, that's thermal throttling. If the temperature is below the limit but the clock is reduced, it's likely power throttling.

Here's an example of a simple script to log these metrics:

while true; do
  nvidia-smi --query-gpu=timestamp,temperature.gpu,clocks.sm,power.draw --format=csv,noheader
  sleep 1
done

Or, if you prefer Python, use NVML:

from pynvml import *
import time

nvmlInit()
handle = nvmlDeviceGetHandleByIndex(0)

while True:
    temp = nvmlDeviceGetTemperature(handle, NVML_TEMPERATURE_GPU)
    clocks = nvmlDeviceGetClockInfo(handle, NVML_CLOCK_SM)
    power = nvmlDeviceGetPowerUsage(handle)
    print(f"{time.time()}, {temp}, {clocks}, {power}")
    time.sleep(1)

You can also query the throttle reasons:

throttle = nvmlDeviceGetCurrentClocksThrottleReasons(handle)
if throttle & NVML_CLOCKS_THROTTLE_REASON_SW_THERMAL:
    print("Thermal throttling")
if throttle & NVML_CLOCKS_THROTTLE_REASON_SW_POWER_CAP:
    print("Power throttling")

The Impact on Inference Latency

Inference latency is directly tied to GPU clock speed. A lower clock means fewer operations per second. For compute-bound kernels (like the matrix multiplications in transformer layers), latency scales roughly inversely with clock frequency. If the clock drops by 20%, latency increases by about 20%. If it drops by 50%, latency doubles.

Memory-bound operations are less affected, but transformer inference is usually a mix. Attention and feed-forward layers are compute-heavy, while the KV cache and embedding lookups are memory-heavy. The overall effect is that latency degrades, but not necessarily linearly with clock drop.

Another subtle effect: when clocks drop, the GPU might also reduce memory clock. This can affect memory bandwidth, further increasing latency for memory-bound parts.

Mitigation Strategies

You can't change the laws of physics, but you can design your system to avoid hitting the limits.

1. Set a Power Cap

If you're running on hardware where you control the power limit (e.g., via nvidia-smi -pl), you can set a cap that prevents the GPU from ever hitting the thermal limit. For example, if the default TGP is 300W but the cooling can only sustain 250W, set the cap to 250W. This means the GPU will never boost above that, but it will also never throttle. The result is a lower but stable clock.

This is a tradeoff: you lose peak performance but gain predictable latency. For latency-sensitive serving, that's often the right call.

2. Improve Cooling

In a data center, ambient temperature and airflow matter. If you're using a server with GPUs, ensure the intake and exhaust are clear, and consider lowering the ambient temperature if possible. On a workstation, make sure the fans are clean and spinning properly.

For air-cooled GPUs, you can sometimes increase the fan curve using tools like nvidia-settings or vendor utilities. But be careful: louder fans might not be acceptable in all environments.

3. Load Balancing

If you have multiple GPUs, distribute the inference requests across them so that no single GPU is saturated. This reduces the sustained power draw per GPU, keeping temperatures lower.

4. Batch Size Management

Large batch sizes increase GPU utilization and power draw. If you're seeing throttling, try reducing the batch size or adding small delays between requests to let the GPU cool down. This is a crude but effective approach.

5. Use Clock Locking

Some GPUs allow you to lock the clock to a specific frequency. For example, nvidia-smi -lgc 1500 locks the SM clock to 1500 MHz. This prevents the GPU from boosting above that, but also prevents it from dropping below. The result is a constant clock, which gives predictable latency.

However, locking the clock doesn't prevent power or thermal throttling if the GPU still hits those limits. You need to combine it with a power cap or sufficient cooling.

Case Study: A Common Pattern

While I can't share specific numbers from my own infrastructure, a common pattern in the industry is that a GPU serving a large language model (like a 7B parameter model) will start at a boost clock of around 1.8 GHz, then settle to a base clock of around 1.5 GHz after a few minutes. If the cooling is inadequate, the temperature will climb to the limit (say 85°C) after an hour or two, and the clock will drop to 1.2 GHz or lower. That's a 20-30% reduction, which directly translates to higher latency.

Teams that have observed this often report that the latency increase is not linear—it might be a 2x increase because the memory clock also drops, or because the GPU's scheduler becomes less efficient at lower clocks.

Conclusion

GPU clock throttling is a silent killer of inference performance. It's not a failure; it's a protection mechanism. But if you're not monitoring the right metrics, you'll be blindsided by latency spikes hours into a run.

The fix is to either prevent throttling by capping power or improving cooling, or to embrace it by locking clocks and designing for a lower, stable performance envelope. The key is to know your hardware's thermal and power characteristics and to monitor them continuously.

In the next article, we'll look at how to use tools like NVML to build a throttling-aware inference server that can adapt its batch size or routing based on current GPU state. Until then, keep your GPUs cool and your latency predictable.