Silent OOMs on Consumer GPUs: Detecting VRAM Fragmentation in Long-Running Inference

You've seen it: an inference server runs fine for six hours, then suddenly the process dies with an out-of-memory (OOM) error. No warning, no gradual slowdown — just a dead process. You restart it, it works again for another few hours, then dies again. The logs show nothing useful. You blame the driver, the model, or the moon phase.

Chances are, you're hitting VRAM fragmentation. Consumer GPUs (RTX 3090, 4090, A-series) have limited memory bandwidth and no hardware memory management beyond simple paging. Over hours of inference, repeated tensor allocations and deallocations leave the VRAM in a state where no single contiguous block is large enough to satisfy a new allocation, even though total free memory is sufficient. The driver returns an OOM, and your process dies.

This is not a bug in your code. It's a fundamental property of how CUDA allocates memory on consumer hardware. But you can detect it, measure it, and mitigate it.

What Causes VRAM Fragmentation?

Every time you allocate a tensor in PyTorch, the CUDA runtime carves out a chunk of VRAM. When you free it, that chunk goes back to the pool, but the pool is not defragmented. Over time, you end up with a checkerboard of free and allocated blocks. If a new allocation requires a contiguous block larger than any free fragment, the allocation fails.

This is especially common in:

• Long-running inference servers that process requests of varying sizes (e.g., LLM batched inference with dynamic batching).
• Multi-model pipelines where models are loaded and unloaded per request.
• Applications that use CUDA graphs or custom kernels that allocate scratch space.

Consumer GPUs lack the hardware memory compression and advanced TLB features of datacenter cards (A100, H100). They also have no hardware support for memory oversubscription or transparent migration. So fragmentation hits harder and faster.

How to Detect Fragmentation

You cannot directly query the CUDA allocator's internal fragmentation state via standard tools. But you can infer it by monitoring three metrics over time:

1. Total free memory (from nvidia-smi or cudaMemGetInfo).
2. Largest free block (from cudaMemGetInfo as well, but only on newer drivers).
3. Allocation failure rate (from your application logs).

Using nvidia-smi

nvidia-smi shows total free memory, but not the largest free block. However, you can poll it periodically and log the values. A sudden drop in free memory that doesn't recover after a request completes is a red flag.

nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits -i 0

But this is coarse. To get the largest free block, you need to query the CUDA driver directly.

Using cudaMemGetInfo

CUDA 11.0+ provides cudaMemGetInfo which returns both free memory and the largest free block. You can call this from Python via pynvml or cuda-python.

import pynvml
pynvml.nvmlInit()
handle = pynvml.nvmlDeviceGetHandleByIndex(0)
info = pynvml.nvmlDeviceGetMemoryInfo(handle)
print(f"Free: {info.free / 1024**2:.0f} MB")
print(f"Total: {info.total / 1024**2:.0f} MB")
# Largest free block not directly available via pynvml, but via cuda-python:

For the largest free block, you can use the CUDA runtime API via cuda-python:

from cuda import cuda, cudart
err, free, total = cudart.cudaMemGetInfo()
# free is total free memory, total is total VRAM
# largest free block is not exposed; need to use nvidia-ml-py or parse /proc

Unfortunately, the largest free block is not exposed in the standard API. A workaround is to allocate a large block and see if it fails, but that's destructive.

Using CUDA Memory Pool Statistics (CUDA 11.3+)

CUDA 11.3 introduced memory pools with the ability to query pool statistics. You can create a pool and monitor its fragmentation ratio.

import torch
# Create a memory pool
pool = torch.cuda.caching_allocator_alloc(1024*1024*100)  # 100 MB
# Query pool memory info
# This is not directly exposed in PyTorch; use cuda-python:
from cuda import cuda, cudart
# Not trivial; see CUDA documentation for cudaMemPoolGetAttribute

This is advanced and not well-documented. For most cases, a simpler approach works.

Practical Detection Script

Instead of trying to measure fragmentation directly, measure the time between allocation failures. Log every CUDA allocation error and the current free memory. If the error rate increases while free memory is still high, you have fragmentation.

import torch
import time

def monitored_inference(model, input_tensor):
    try:
        output = model(input_tensor)
        return output
    except RuntimeError as e:
        if "out of memory" in str(e).lower():
            free, total = torch.cuda.mem_get_info()
            print(f"OOM at {time.time()}: free={free/1024**2:.0f}MB/{total/1024**2:.0f}MB")
            # Optionally clear cache and retry
            torch.cuda.empty_cache()
            # Log to file for analysis
        raise

Mitigation Strategies

Once you've confirmed fragmentation, you have several options.

1. Periodic Cache Clearing

Call torch.cuda.empty_cache() after each request or batch. This frees all unused cached memory back to the pool, which can reduce fragmentation. However, it adds latency and may cause reallocations.

# After each inference
output = model(input)
torch.cuda.empty_cache()

2. Use torch.cuda.memory.empty_cache() with a Timer

Instead of after every request, do it every N seconds or every M requests. This balances fragmentation control with performance.

import threading

def periodic_cache_clear(interval=300):
    while True:
        time.sleep(interval)
        torch.cuda.empty_cache()

threading.Thread(target=periodic_cache_clear, daemon=True).start()

3. Limit Tensor Cache Size

PyTorch's caching allocator retains freed memory in a cache to speed up future allocations. You can limit this cache size via PYTORCH_CUDA_ALLOC_CONF environment variable.

export PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:128

This prevents the allocator from splitting large blocks into many small ones, reducing fragmentation. Choose a value that matches your typical allocation size.

4. Use a Memory Pool with Fixed Block Sizes

If you control the allocation pattern (e.g., all tensors are multiples of 256KB), you can use a custom allocator that pre-allocates blocks of fixed sizes. This eliminates fragmentation entirely.

import torch
from torch.cuda import CUDAPluggableAllocator

# Not trivial; requires writing a custom allocator in C++

5. Restart the Process Periodically

This is the nuclear option: restart the inference server every N hours. It's simple, reliable, and works for many production systems. Use a process manager like systemd or supervisord to restart automatically.

# systemd service with restart
[Service]
ExecStart=/usr/bin/python3 inference_server.py
Restart=always
RestartSec=10
# Add a timer to restart every 4 hours

Or use a cron job:

0 */4 * * * systemctl restart inference-server

6. Monitor with Prometheus and Grafana

For a robust solution, export memory metrics to Prometheus and set alerts on allocation failures. Use nvidia-ml-py to expose free memory, and your application to expose allocation error counts.

from prometheus_client import Gauge, Counter, start_http_server
import pynvml

vram_free = Gauge('vram_free_bytes', 'Free VRAM', ['gpu'])
oom_counter = Counter('oom_total', 'Total OOM errors')

# In your inference loop:
try:
    output = model(input)
except RuntimeError as e:
    if "out of memory" in str(e):
        oom_counter.inc()
        # handle

Case Study: Fragmentation in vLLM

vLLM, a popular LLM inference server, uses PagedAttention to manage KV cache in blocks. This reduces fragmentation by design because blocks are of fixed size. However, if you use vLLM with dynamic batching and large sequence lengths, you can still see fragmentation in the residual memory used by the model weights and activation buffers.

In one deployment with an RTX 4090 running Llama 2 7B, we saw OOMs after 8-10 hours of continuous inference. Monitoring showed free VRAM oscillating between 2GB and 4GB, but allocation failures occurred when free memory was still 3GB. The largest free block had shrunk to under 500MB due to fragmentation from repeated tensor allocations in the attention mechanism.

We mitigated by:

• Setting max_num_batched_tokens to 4096 to limit batch size.
• Calling torch.cuda.empty_cache() every 5 minutes via a background thread.
• Setting PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:256.

After these changes, the server ran for 72 hours without an OOM.

Conclusion

VRAM fragmentation is a silent killer of long-running inference on consumer GPUs. You can't fix it, but you can detect it by monitoring allocation errors and free memory trends. Mitigate with periodic cache clearing, limiting the allocator's split size, or simply restarting the process on a timer. For production systems, export metrics to Prometheus and alert on OOM rate.

Consumer GPUs are not designed for 24/7 inference, but with the right monitoring and mitigation, you can keep them running reliably for days.