Silent CUDA Leak: How Attention Masks Took Down Three Agents

A silent CUDA memory leak is the worst kind of production bug. It doesn't fail fast; it fails slow, after your monitoring has gone quiet and your agents have been running for eight hours. When three of our agents suddenly died overnight, we assumed it was a network issue or a OOM killer from a misconfigured container. But the logs told a different story: each agent had been processing requests normally, then abruptly stopped responding. The host was fine, the GPU was still alive, but the CUDA context had ballooned to the point where any new allocation failed. The culprit? Attention masks.

This is the story of how we traced a memory leak that only manifests after hours of runtime, and the lesson that attention masks are not as harmless as they seem.

The Symptom: Agents Die After Eight Hours

Our agent runtime runs on a stack of three Hetzner servers connected via WireGuard, with Qwen models served through vLLM. The agents handle a mix of short and long prompts, from simple queries to multi-turn conversations with long context windows. Everything works fine for the first few hours. Then, around hour eight, the agents start throwing CUDA OOM errors. Not all at once, but one by one, as if a timer was going off.

The first thing we checked was obvious: was there a memory leak in our agent code? We had a nightly LoRA fine-tune job that runs on the same GPUs, and we suspected it might be leaving stale tensors. But the leak persisted even when the fine-tune job was disabled. We then looked at vLLM's memory management, thinking it might be a bug in the paging logic. But vLLM has been stable for us; the leak wasn't there.

We started collecting nvidia-smi snapshots every minute across all three servers, logging the memory usage per process. The pattern was clear: the memory usage would grow linearly with the number of requests processed, but only for agents that had been running for a long time. Fresh agents were fine. The leak was in our agent code, not in the inference server.

The Hunt: Profiling CUDA Allocations

We needed to find where the memory was going. We used nsys and cuda-gdb to profile the agent process, but the leak was too slow to catch in a short session. We wrote a simple script that called the same inference endpoint with a dummy prompt in a loop, and watched the memory usage climb. It took about 1000 iterations to see a noticeable increase, but it was there.

The key was to isolate what changed between iterations. We were passing a attention_mask to the model, as most transformer libraries require. We built the mask using a list comprehension that created a new tensor for each request. The mask was created with torch.tensor(mask_list, device='cuda'), which allocates a new tensor on the GPU each time. That's fine if you're not holding onto it, but we were storing the mask in a global cache to avoid recomputing it for repeated prompts. The cache was a simple dictionary that never evicted entries. Over time, the cache grew, and each entry held a CUDA tensor. After hours of runtime, the cache had thousands of entries, each holding a small but non-negligible chunk of GPU memory.

But that alone wouldn't explain the crash. The cache was small, maybe a few MB. The real problem was that the mask tensors were not being freed because we were holding references to them in the cache, and also because we were accidentally creating a new mask for every token generation step, not just once per request.

The Root Cause: Mask Re-creation Every Step

In our agent loop, we had a function that prepared the input for the model. It took the conversation history, tokenized it, and built an attention mask. The mask was built by checking which tokens were padding. The bug was that we were creating the mask inside the loop that generates each token, not once per request. So for a 100-token response, we created 100 masks, each a new CUDA tensor. The previous mask was supposedly freed, but because we were using a Python list to store the mask tensors for the entire generation, they were all kept alive.

The code looked something like this:

# Simplified agent loop
for step in range(max_steps):
    input_ids = torch.tensor(context_ids, device='cuda')
    attention_mask = torch.tensor(mask_list, device='cuda')
    outputs = model.generate(input_ids, attention_mask=attention_mask, ...)
    # ...
    context_ids.append(new_token)
    mask_list.append(1)  # we kept appending to mask_list

The mask_list was a Python list that we kept appending to, and then we converted it to a tensor at each step. This meant that the list grew with each step, and the tensor conversion created a new tensor each time, which was kept alive because it was passed to model.generate. The model might keep a reference to it internally for the forward pass, but after that, it should be freed. However, because we were storing the mask in the mask_list and then creating a new tensor from it, the old tensor was still referenced by the list (since the list holds the mask list, not the tensor). But we were also storing the tensor in a global variable for debugging, which kept it alive.

We found the leak by adding a torch.cuda.memory_summary() call after each generation step and monitoring the allocated memory. The memory would increase by roughly the size of the mask tensor each step. After 1000 steps, that's 1000 mask tensors.

The fix was simple: build the mask once per request, and reuse it for all steps. Also, we stopped storing the mask in a global variable and made sure to delete any references after use.

The Tracing Technique: Memory Snapshots and Object Graphs

To trace this, we used a combination of tools:

• torch.cuda.memory_snapshot() to get a detailed view of what tensors are allocated.
• tracemalloc for Python-side allocations, but that didn't help with CUDA memory.
• pytorch_memlab to track the line of code that allocated each tensor.

We also used gc.collect() and torch.cuda.empty_cache() to see if the memory would be freed, but it wasn't, because the references were still alive.

The most effective technique was to use torch.cuda.memory_snapshot() and then analyze the snapshot to find tensors that were allocated but not freed. We wrote a script that dumped the snapshot to a file, and then used a custom parser to group tensors by their allocation point. We found that the allocation point was the line that created the attention mask.

The Fix: Reuse Masks and Cache with Eviction

We refactored the agent loop to build the attention mask only once per request, before the generation loop. We also changed the cache to use an LRU eviction policy so that old entries are removed when the cache reaches a certain size. The mask tensor is now created on the CPU and moved to the GPU only once, and then reused for all steps. We also made sure to set torch.cuda.set_per_process_memory_fraction to a safe limit to prevent the entire GPU from being consumed.

The code now looks like:

# After refactor
input_ids = torch.tensor(context_ids, device='cuda')
attention_mask = torch.tensor(mask_list, device='cuda')
for step in range(max_steps):
    outputs = model.generate(input_ids, attention_mask=attention_mask, ...)
    # ...

We also added a check to ensure that the mask tensor is not accidentally recreated inside the loop.

Lessons Learned: Attention Masks Are Not Free

This incident taught us several things:

1. Attention masks are tensors too. They occupy GPU memory, and if you create them in a loop, you're allocating memory every iteration. Always create them once per request.
2. Caching is dangerous if you don't evict. Any cache that holds GPU tensors must have a bounded size. Unbounded caches will eventually exhaust memory.
3. Monitor memory over time, not just at peak. A leak may not show up in a short test. Use continuous monitoring to catch slow growth.
4. Use memory profiling tools early. torch.cuda.memory_snapshot() is your friend. Don't wait for a crash to start profiling.

General Pattern: Memory Leaks in Agent Runtimes

This is not an isolated incident. Many agent runtimes that call LLM APIs or run local models have similar issues. A common pattern is to store conversation history in a list and rebuild the input tensors for each turn, forgetting to free the old ones. Another is to use a global cache for embeddings or masks without eviction.

We've seen teams report similar issues on forums: agents that run for hours start slowing down or crashing due to GPU memory growth. The typical cause is not the model itself but the surrounding code that manages tensors.

A good practice is to structure your agent loop to separate the data preparation from the inference call. Build the input tensors once, and if you need to extend the context, do it by concatenating tensors, not by rebuilding from Python lists.

Conclusion: The Silent Leak Is Now Silent No More

We've fixed the leak, and our agents now run for days without issue. The key was to treat attention masks as first-class memory citizens and to profile memory usage over long periods. If you're running agents on GPUs, check your attention mask handling. It might be the silent killer.

Remember: the GPU is a finite resource, and every tensor you create has a cost. Be mindful of what you allocate, and always clean up after yourself.