A GPU That Died at 92°C: Thermal Postmortem of a 4-Day Autonomous Agent Run

Last month, I ran a 4-day autonomous agent experiment on a single RTX 4090. The goal was to have a swarm of agents continuously process a stream of documents, summarize them, and update a vector store. I set it up, let it run, and on day four the GPU died. The card was running at 92°C for hours before it gave up. This is the postmortem.

If you're running long inference jobs or autonomous agents on consumer hardware, read this. The mistakes I made are easy to make and expensive to fix.

The Setup

Hardware: RTX 4090 24GB, air-cooled in a mid-tower case. Software: llama.cpp server (v1.1.0) with a Qwen2.5-14B-Instruct GGUF quantized to Q4_K_M. The agent swarm was a Python script that used LangGraph to orchestrate a few agents: one to fetch documents, one to summarize, one to embed and store into a Postgres + pgvector database. The whole thing ran as a systemd service.

The workload was continuous: every 30 seconds, a new document would be picked up, summarized, embedded, and stored. The GPU was under load almost 100% of the time. That's the first mistake.

The Timeline

Day 1: All good. GPU temp hovered around 70°C. Fans spun up to 70% but that was normal. I checked nvidia-smi every few hours.

Day 2: Temps climbed to 80°C. I noticed the fans were at 100% but the card was still hot. I assumed it was because of the workload. I didn't think much of it.

Day 3: Temps hit 88°C. I started to worry. I checked the case airflow: the card was sandwiched between a CPU cooler and a hard drive cage. There was barely any space for air to move. I thought about opening the case but didn't.

Day 4: At around 3am, the systemd service logged an error: "CUDA error: an illegal memory access was encountered." The GPU had died. The card was physically dead—no display output, not recognized by the system.

The Root Cause: Thermal Throttling + Poor Airflow

Let's be clear: 92°C is not a death sentence for a GPU. The RTX 4090 has a thermal throttle at 84°C, and it can operate at 92°C but with reduced performance. The real problem was that the card was constantly thermal throttling, which caused the memory and core to degrade over time. High temperatures accelerate electromigration, which eventually causes permanent damage.

The immediate cause was poor airflow. The card was in a cramped case with no intake fans. The only exhaust was a single 120mm fan at the back. The card was recirculating its own hot air. The GPU's fans were spinning at 100% but they were just pushing hot air around.

The deeper cause was that I didn't have proper thermal monitoring in place. I relied on nvidia-smi manually, not on a continuous monitoring system. By the time I noticed the trend, it was too late.

What I Should Have Done

1. Set a Hard Thermal Limit

The RTX 4090 has a configurable temperature limit via nvidia-smi. I should have set it to 80°C. That would have forced the card to throttle earlier, reducing performance but preventing damage. I didn't because I thought 92°C was safe. It's not for sustained loads.

2. Improve Airflow

I should have mounted the card in a case with proper airflow. A simple fix: open the side panel. But that's not a long-term solution. I should have added intake fans or used a blower-style card. For long-running workloads, consider a server chassis with high static pressure fans.

3. Use Continuous Monitoring with Alerts

I should have set up a monitoring stack that logs GPU temps every minute and alerts me if the temp exceeds a threshold. Tools like Prometheus + node_exporter + grafana are overkill for a single box, but a simple script that logs to a file and sends a Telegram message would have been enough.

The Fix: What I Built After

After the card died, I rebuilt the setup with the following changes:

• New case: Fractal Design Define 7 with three intake fans and two exhaust. The GPU now has direct airflow from the front.
• Thermal limit: Set nvidia-smi -pl 300 (power limit) and -temp-limit 80. That's a hard cap.
• Monitoring: A systemd service that runs a Python script every 60 seconds. It reads nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader,nounits, logs to a file, and if the temp exceeds 75°C, sends an alert via ntfy.sh.

Here's the script I use:

#!/usr/bin/env python3
import subprocess
import time
import requests

def get_temp():
    out = subprocess.check_output(["nvidia-smi", "--query-gpu=temperature.gpu", "--format=csv,noheader,nounits"])
    return int(out.strip())

def send_alert(temp):
    requests.post("https://ntfy.sh/mygpu-alerts",
                  data=f"GPU temp {temp}°C".encode(),
                  headers={"Title": "GPU Overheat"})

while True:
    temp = get_temp()
    with open("/var/log/gpu_temp.log", "a") as f:
        f.write(f"{time.time()} {temp}\n")
    if temp > 75:
        send_alert(temp)
    time.sleep(60)

And the systemd unit:

[Unit]
Description=GPU Temperature Monitor
After=network.target

[Service]
ExecStart=/usr/local/bin/gpu_monitor.py
Restart=always

[Install]
WantedBy=multi-user.target

Lessons for Long-Running Inference

If you're running autonomous agents that hammer the GPU 24/7, treat the GPU as a server component, not a desktop card. Here are the concrete rules I now follow:

• Never run a consumer GPU at 100% load indefinitely without a hard thermal limit. Set the limit to 80°C or lower. You'll lose a few percent of performance but you'll save the card.
• Invest in proper cooling. Undervolting is also an option. I now use nvidia-smi -pl 250 to reduce power draw, which cuts temps by 5-10°C with minimal performance loss.
• Monitor, monitor, monitor. Automate it. Don't rely on manual checks.
• Use a power limiter. For inference, you don't need the full 450W. Setting a power limit reduces heat and energy costs.
• Consider using an older, cheaper GPU for long-running jobs. If you're doing 24/7 inference, a used RTX 3060 or even a Tesla P40 might be more appropriate. They're designed for sustained load.

The Role of the Software

Software also plays a role. llama.cpp by default uses as many threads as possible, which can cause the GPU to work harder than necessary. Setting --threads to a lower number and using --batch-size can reduce power draw. Also, consider using vLLM if you need higher throughput, but it's more memory-hungry.

For agent orchestration, I now use a queue-based system. The agents don't constantly hit the GPU; they pull tasks from a Redis queue and process in bursts. This reduces average GPU load and gives the card time to cool down.

The Cost of Failure

A dead RTX 4090 is a $1600 mistake. Plus the downtime of the experiment. If you're running production workloads, that's unacceptable. The fix is simple: set a thermal limit and monitor. It takes 10 minutes to set up and saves you from a headache.

Final Thoughts

This postmortem is not about blaming the GPU. It's about understanding that autonomous agents, by their nature, run for long periods without human intervention. That means your infrastructure must be designed for that. Thermal management is part of that design.

Now, when I run long experiments, I have a checklist:

• Thermal limit set
• Power limit set
• Monitoring active
• Alerts configured
• Airflow verified

It's boring, but it works. Don't learn this the hard way like I did.

If you have questions about your own setup, feel free to reach out. I'm happy to help you avoid a similar postmortem.