Durable Checkpointing Under Fire: How We Recover Agents After a GPU Hard Lock Without Lost Steps

A GPU hard lock is the silent killer of long-running autonomous agents. You've seen it: the fan spins up, the screen freezes, and your agent—mid-thought—vanishes. When you reboot, the agent has no memory of the last 30 minutes of work. Steps are lost. State is corrupted. The mission fails.

We run agent swarms on-prem for data-sensitive clients. Crashes are not an option. After a particularly nasty incident where a hard lock wiped out 12 hours of a supply-chain optimization agent, we decided to build a checkpointing system that survives anything short of an SSD failure. Here's how we did it.

The Problem with Naive Checkpointing

Most agent frameworks save state every N steps to a JSON file. That's fine for demos. In production, it leaks like a sieve:

• Crash between saves: Lost steps.
• Partial writes: Corrupt state.
• Disk full: Silent failure.
• No replay: You can't re-execute the last action because the action itself wasn't logged.

We needed a system that guarantees every step is either fully committed or fully recoverable, with no gaps. And we needed it to work under the stress of a GPU hard lock—where the kernel panics, the PCI bus resets, and the machine comes back with dirty filesystems.

Architecture: Postgres as Control Plane

We already use Postgres (16.2) for agent metadata. Why not use it for checkpointing? Postgres gives us ACID transactions, crash recovery, and point-in-time recovery out of the box. We added a dedicated checkpoint table:

CREATE TABLE agent_checkpoints (
    agent_id UUID NOT NULL,
    step_number BIGINT NOT NULL,
    checkpoint_data JSONB NOT NULL,
    action_log JSONB NOT NULL,  -- the action that led to this checkpoint
    created_at TIMESTAMPTZ DEFAULT now(),
    PRIMARY KEY (agent_id, step_number)
);

Every step, the agent writes its state and the action that produced it in a single transaction. If the write succeeds, the step is durable. If the GPU crashes during the write, Postgres rolls back the partial write—no corrupt state.

But wait: the action itself might have been executed against an external API (e.g., a database query or a web request). If the action was committed externally but the checkpoint write failed, we'd have a duplicate action on recovery. To handle that, we use idempotency keys:

CREATE TABLE action_idempotency (
    agent_id UUID NOT NULL,
    idempotency_key UUID NOT NULL,
    step_number BIGINT NOT NULL,
    result JSONB,
    PRIMARY KEY (agent_id, idempotency_key)
);

The agent generates a unique key per action. Before executing, it checks if the key already has a result. If yes, it replays the result without re-executing. If no, it executes, writes the result, and then writes the checkpoint. This guarantees exactly-once semantics even if the crash happens between the action and the checkpoint.

systemd Watchdog: Detect the Crash

A GPU hard lock usually freezes the entire machine. The agent process dies, but the OS might not notice for minutes. We use systemd's watchdog feature to detect hangs within seconds.

Service file snippet:

[Service]
Type=notify
WatchdogSec=30
Restart=always
RestartSec=5

The agent calls sd_notify("WATCHDOG=1") every 10 seconds. If the GPU locks, the agent stops notifying. After 30 seconds, systemd kills the process (if still alive) and restarts it. The agent's SIGTERM handler does a final checkpoint flush if possible, but we don't rely on it.

Recovery: Deterministic Replay

When the agent restarts, it reads the last checkpoint from Postgres. Then it replays all actions after that checkpoint by scanning the action log. But wait: the actions that were in-flight during the crash might have been partially executed. We handle this with the idempotency table: any action that has a result is replayed from cache; any action without a result is re-executed.

The recovery logic in pseudocode:

last_checkpoint = get_latest_checkpoint(agent_id)
current_step = last_checkpoint.step_number

while True:
    next_action = get_next_action_to_attempt(current_step)
    if next_action is None:
        break
    
    key = next_action.idempotency_key
    cached = get_idempotency_result(key)
    if cached:
        result = cached.result
    else:
        result = execute_action(next_action)
        store_idempotency_result(key, result)
    
    new_state = apply_result(last_checkpoint.state, result)
    write_checkpoint(agent_id, current_step, new_state, next_action)
    current_step += 1

This loop replays every uncommitted step deterministically. Because actions are idempotent, duplicates are harmless. Because checkpoints are atomic, we never have partial state.

Handling GPU State

Checkpointing the agent's Python state is easy. Checkpointing the GPU memory (model weights, KV cache) is harder. We don't try to save GPU memory directly. Instead, we rely on the fact that our agents use stateless inference: each step loads the model from disk (vLLM 0.5.3 with --load-format safetensors) and runs inference. The KV cache is rebuilt from scratch on each step. This is slower but makes recovery trivial—no GPU state to save.

For agents that fine-tune on the fly (LoRA adapters), we checkpoint the adapter weights every 10 steps to disk and to Postgres (as a bytea blob). On recovery, we reload the latest adapter before replay.

Performance and Trade-offs

Writing every step to Postgres adds latency. In our benchmarks, a typical checkpoint write takes 5–15 ms (local Postgres on NVMe). For agents that run 100 steps per minute, that's 0.5–1.5 seconds of overhead per minute—acceptable for most workloads.

We batch checkpoints for high-frequency agents (e.g., 1000 steps/min) by writing every 10 steps but flushing the action log every step. The action log is append-only and cheap. If a crash happens between checkpoint writes, we replay up to 10 steps.

Real-World Recovery

We've tested this system by inducing GPU hard locks with a kernel module that triggers a PCIe bus reset. The agent crashes, systemd restarts it, and within 5 seconds the agent is back to work. The last checkpoint is always consistent. Steps lost: zero.

One caveat: if the crash corrupts the Postgres data directory (e.g., a full disk or a faulty SSD), we're dead. We mitigate with streaming replication to a second machine. The agent's connection string points to a failover endpoint (pgbouncer 1.22 with server_check_delay=30). If the primary goes down, the agent reconnects to the replica and continues from the last checkpoint.

Lessons Learned

• Always log the action, not just the state. Without the action, you can't replay deterministically.
• Idempotency keys are non-negotiable. Even if you think your actions are idempotent, they probably aren't.
• Test with real GPU crashes. Simulating a hard lock is easy: echo c > /proc/sysrq-trigger triggers a kernel panic. Do it in a staging environment.
• Monitor checkpoint lag. We alert if the checkpoint table grows faster than expected or if the last checkpoint is older than 5 minutes.

Conclusion

Durable checkpointing is not about saving state—it's about saving the ability to replay. By combining Postgres transactions, idempotency keys, and systemd watchdog, we've built a system that survives GPU hard locks without losing a single step. The code is open-source in our agent framework. You can steal the patterns.

Now go crash your agents. They'll come back stronger.