Checkpointing Agent Conversations to Postgres: A Binary Format That Survived a GPU Crash

TITLE: Checkpointing Agent Conversations to Postgres: A Binary Format That Survived a GPU Crash

BODY: We run a fleet of autonomous agents powered by Qwen on vLLM, orchestrated over a WireGuard mesh across three Hetzner servers. Each agent carries a conversation context that can span thousands of turns — instructions, tool outputs, embedded memories from BGE-M3, graph nodes from Neo4j. Losing that context to a GPU hang or OOM kill means breaking a user's workflow. So we checkpoint aggressively into Postgres. But the naive approach — serializing the whole context as JSON — became a bottleneck. That's when we switched to a binary format.

The Problem with JSON Checkpoints

Our agents are stateful. Each carries a payload that includes:

• A list of message dicts (role, content, tool_calls, etc.)
• Embedding vectors for recent context (list of floats, 768 dimensions each)
• A small graph of entities extracted so far (list of node dicts)
• Agent metadata (task ID, parent agent ID, temperature, etc.)

Early on, we persisted these as JSON blobs. The first issue: size. A single agent conversation after 200 turns could balloon to ~500 KB of JSON, mostly from repeated keys like "role": "user". Multiplied by hundreds of concurrent agents, the I/O to dump and reload became a drag on the database server.

The second issue: deserialization cost. Python's json.loads on a 500 KB string takes non-trivial CPU time. After a crash, we had to reload dozens of agents simultaneously — the recovery window stretched into minutes. Not acceptable.

The third issue: schema evolution. JSON is flexible, but when we changed the internal structure (e.g., added a new field to tool calls), older checkpoints either failed to load or required migration scripts. We needed something more explicit.

Choosing a Binary Serialization Format

We evaluated a few options:

• Protobuf: Explicit schema, compact, but requires code generation and a schema registry. Overkill for our dynamic agent contexts.
• Pickle: Python-native, but dangerous for untrusted data and brittle across Python versions. Not acceptable.
• MessagePack: Schema-less but more compact than JSON; supports native binary; fast in Python with the msgpack library. No code generation needed.

We went with MessagePack. It's a drop-in replacement for JSON in many cases, but produces smaller payloads (no key repetition in maps can be compressed away if you use a fixed schema convention). For agent contexts, teams typically observe a reduction in payload size compared to JSON (though exact numbers vary by structure). More importantly, deserialization was significantly faster because the parser is simpler and we didn't have to allocate as many string objects.

Database Schema for Checkpoints

We store each checkpoint as a row in a single table:

CREATE TABLE agent_checkpoints (
    agent_id UUID NOT NULL,
    conversation_id UUID NOT NULL,
    step BIGINT NOT NULL,
    checkpoint BYTEA NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    PRIMARY KEY (agent_id, step)
);

CREATE INDEX idx_checkpoints_latest
    ON agent_checkpoints (agent_id, step DESC);

The checkpoint column holds the raw MessagePack binary. We rely on the database user to have no write access to old rows — we always insert new steps. A background vacuum keeps the table from growing unbounded.

Why a separate step counter? Because agents can fork (parallel tool calls), and we need to track which branch is the canonical one. The step is monotonically increasing within a conversation. On recovery, we simply get MAX(step) WHERE agent_id=?.

Checkpointing Logic in Python

Serialization is straightforward. Here's the core function:

import msgpack

def checkpoint_agent(agent_id, conversation_id, step, agent_state):
    # agent_state is a dict with all relevant data
    packed = msgpack.packb(agent_state, use_bin_type=True)
    
    with db_pool.connection() as conn:
        conn.execute(
            """INSERT INTO agent_checkpoints
               (agent_id, conversation_id, step, checkpoint)
               VALUES (%s, %s, %s, %s)
               ON CONFLICT (agent_id, step) DO NOTHING""",
            (agent_id, conversation_id, step, psycopg2.Binary(packed))
        )

We use ON CONFLICT DO NOTHING because in rare cases two replicas might attempt the same step. The step number is assigned by a global monotonic counter (we use a simple atomic integer per agent, not a distributed sequence — the agent only runs on one node).

Recovery is equally simple:

def load_latest_checkpoint(agent_id):
    with db_pool.connection() as conn:
        row = conn.fetchone(
            """SELECT checkpoint, step FROM agent_checkpoints
               WHERE agent_id = %s
               ORDER BY step DESC
               LIMIT 1""",
            (agent_id,)
        )
    if row is None:
        return None, 0  # no checkpoint, start fresh
    state = msgpack.unpackb(row[0], raw=False)  # raw=False for dicts, not bytes
    return state, row[1]

That's it. After a GPU crash, we iterate over all known agent IDs, load the latest checkpoint, and resume execution from that step. Because the binary is small and the query is a simple index scan, we can restore tens of agents in under a second.

Surviving a Real GPU Crash

One afternoon, a rogue CUDA kernel on a vLLM instance caused a full GPU hang. The entire process died. RAM was lost. But our agent state lived in Postgres. When the node came back, the orchestrator (a simple Python loop on the control plane) detected the missing agents and began recovery. Each agent re-fetched its conversation history from the latest checkpoint, re-embedded new user messages with BGE-M3, and resumed the conversation flow. Users saw a brief pause — not dropped connections.

The binary format was critical here. If we had been writing JSON, the I/O storm during recovery would have bottlenecked Postgres, and the CPU cost of parsing would have delayed agent resumption. With MessagePack, the CPU spent more time on the actual model inference than on serialization.

Trade-offs and Edge Cases

Staleness: We checkpoint every N steps (N=10 in our setup). If a crash happens between checkpoints, those last few steps are lost. The agent will re-process the last tool output, which is idempotent for our design. We accept this trade-off for lower checkpoint overhead.

Schema changes: Since MessagePack is schema-less, adding a new field to the agent state is fine — old checkpoints just lack that field. We handle missing keys with .get() defaults. No migration needed.

Binary size: Although MessagePack is smaller than JSON, we periodically compress old checkpoints with Zstd to save space. The checkpoint column could be a COMPRESSED column if using a Postgres extension, but we prefer application-level compression for portability.

Concurrent writers: Only one agent process writes to its own agent_id. For agents that spawn sub-agents, we use a coordinator that writes checkpoints sequentially. No locking issues.

Alternatives Considered

ARIES-style logging: For a while we considered storing every action as a log and replaying to reconstruct state. That's powerful but overkill for our use case — we don't need full auditability, just crash resilience.

Redis persistence: Redis with AOF could work, but we already had Postgres running for metadata and Qdrant for vectors. Adding another stateful service increased operational overhead.

Flat files: We tested writing to NVMe SSDs directly. File corruption during a power loss was a real risk, and distributed recovery across nodes required NFS or S3, adding latency.

Postgres with MessagePack hit the sweet spot: durable, human-understandable (the binary is opaque, but the schema is simple), and fast to reload.

Code: Complete Example

Here's a minimal self-contained pattern you can adapt:

import msgpack
import psycopg2
from uuid import uuid4

DB_DSN = "dbname=agents host=localhost user=agent password=secret"

def save(agent_id, step, state_dict):
    conn = psycopg2.connect(DB_DSN)
    cur = conn.cursor()
    packed = msgpack.packb(state_dict, use_bin_type=True)
    cur.execute(
        "INSERT INTO agent_checkpoints (agent_id, conversation_id, step, checkpoint) "
        "VALUES (%s, %s, %s, %s) ON CONFLICT DO NOTHING",
        (agent_id, uuid4(), step, packed)
    )
    conn.commit()
    cur.close()
    conn.close()

def load(agent_id):
    conn = psycopg2.connect(DB_DSN)
    cur = conn.cursor()
    cur.execute(
        "SELECT checkpoint, step FROM agent_checkpoints WHERE agent_id = %s "
        "ORDER BY step DESC LIMIT 1",
        (agent_id,)
    )
    row = cur.fetchone()
    cur.close()
    conn.close()
    if row:
        return msgpack.unpackb(row[0], raw=False), row[1]
    return {}, 0

Final Thoughts

Checkpointing agent conversations to Postgres with a binary format isn't glamorous, but it's the kind of engineering that directly impacts uptime. The combination of a well-understood relational store, a fast serialization library, and a simple recovery loop has given us confidence that even if a GPU dies mid-turn, the conversation can go on.

The lesson: don't overcomplicate state persistence. Pick a format that's compact and fast to parse, store it in a battle-tested database, and keep the recovery logic as a single SQL query + a deserialize call. Your future self (and your users) will thank you.