Checkpointing Agent Conversations to Postgres: A Format That Survived a GPU Crash

We lost 12 hours of agent work when a GPU OOM killed a 70B model mid-turn. The conversation was gone — no recovery, no retry, just a blank terminal. That was the day we decided every agent conversation must survive a power cycle.

This post walks through the Postgres checkpoint format we built: a schema that serializes full conversation state, tool call histories, LoRA adapter references, and agent metadata in a single row, with <50 ms write latency. We’ll cover the design decisions, the SQL, the recovery logic, and the hard lessons from running this in production.

Why Checkpoint Agents Differently

A typical web app checkpoint is a user session — store a cookie, maybe a Redis key, and you’re done. Agent conversations are different:

• State is large. A 20-turn conversation with tool outputs can be 50–200 KB of JSON.
• State is nested. Each turn has a user message, assistant response, tool calls, tool results, and internal reasoning steps.
• State is non-deterministic. You can’t just replay the input log — the LLM’s output depends on the exact sequence of tokens, temperature, and even GPU memory layout.
• Recovery must be exact. If you restore a checkpoint, the agent must continue from the exact same logical state, including which LoRA adapters were loaded and which tools were invoked.

We needed a format that could be written atomically, read quickly, and survive a full system crash. Postgres with its JSONB and transactional guarantees was the obvious choice.

The Schema: One Row Per Conversation

After several iterations, we settled on a single table with a JSONB column for the core state, plus indexed metadata columns for querying. Here’s the DDL:

CREATE TABLE agent_checkpoints (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    conversation_id UUID NOT NULL,
    agent_id TEXT NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    turn_number INTEGER NOT NULL,
    state JSONB NOT NULL,
    version INTEGER NOT NULL DEFAULT 1,
    UNIQUE (conversation_id, turn_number)
);

CREATE INDEX idx_checkpoints_conversation_updated
    ON agent_checkpoints (conversation_id, updated_at DESC);

Why JSONB instead of normalized tables? Because the state structure changes frequently as we add new agent capabilities. A normalized schema would require migrations every sprint. With JSONB, we can add fields without schema changes. The trade-off is that we can’t index into deeply nested fields — but we don’t need to. We only query by conversation_id and turn_number.

The State JSON Structure

The state column contains a JSON object with these top-level keys:

{
  "version": 2,
  "conversation": [
    {
      "role": "user",
      "content": "What's the weather in Berlin?",
      "timestamp": "2025-03-15T10:00:00Z"
    },
    {
      "role": "assistant",
      "content": "Let me check the weather API.",
      "tool_calls": [
        {
          "tool_name": "get_weather",
          "arguments": {"city": "Berlin"},
          "result": {"temperature": 12, "condition": "cloudy"},
          "duration_ms": 340
        }
      ],
      "timestamp": "2025-03-15T10:00:02Z"
    }
  ],
  "agent_config": {
    "model": "llama-3-70b",
    "lora_adapters": ["weather-lora-v2", "router-lora-v1"],
    "temperature": 0.3,
    "max_tokens": 4096
  },
  "tool_registry": ["get_weather", "search_web", "send_email"],
  "metadata": {
    "user_id": "user_abc123",
    "session_id": "sess_xyz789",
    "input_tokens": 1523,
    "output_tokens": 421,
    "cost_estimate_usd": 0.023
  }
}

Key design decisions:

• Tool calls are stored inline with the assistant message, not as separate rows. This keeps the conversation array self-contained and makes recovery trivial — you just replay the array.
• Agent config is checkpointed because LoRA adapters and model parameters affect behavior. If you restore a checkpoint, you must load the exact same adapters.
• Tool registry is stored to validate that the agent didn’t lose access to tools after recovery.
• Metadata is for observability, not recovery. We use it for cost tracking and debugging.

Writing Checkpoints: Atomic and Fast

We write a checkpoint after every turn — both user and assistant turns. The write is a single INSERT ... ON CONFLICT (upsert) that either creates a new turn or updates the latest one if the agent is still processing.

import asyncpg
import json
from uuid import uuid4

async def save_checkpoint(pool, conversation_id, agent_id, turn_number, state):
    async with pool.acquire() as conn:
        await conn.execute("""
            INSERT INTO agent_checkpoints
                (conversation_id, agent_id, turn_number, state, version)
            VALUES ($1, $2, $3, $4, 1)
            ON CONFLICT (conversation_id, turn_number)
            DO UPDATE SET
                state = EXCLUDED.state,
                updated_at = now(),
                version = agent_checkpoints.version + 1
        """, conversation_id, agent_id, turn_number, json.dumps(state))

We also batch writes when the agent is streaming — we buffer state in memory and flush every 500 ms or after a tool call completes, whichever comes first. This keeps the DB load low while ensuring we never lose more than one turn of work.

Recovery: Restoring the Exact State

When an agent crashes and restarts, the recovery process is:

1. Query the latest checkpoint by conversation_id ordered by updated_at DESC.
2. Deserialize the state JSON.
3. Rebuild the agent object with the same agent_config (model, LoRA adapters, temperature).
4. Replay the conversation array to rebuild the internal chat history.
5. Validate that all tools in tool_registry are still available.
6. Continue from the last turn — the agent doesn’t know it crashed.

async def recover_conversation(pool, conversation_id):
    async with pool.acquire() as conn:
        row = await conn.fetchrow("""
            SELECT state, agent_id, turn_number
            FROM agent_checkpoints
            WHERE conversation_id = $1
            ORDER BY updated_at DESC
            LIMIT 1
        """, conversation_id)
    if not row:
        return None  # No checkpoint found
    state = json.loads(row['state'])
    agent_id = row['agent_id']
    turn_number = row['turn_number']
    # Rebuild agent here with state['agent_config'] and state['conversation']
    return agent

Performance Numbers

We benchmarked checkpoint writes on a standard Postgres 16 instance (db.r6g.large, 2 vCPU, 16 GB RAM) with 100 concurrent agents:

• P50 write latency: 12 ms
• P99 write latency: 47 ms
• Read latency (single row by conversation_id): 2 ms
• Recovery time (deserialize + rebuild agent): <100 ms
• Storage per 1000 conversations (avg 50 KB each): ~50 MB

These numbers held up under load because the table has no foreign keys and we use JSONB with no indexes on the inner fields. The writes are simple inserts with a unique constraint.

Lessons Learned from Production

1. Always Store the Agent Config

Early versions of our checkpoint only stored the conversation array. When we recovered, we’d load the agent with default settings — different temperature, different LoRA adapters. The agent would start generating responses that didn’t match its previous behavior. We now store the full config and validate it on recovery.

2. Version Your State JSON

We added a version field after the first schema change. Without it, we couldn’t tell if a checkpoint was written by the old or new format. Now we run migration scripts that bump the version and transform the data.

3. Don’t Checkpoint Mid-Turn

We considered checkpointing after every token or after every tool call. That would have given us finer granularity, but the overhead was too high — writes became 200+ ms and recovery logic became complex. We settled on checkpointing after each complete turn (user message + assistant response + tool calls). If the crash happens mid-turn, the agent will replay the last user message and generate a new response. This is acceptable because the cost of regenerating a single turn is far less than the complexity of partial recovery.

4. Use ON CONFLICT for Idempotency

If two processes try to write the same turn (e.g., after a retry), the upsert prevents duplicate rows. The version column increments so we can detect conflicts later.

5. Compress Large Tool Outputs

Some tool calls return images or large JSON blobs. We compress those fields before storing: we replace result with a compressed version and store the compression algorithm in a sibling field. On recovery, we decompress lazily.

Edge Cases We Didn’t Expect

• Clock skew: If the Postgres server and the agent server have different clocks, updated_at can be misleading. We now use now() on the DB side only.
• Orphaned conversations: Agents that crash before writing the first checkpoint leave no trace. We added a heartbeat mechanism that writes a minimal checkpoint with just the conversation_id and agent_id before the first turn.
• Large conversations: After 100+ turns, the JSONB row can exceed 1 MB. We’re considering archiving old turns to a separate conversation_history table, but so far Postgres handles it fine.

The Final Format

After six months of iteration, the checkpoint format has been stable. It saved us three times in production — twice from GPU crashes, once from a network partition that killed the agent process. Recovery is automatic and transparent to the user.

The full schema and Python helper functions are open-sourced in our agent framework repository. You can adapt them to any agent architecture that uses Postgres.

If you’re building agent systems that need to survive failures, start with this: one table, one JSONB column, one write per turn. It’s simple, it’s fast, and it works.