Checkpointing at the Action Level: Why We Store Every Agent Step and How It Saves Runs

You’ve seen it happen. An agent runs for three hours, calls twenty APIs, generates ten thousand tokens, and then—crash. A transient network error, an OOM killer, a database connection timeout. The run is lost. You restart from scratch, burning time and compute.

Most agent frameworks checkpoint at the task level. If a task has ten steps and step five fails, you lose steps five through ten. That’s wasteful. Worse, it makes debugging nearly impossible because you can’t replay the exact sequence that led to the failure.

We checkpoint at the action level. Every single step—every tool call, every LLM completion, every state transition—is persisted atomically. If the agent dies, we resume from the last completed action, not the last completed task. This article explains why that matters, how we implement it with Postgres, and the hard-won lessons from production.

The Problem with Task-Level Checkpointing

Most agent orchestrators (e.g., LangChain, AutoGPT, CrewAI) default to checkpointing at task granularity. A task might be “research competitor pricing” which involves five sub-steps: scrape a page, extract data, summarize, compare, format. If the compare step fails, you lose the entire task’s progress. The agent re-scrapes, re-extracts, re-summarizes. That’s not just slow—it’s expensive. Each LLM call costs money. Each API call risks rate limits.

Task-level checkpointing also hides the real error. The root cause might be a malformed output from the summarizer, but you never see it because the whole task is retried. You burn time and money chasing ghosts.

Action-Level Checkpointing: A Minimal Definition

An action is the smallest unit of work that changes the agent’s state. It could be:

• An LLM completion (prompt + response)
• A tool invocation (function name + arguments + result)
• A conditional branch decision (which path was taken)
• A variable assignment (key + value)

We store each action as a row in a checkpoints table. The schema is dead simple:

CREATE TABLE checkpoints (
    run_id      UUID NOT NULL,
    step_index  INT  NOT NULL,          -- monotonically increasing per run
    action_type TEXT NOT NULL,          -- 'llm_completion', 'tool_call', 'branch', etc.
    input       JSONB NOT NULL,         -- the action's input
    output      JSONB,                  -- the action's output (null if not yet completed)
    created_at  TIMESTAMPTZ DEFAULT now(),
    PRIMARY KEY (run_id, step_index)
);

That’s it. No fancy graph database. No event sourcing framework. Just Postgres with a composite primary key. We batch inserts in transactions so that a single run’s checkpoint history is consistent.

How Recovery Works

When an agent crashes and restarts, it queries the most recent checkpoint for its run:

SELECT step_index, action_type, input, output
FROM checkpoints
WHERE run_id = $1
ORDER BY step_index DESC
LIMIT 1;

If the last action is completed (output is not null), we start the next action. If it’s incomplete (output is null), we re-execute only that action. This is critical: we never redo a completed action. Idempotency is assumed at the action level—if a tool call is not idempotent, you need to handle that at the tool layer, not the checkpoint layer.

We also store the full input and output so that recovery can reconstruct the agent’s exact state. For LLM completions, that means the entire message history up to that point is implicit—we can replay the conversation by reading all checkpoints in order.

Why Postgres?

We use Postgres 16 with pgvector for other parts of the stack, but for checkpoints, plain JSONB is sufficient. The advantages:

• Transactional guarantees: We can atomically write a batch of checkpoints. If the agent crashes mid-write, the last consistent state is preserved.
• Point-in-time recovery: With WAL archiving, we can restore a run’s state to any second.
• No extra infrastructure: We already run Postgres for metadata. Adding a table is trivial.
• Queryability: You can analyze failed runs by querying the checkpoint history. “Show me all runs where the last action was a tool call to search_web with status code 429.”

We considered Qdrant for vector embeddings of actions (to find similar past failures), but that’s a future optimization. The checkpoint store itself is relational.

Real-World Impact: A Case Study

One of our agents monitors a SaaS dashboard and escalates anomalies. It runs a loop: scrape metrics, detect outliers, write a summary, send an alert. Each loop is about 15 seconds and involves 3 LLM calls and 4 API calls. The agent runs 24/7, completing ~5,760 loops per day.

Before action-level checkpointing, a crash (which happened ~3 times per week due to transient network issues) would lose the current loop. That’s 15 seconds of work—not terrible. But the real pain was debugging: without per-action logs, we couldn’t tell which step failed. We’d see “loop failed” and have to guess.

After implementing action-level checkpoints, we could pinpoint exactly which API call returned a 503. We added a retry with exponential backoff at the action level, and crash frequency dropped to near zero. When a crash does happen, recovery is instant—the agent picks up from the failed action, not the start of the loop.

Another agent runs a multi-hour research pipeline: scrape 100 pages, extract entities, build a knowledge graph. A crash at 3 hours used to lose everything. Now it resumes from the last completed scrape. We saved an average of 2.5 hours per crash. Over a month, that’s dozens of hours of compute.

Performance Considerations

Storing every action is cheap. An action record is ~1 KB on average (input + output). A busy agent doing 1,000 actions per hour generates ~1 MB of checkpoint data per hour. Postgres handles that easily. We use a separate tablespace on fast SSD storage for checkpoints to avoid contention with other tables.

We batch writes in groups of 100 actions with a single INSERT INTO ... VALUES (...), (...), ... statement. This keeps write latency under 1 ms per batch. Reads are by primary key—always fast.

One gotcha: the output field for LLM completions can be large (10 KB+). We considered compressing it with pg_column_compress (available in Postgres 16 via toast_compress), but we found that the overhead of compression wasn’t worth it for our throughput. Your mileage may vary.

Lessons Learned

Lesson 1: Make actions idempotent. If a tool call creates a resource (e.g., sends an email), you must ensure that replaying the action doesn’t duplicate it. We use idempotency keys: each action carries a unique action_id (UUID), and the downstream service deduplicates based on that key. This is a hard requirement for safe recovery.

Lesson 2: Store the full input, not just a reference. We initially stored only the action type and a pointer to external state (e.g., “the message history is in Redis”). That broke recovery when Redis was cleared. Now we inline everything. Storage is cheap; debugging time is not.

Lesson 3: Prune old checkpoints. We keep checkpoints for 30 days, then archive to S3 as JSONL files. A cron job runs daily:

DELETE FROM checkpoints
WHERE created_at < now() - interval '30 days';

We also have a retention policy for failed runs: keep forever (for debugging), but compress with pg_repack after 90 days.

Lesson 4: Monitor checkpoint throughput. We track checkpoints_inserted_per_second and checkpoints_read_per_second in Prometheus. A sudden drop in writes usually means the agent is stuck or crashed. A spike in reads indicates a recovery event.

When Not to Use Action-Level Checkpointing

It’s not always the right choice. If your agent runs for seconds and failures are cheap, the overhead of writing a database row per action may not be worth it. Also, if your actions have side effects that are impossible to make idempotent (e.g., launching a rocket), you need a different approach—perhaps a two-phase commit or a saga pattern.

But for 90% of autonomous agents—research bots, monitoring systems, data pipelines—action-level checkpointing is a net win. The cost is a few milliseconds per action. The benefit is resilience, debuggability, and peace of mind.

Implementation Guide

Here’s a minimal Python implementation using psycopg2 and a decorator:

import psycopg2
from functools import wraps
from uuid import uuid4

conn = psycopg2.connect("dbname=agents")
cursor = conn.cursor()

def checkpoint(action_type):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            run_id = kwargs['run_id']
            step_index = kwargs['step_index']
            action_id = str(uuid4())
            input_data = {'args': args, 'kwargs': kwargs}
            # Write incomplete checkpoint
            cursor.execute(
                """INSERT INTO checkpoints (run_id, step_index, action_type, input)
                   VALUES (%s, %s, %s, %s)
                   ON CONFLICT (run_id, step_index) DO NOTHING""",
                (run_id, step_index, action_type, psycopg2.extras.Json(input_data))
            )
            conn.commit()
            try:
                result = func(*args, **kwargs)
                # Update with output
                cursor.execute(
                    """UPDATE checkpoints SET output = %s
                       WHERE run_id = %s AND step_index = %s""",
                    (psycopg2.extras.Json(result), run_id, step_index)
                )
                conn.commit()
                return result
            except Exception:
                conn.rollback()
                raise
        return wrapper
    return decorator

In practice, you’ll want connection pooling (pgbouncer), retry logic, and batch inserts. But the core idea is simple: write before, update after.

The Future: Checkpointing as a Service

We’re building a generic checkpointing service that any agent can use via gRPC. It will support multiple backends (Postgres, S3, etc.) and provide a consistent API for checkpointing, recovery, and replay. The goal is to make action-level checkpointing the default, not an afterthought.

Until then, the pattern is proven. Stop losing runs. Start checkpointing every action.