Designing a Self-Healing Control Plane for Agent Swarms

Agent swarms are fragile. When a node goes down, you lose not just the agent processes but their accumulated context — conversation history, tool call results, intermediate reasoning steps. Most teams paper over this with external databases for long-term memory, but the control plane itself (the orchestrator that decides which agent does what) is often a single point of failure. This article shows how to build a control plane that survives crashes and restarts agents with full context, using Postgres as the source of truth and deterministic state machines for recovery.

Why Context Loss Happens

A typical agent swarm runs as a set of long-lived processes. Each agent holds in-memory state: a list of messages, a stack of function calls, a pointer to the current step in a plan. If the orchestrator crashes, that state disappears. You can persist to Redis or a file, but if the crash happens mid-write, you get corruption. The only way to guarantee recovery is to make every state transition durable before acting on it.

The Postgres-As-Control-Plane Pattern

Instead of storing state in the agent process, we push it into Postgres. Each agent is a lightweight worker that reads its next task from the database, executes it, and writes the result back. The control plane is just a set of SQL queries and a supervisor process that polls for zombie agents.

Schema Design

We need three tables: agents, tasks, and context.

CREATE TABLE agents (
    id UUID PRIMARY KEY,
    swarm_id UUID NOT NULL,
    status TEXT NOT NULL DEFAULT 'idle', -- idle, running, crashed, completed
    current_task_id UUID,
    heartbeat_at TIMESTAMPTZ,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE tasks (
    id UUID PRIMARY KEY,
    agent_id UUID REFERENCES agents(id),
    type TEXT NOT NULL, -- 'reason', 'tool_call', 'respond'
    input JSONB,
    output JSONB,
    status TEXT NOT NULL DEFAULT 'pending', -- pending, running, done, failed
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE context (
    agent_id UUID REFERENCES agents(id),
    key TEXT,
    value JSONB,
    PRIMARY KEY (agent_id, key)
);

Each agent writes its heartbeat every 5 seconds. The supervisor runs a cron job that checks for agents whose heartbeat_at is older than 15 seconds. Those are considered crashed.

Deterministic Recovery with State Machines

When an agent crashes, it may have been in the middle of a task. We cannot simply restart it and hope for the best — we need to know exactly where it left off. The solution is to model each agent's lifecycle as a state machine with persistence.

State Transitions

Each task goes through: pending -> running -> done or failed. The agent writes the transition to tasks.updated_at and tasks.status before starting execution. If it crashes after marking running but before writing output, the supervisor sees a running task with old updated_at and can reset it to pending.

But what about the agent's internal context? For example, an agent that was in the middle of a multi-step reasoning chain has a stack of intermediate thoughts. We persist that stack in the context table every time it changes.

-- Agent writes context before each step
INSERT INTO context (agent_id, key, value)
VALUES ($1, 'reasoning_stack', $2)
ON CONFLICT (agent_id, key)
DO UPDATE SET value = EXCLUDED.value;

On restart, the agent reads its context from Postgres and reconstructs its state. No in-memory state is trusted.

Supervisor Implementation

The supervisor is a simple Go or Python process that runs as a systemd service. It does two things:

1. Heartbeat monitoring: Every 10 seconds, query for agents with stale heartbeats.
2. Task recovery: For each crashed agent, find its running tasks and reset them to pending. Then spawn a new agent process that picks up the pending tasks.

# supervisor.py (simplified)
import psycopg2
import subprocess
import time

conn = psycopg2.connect("dbname=swarm")

def recover_crashed_agents():
    with conn.cursor() as cur:
        cur.execute("""
            UPDATE tasks
            SET status = 'pending', updated_at = NOW()
            WHERE agent_id IN (
                SELECT id FROM agents
                WHERE heartbeat_at < NOW() - INTERVAL '15 seconds'
                AND status = 'running'
            )
            AND status = 'running';
        """)
        cur.execute("""
            UPDATE agents
            SET status = 'crashed'
            WHERE heartbeat_at < NOW() - INTERVAL '15 seconds'
            AND status = 'running';
        """)
        conn.commit()
        # Spawn new agents for crashed ones
        cur.execute("""
            SELECT id FROM agents WHERE status = 'crashed';
        """)
        for row in cur.fetchall():
            agent_id = row[0]
            subprocess.Popen(["python", "agent_worker.py", str(agent_id)])
            cur.execute("UPDATE agents SET status = 'idle' WHERE id = %s;", (agent_id,))
        conn.commit()

while True:
    recover_crashed_agents()
    time.sleep(10)

This supervisor is stateless itself — it relies entirely on Postgres. If it crashes, systemd restarts it, and it picks up where it left off.

Handling Long-Running Tasks

Some tasks take minutes (e.g., running a large model inference). If the agent crashes during inference, the model call might be orphaned. To handle this, we wrap each external call in a transaction with a timeout. If the agent doesn't report back within the expected time, the supervisor cancels the task and resets it.

For LLM inference with vLLM or llama.cpp, we can use a separate queue. The agent submits a request to the queue and polls for completion. If the agent crashes, the supervisor can reassign the request to another agent by updating the task's agent_id.

Context Persistence Strategy

Not every piece of context needs to be persisted on every change. We use a write-ahead log approach: before executing any action that modifies the agent's state, we write the new state to the context table. If the write succeeds, we proceed. If it fails, we retry. This ensures that the agent's state is always consistent with what's in the database.

We also batch context writes to reduce overhead. For example, we collect all context changes during a reasoning step and flush them in a single transaction.

# agent_worker.py (context write batching)
def flush_context(agent_id, context_updates):
    with conn.cursor() as cur:
        for key, value in context_updates.items():
            cur.execute("""
                INSERT INTO context (agent_id, key, value)
                VALUES (%s, %s, %s)
                ON CONFLICT (agent_id, key)
                DO UPDATE SET value = EXCLUDED.value;
            """, (agent_id, key, json.dumps(value)))
        conn.commit()

Testing the Recovery

To validate, we kill agents randomly and check that tasks complete eventually. We also simulate a supervisor crash by killing the supervisor process and verifying that systemd restarts it and recovery proceeds.

# Kill all agent processes
pkill -f agent_worker.py
# Wait 20 seconds, then check tasks
psql -c "SELECT status, count(*) FROM tasks GROUP BY status;"
# Should see all 'running' tasks become 'pending' or 'done'

Trade-offs

This design trades latency for reliability. Every state transition requires a database round-trip. For most agent swarms, this is acceptable — human interaction times are seconds, and a 50ms database write is negligible. For high-frequency trading bots, you'd want something faster.

Also, Postgres becomes a bottleneck. Use pgbouncer for connection pooling and consider read replicas for monitoring queries. The supervisor's heartbeat check can use a replica to avoid loading the primary.

Real-World Example

We run a swarm of 50 agents that do customer support triage. Each agent handles a conversation with a user. If the server hosting the agents crashes, the supervisor (running on a separate machine) detects the missing heartbeats, resets all running tasks to pending, and spawns new agents on a backup server. The new agents read the conversation history from the context table and resume exactly where the old agents left off. Users see a brief pause, but the conversation continues seamlessly.

Conclusion

A self-healing control plane is not magic. It's a matter of making every state transition durable and having a stateless supervisor that can reconstruct state from the database. Postgres is the perfect foundation — it's battle-tested, transactional, and supports JSONB for flexible context storage. With a few tables, a heartbeat loop, and deterministic state machines, you can build a control plane that survives crashes without losing a single message.

No fluff. No marketing. Just a pattern that works.