How We Recovered a 14-Hour Agent Swarm After 13 Minutes of Poisoned Data

We run a multi-agent orchestration pipeline that ingests financial filings, embeds them with BGE-M3, and stores vectors in Qdrant for downstream retrieval-augmented generation (RAG). A typical run spans 14 hours, processing 50,000+ documents across 24 agents. One morning, a single corrupted PDF—a malformed XBRL file—poisoned the vector store at minute 13. The contamination cascaded: every subsequent agent that queried that vector got a garbage response, which it then fed back into the swarm. By minute 45, the entire pipeline was hallucinating nonsense.

We caught it at minute 47 via a monitoring alert that flagged an anomalous spike in cosine similarity scores. But the damage was done: 34 minutes of poisoned state, affecting 12 agents and 1,200+ documents. The obvious fix? Kill the run, purge the vector store, and restart from scratch. That’s another 14 hours. Not acceptable for a production pipeline with SLAs.

Instead, we did a surgical recovery using checkpoint injection and partial rollback. Here’s exactly how.

The Poisoning Mechanism

Our swarm uses a shared Qdrant collection as the persistent memory layer. Each agent writes its output (a document chunk + embedding) to the collection, and reads from it to inform subsequent steps. The corrupted PDF produced an embedding vector that was nearly orthogonal to any legitimate financial text—cosine similarity < 0.1 with its nearest neighbor. But because our ingestion pipeline doesn’t validate embedding quality (a mistake we’ve since fixed), that vector was stored and indexed.

When the next agent queried for “revenue recognition policies,” the poisoned vector was the top-1 result (because its magnitude was oddly high). The agent then used that garbage to generate a summary, which it wrote back to the collection. That summary’s embedding was also garbage, and so on. Classic cascading failure.

Recovery Strategy: Checkpoint Injection

We run each agent as a systemd service with a pre- and post-checkpoint hook. Every 5 minutes, each agent writes its current state (in-memory queues, cursor positions, and a snapshot of the Qdrant collection ID) to a PostgreSQL table using pgvector for fast similarity search of checkpoints. Yes, we use Postgres as the control plane. The checkpoint table looks like this:

CREATE TABLE agent_checkpoints (
    id BIGSERIAL PRIMARY KEY,
    agent_name TEXT NOT NULL,
    run_id UUID NOT NULL,
    checkpoint_time TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    state JSONB,
    collection_snapshot_id TEXT,
    parent_checkpoint_id BIGINT REFERENCES agent_checkpoints(id)
);

CREATE INDEX idx_checkpoints_run_time ON agent_checkpoints(run_id, checkpoint_time);

When the poisoning was detected, we had a clean checkpoint for every agent at minute 10 (three minutes before the poison). But we couldn’t just restore all agents to minute 10 and replay—that would lose 37 minutes of valid work done by agents that never touched the poison. We needed to selectively roll back only the poisoned agents.

Identifying the Poisoned Agents

We traced the contamination using a lineage query. Each agent’s output includes a source_ids array—the Qdrant point IDs it retrieved. We wrote a quick script to find all points that were inserted after minute 13 and had a cosine similarity < 0.2 with any known-good financial embedding (we keep a reference set of 10,000 embeddings from a prior clean run). Those points were flagged as corrupted. Then we found all agents that had retrieved at least one corrupted point. That gave us 12 agents.

Partial Rollback and Replay

For each poisoned agent, we did the following:

1. Roll back state to the last clean checkpoint. We restored the agent’s in-memory state from the state JSONB column at the checkpoint just before the poison (minute 10 for most, minute 12 for one agent that was slower).

2. Delete corrupted points from Qdrant. We used the Qdrant REST API to delete points by ID. The IDs were collected during the lineage query.

3. Replay the agent from the checkpoint. The agent resumed processing from its cursor position, but this time the vector store was clean. However, we had to ensure that agents that depended on outputs from poisoned agents (which we had just deleted) could still function. Those dependencies were resolved by re-running the poisoned agents’ outputs—but only the ones that were actually consumed downstream.

To avoid a full replay, we used a dependency graph built from the checkpoint lineage. Each checkpoint record has a parent_checkpoint_id that points to the checkpoint of the agent that produced the input. We traversed the graph from the poisoned nodes upward, marking only the minimal set of agents that needed re-execution. In our case, that was 5 agents out of 12—the others had outputs that were never consumed (e.g., intermediate summaries that were later overwritten).

The Recovery Script

We automated the rollback with a Python script that used psycopg2 for Postgres and qdrant_client for Qdrant. Here’s the core logic:

import psycopg2
from qdrant_client import QdrantClient

conn = psycopg2.connect("dbname=control_plane")
cur = conn.cursor()
qdrant = QdrantClient(host="localhost", port=6333)

# Step 1: Find poisoned points
cur.execute("""
    SELECT point_id FROM poisoned_points WHERE run_id = %s
""", (run_id,))
poisoned_ids = [row[0] for row in cur.fetchall()]

# Step 2: Delete from Qdrant
qdrant.delete_points(
    collection_name="financial_embeddings",
    points=poisoned_ids
)

# Step 3: Roll back each poisoned agent
cur.execute("""
    SELECT agent_name, state, collection_snapshot_id
    FROM agent_checkpoints
    WHERE run_id = %s AND checkpoint_time <= %s
    ORDER BY checkpoint_time DESC
    LIMIT 1
""", (run_id, poison_time))
for agent_name, state, snapshot_id in cur.fetchall():
    # Restore agent state (pseudo)
    restore_agent_state(agent_name, state)
    # Optionally restore Qdrant snapshot (we didn't need to here)

We then triggered the replay for the minimal set of agents via systemd service restart with a --resume-from-checkpoint flag.

Results and Lessons

The recovery took 23 minutes: 5 minutes to identify poisoned agents, 3 minutes to delete points, 15 minutes to replay the 5 agents. Total downtime: 23 minutes vs. 14 hours. The pipeline finished 13.5 hours after the original start time—only 23 minutes late.

Key lessons:

• Always validate embedding quality at ingestion. We now run a pre-check: if an embedding’s max cosine similarity to any known-good reference embedding is below 0.3, we reject it and log the document for manual review. This would have caught the poisoned PDF at minute 13.
• Checkpoint every 5 minutes. That granularity was fine for our throughput. If your agents are faster, checkpoint more often. The overhead of writing JSONB to Postgres is negligible compared to embedding computation.
• Dependency graphs save replay time. Without the lineage query, we would have replayed all 12 poisoned agents. The graph cut that to 5.
• Don’t trust the vector store implicitly. A single bad vector can corrupt an entire swarm. Treat your vector store as a shared mutable state that needs transactional integrity.

What We Changed Post-Mortem

1. Embedding validation filter – a BGE-M3 model wrapper that checks embedding quality before insertion.
2. Automatic checkpoint lineage – every agent now logs its parent_checkpoint_id automatically. No manual tracing needed.
3. Rollback button in the monitoring dashboard – one click to roll back the last N minutes for a given run.
4. Qdrant snapshots – we now take a snapshot of the collection every 10 minutes. That’s a safety net if the point deletion approach fails.

This was a wake-up call. Agent swarms are powerful, but they’re also brittle. Without checkpoint injection and partial rollback, a 13-minute poison would have cost us a full day. Now we can recover in minutes.

Tools used: Postgres 15 with pgvector 0.5.1, Qdrant 1.7.3, BGE-M3 (via llama.cpp), systemd 252, Python 3.11 with qdrant-client 1.7.0 and psycopg2 2.9.9.