Deterministic Replay of Agent Swarms: A Clock-Free Event Log

You've seen it: an agent swarm that works in staging, fails in production, and nobody can reproduce it. The logs show a cascade of messages, but the exact ordering is lost. You suspect a race condition, but proving it is a nightmare.

The root cause is that most logging systems rely on wall-clock timestamps. Wall clocks lie. They drift, they jump, and they don't capture causal order. When you have multiple agents running concurrently, wall-clock time is useless for reconstructing the true sequence of events.

In this post, I'll show you a clock-free event log that gives you deterministic replay of any agent swarm. You'll be able to re-run a failed session exactly as it happened, down to the last message, and debug it like it's a single-threaded program.

The Problem: Non-Determinism in Agent Swarms

Agent swarms are inherently non-deterministic. Each agent makes decisions based on its own context, LLM outputs are stochastic, and network latency varies. But the biggest source of non-determinism is the interleaving of events. If two agents send messages to each other, the order in which those messages are processed can change the outcome.

Traditional logging records timestamps like 2025-04-01T13:22:01.123Z. That tells you when something happened on a given machine, but it doesn't tell you which event happened first across machines. If Agent A sends a message at 13:22:01.123 and Agent B sends at 13:22:01.124, you can't be sure that A's message was processed before B's, because clocks aren't synchronized.

Even on a single machine, wall-clock time can go backward (NTP adjustments) or skip forward (leap seconds). And when you have multiple processes, scheduling is unpredictable.

The Solution: A Clock-Free Event Log

Instead of timestamps, use a logical clock that captures causality. The classic approach is Lamport timestamps, but for replay we need something stronger: a total order that is consistent with causality.

Here's the design:

• Every event in the system gets a unique identifier: a monotonically increasing integer, assigned by a central sequencer (or a distributed sequencer with a leader).
• Each event also carries a vector clock (or a DAG of dependencies) to capture causal history.
• The event log is an append-only sequence of events, each with its full payload.

For replay, we don't need the vector clock if we have a total order. But the total order must respect causal order: if event A causes event B, then A must come before B in the log.

Implementing the Event Log

Let's get concrete. We'll use Postgres as the backbone, because it's reliable and easy to reason about. We'll create a table for events:

CREATE TABLE events (
    id BIGSERIAL PRIMARY KEY,
    agent_id TEXT NOT NULL,
    event_type TEXT NOT NULL,
    payload JSONB NOT NULL,
    seq BIGINT NOT NULL UNIQUE,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

The seq column is our logical clock. It's a global sequence that every event gets, regardless of which agent produced it. To get the next sequence number, we use a central service or a Postgres sequence:

CREATE SEQUENCE event_seq;

When an agent wants to log an event, it calls a function that does:

INSERT INTO events (agent_id, event_type, payload, seq)
VALUES ($1, $2, $3, nextval('event_seq'))
RETURNING seq;

But wait: if agents are distributed, they can't all use the same Postgres sequence without a round-trip. That's fine—the overhead is small, and we can batch. Alternatively, use a dedicated sequencer service that hands out ranges.

Capturing Causality

A total order is enough for replay if we ensure that every event is logged before it can influence another event. But what about messages sent between agents? When Agent A sends a message to Agent B, we log the send event, and when B receives it, we log the receive event. The receive event must have a sequence number greater than the send event.

To enforce that, we can have the messaging layer log both events atomically. For example, if using Redis streams or Kafka, we can produce a single event that contains both the send and receive sides. But for simplicity, we can just log the send event and then the receive event, and trust that the sequencer assigns increasing numbers.

But there's a subtlety: if Agent A sends a message and then crashes before the receive event is logged, we still have the send event. That's fine—we can replay the crash.

Replay: Deterministic Execution

Once you have the event log, replaying a session is straightforward. You read the events in order of seq and feed them to a deterministic simulator of your agent swarm.

The key is that your agents must be deterministic given the same sequence of events. That means:

• No reliance on wall-clock time; use event sequence numbers as the only time source.
• No random number generation without a seed; if you need randomness, seed it with the event sequence.
• No network calls; instead, use the event log as the sole source of inputs and outputs.

This is similar to how we test distributed systems with deterministic simulation (e.g., FoundationDB's simulation).

The Agent Runtime

To make replay possible, you need to structure your agents so that all external interactions go through the event log. Here's a minimal runtime in Python:

class AgentRuntime:
    def __init__(self, event_log):
        self.event_log = event_log
        self.state = {}

    def send(self, target_agent, message):
        event = {
            'type': 'send',
            'from': self.agent_id,
            'to': target_agent,
            'message': message,
        }
        self.event_log.append(event)

    def receive(self, event):
        self.state.update(event['payload'])
        # deterministic logic based on event

In production, the event log is backed by Postgres. In replay, you read from a file or a table.

Handling LLM Non-Determinism

One of the biggest challenges is that LLM outputs are non-deterministic. If you're using an LLM to make decisions, the same input can produce different outputs. To make replay work, you need to log the exact LLM response for each call and replay that response instead of calling the LLM again.

Here's how:

• When an agent calls an LLM, log the request and the response as an event.
• During replay, when you encounter that event, you use the logged response instead of calling the API.

This is similar to mocking, but it's automatic because the event log captures everything.

For example, using vLLM or llama.cpp, you can set a seed for deterministic generation, but even then, you might have floating-point differences. So logging is the safest bet.

Case Study: Debugging a Flaky Swarm

Let me give you a real scenario. We had a swarm of 5 agents that were supposed to coordinate on a task. Every few runs, one agent would send a message that another agent would ignore, leading to a deadlock.

We added the event log, and after a failure, we replayed the session. We saw that the deadlock was caused by a race: Agent B sent a 'task_complete' message to Agent C, but Agent C had already moved on to a different state because it processed a 'timeout' event that occurred before the message arrived.

With the event log, we could see the exact order: the timeout event had a lower sequence number than the task_complete event. That meant the timeout was processed first, and the message was ignored because of a state check.

The fix was to change the agent logic to accept late messages. We verified by replaying the same event log after the fix, and the behavior was correct.

Tooling: Building the Log

You can build this with off-the-shelf components. Here's a stack:

• Postgres for the event log. Use BIGSERIAL or a sequence for the logical clock.
• pgbouncer for connection pooling, because agents will be writing frequently.
• pgvector is optional if you need to store embeddings for retrieval.
• Redis or Kafka for messaging, but you'll log every message as an event.
• systemd to run your agents as services, with graceful shutdown on failure.

I recommend using a simple REST API or a message queue to append events. For performance, you can batch inserts.

What About Distributed Sequencers?

If you have a multi-region swarm, a single Postgres sequence might be a bottleneck. You can use a distributed sequencer like a leader-based service that hands out ranges. Each agent gets a block of sequence numbers and uses them locally, then flushes to the log.

But for most self-hosted setups, a single Postgres is fine. You can handle hundreds of events per second without breaking a sweat.

Testing: Property-Based Replay

Once you have deterministic replay, you can do property-based testing. Generate random event sequences, run the swarm, and assert that safety properties hold (e.g., no deadlocks, all messages eventually processed).

Because replay is deterministic, you can run the same sequence multiple times and get the same result. That's the foundation of a solid test suite.

I like to use Hypothesis for property-based testing in Python. You can generate event sequences and check invariants.

The Clock-Free Principle

The core idea is to eliminate wall-clock time from your system's observable behavior. Instead of time.sleep() or datetime.now(), use event sequence numbers. If you need a timeout, implement it as a logical timer that fires after a certain number of events, not after a certain number of milliseconds.

This might sound radical, but it's how distributed systems are tested. It forces you to reason about causality rather than time, and it makes your system more robust because it doesn't depend on timing assumptions.

Integrating with Existing Agents

You don't have to rewrite your agents from scratch. You can wrap them with a thin layer that intercepts all I/O and logs it. For example, if your agent uses requests to call an external API, you can monkey-patch it to log the request and response.

But for a clean design, I recommend structuring agents as pure functions of their inputs. The event log becomes the only input.

Performance Overhead

There's a cost to logging every event. But it's usually acceptable. For a swarm of 10 agents, you might generate a few thousand events per second. Postgres can handle that easily.

If you need higher throughput, you can use a more efficient log like Apache Kafka, but that adds complexity. Start with Postgres and see if it's enough.

Conclusion

Deterministic replay is not a luxury; it's a necessity for debugging agent swarms. By using a clock-free event log based on a logical sequence, you can turn non-deterministic chaos into a reproducible sequence of events. You can replay failures, test edge cases, and build confidence in your system.

I've used this approach in production with Postgres as the backbone, and it's been a game-changer. The next time your swarm misbehaves, you'll be able to reproduce it exactly and fix it in minutes, not days.

Stop chasing ghosts. Start logging events without clocks.