In-Process KV with WAL: Rethinking Agent State Sync

When you run a swarm of autonomous agents, the hardest part is rarely the model. It's the state. Each agent carries a bag of variables—current goal, conversation history, tool call results, partial plans—that must stay consistent across retries, restarts, and concurrent actions. For a long time, the default answer was to push that state into Redis: fast, familiar, and network-accessible. But as our agent workloads grew, we started to question that choice. The network hop, the serialization overhead, the operational burden of keeping Redis alive and tuned—all of it felt like overhead for what is fundamentally a local problem. So we experimented with moving state into an in-process key-value store with a write-ahead log (WAL). The results were surprising: not only did we simplify our architecture, but we also removed a whole class of failure modes. This article walks through the reasoning, the trade-offs, and a reference implementation that you can adapt to your own swarm.

The Problem with Redis for Agent State

Redis is a fantastic piece of software. It's fast, battle-tested, and supports a rich set of data structures. But for agent state, it has several inherent drawbacks that become more pronounced as your swarm grows.

First, the network round-trip. Every read and write to Redis crosses the network. Even on a loopback interface, that's a few microseconds; on a real network, it's tens to hundreds of microseconds. When an agent needs to update its state on every action—which can happen multiple times per second—those microseconds add up. More importantly, they introduce jitter. A burst of traffic on the network can cause a state write to take 10x longer than usual, which can stall an agent's decision loop.

Second, serialization overhead. Redis stores values as byte strings. To store a complex agent state object, you need to serialize it—typically to JSON—on every write, and deserialize on every read. That's CPU work that happens twice per state access. With a local store, you can keep the object in memory and only serialize when you need to persist it to disk.

Third, operational complexity. Redis is a separate service that needs to be deployed, monitored, backed up, and secured. For a small swarm, that's a lot of moving parts. If Redis goes down, your agents lose their state and may crash or behave erratically. You need to handle connection retries, timeouts, and reconnects in your agent code. All of that is complexity that doesn't exist if the state lives in the same process as the agent.

Fourth, the consistency model. Redis, especially in a single-instance setup, gives you strong consistency for individual keys. But if you need to update multiple keys atomically—for example, when an agent transitions from one state to another—you need transactions (MULTI/EXEC) or Lua scripts. That's doable, but it adds complexity. In an in-process store, you can use a simple mutex to make any sequence of operations atomic.

None of these are deal-breakers for every use case. Redis is still excellent for shared caches, pub/sub, and cross-process coordination. But for agent state, which is often local to a single agent or a small group of agents, the overhead is disproportionate.

The In-Process Alternative

An in-process KV store is exactly what it sounds like: a key-value map that lives inside your agent's process. You access it directly, without any network or serialization overhead. To make it durable, you add a write-ahead log: every mutation is first appended to an append-only file, then applied to the in-memory map. If the process crashes, you can replay the log to restore the state.

This pattern is well-known in the database world—it's how systems like SQLite and LevelDB achieve durability. But applying it to agent state is a natural fit, because agents are typically long-running processes that need to survive restarts.

The key insight is that agent state is not a shared resource. It's owned by the agent. Other components may need to read it, but they can do so via a query interface that runs in the same process. There's no need for cross-process concurrency control because there's only one writer: the agent itself.

Trade-offs and When to Use Which

Before diving into code, let's be honest about the trade-offs. The in-process approach is not a universal replacement for Redis. It's a tool for a specific job.

Pros:

• Zero network latency. State access is a memory read/write.
• No serialization overhead. You store objects directly.
• Simplified ops. No separate service to manage.
• Strong consistency. A single mutex gives you atomicity for composite operations.
• Lower resource usage. No need for a separate Redis process.

Cons:

• Limited to a single process. If you need multiple processes to share state, you're out of luck.
• No built-in replication. Redis has master-slave replication; your in-process store doesn't.
• Manual recovery. You have to implement log replay and compaction.
• Memory bound. Your state must fit in memory (which is usually fine for agent state).

When to use Redis:

• When state needs to be shared across multiple processes or machines.
• When you need pub/sub or other Redis-specific features.
• When you have a large state that doesn't fit in memory.
• When you need a battle-tested solution and don't want to build your own.

When to use an in-process store:

• When state is local to a single agent or a small group of agents.
• When you want to minimize latency and jitter.
• When you want to simplify your architecture and reduce moving parts.
• When you're running a swarm on a single machine (or a few machines with one agent per process).

In our case, we run a swarm of agents on three dedicated servers. Each agent runs as a separate process. The state of each agent is only accessed by that agent and occasionally by a supervisor process that runs on the same machine. There was no need for cross-machine state sharing. So we moved to an in-process store and never looked back.

Reference Implementation

Let's build a minimal in-process KV store with a WAL in Python. We'll use json for serialization and threading for locking. This is a reference implementation—you can adapt it to your language and needs.

import json
import os
import threading
from pathlib import Path

class KVStore:
    def __init__(self, path: str):
        self.path = Path(path)
        self.lock = threading.RLock()
        self.data = {}
        self.wal_fd = None
        self._load()
        self._open_wal()

    def _load(self):
        # Load initial data from a snapshot if it exists
        snapshot = self.path.with_suffix('.snapshot')
        if snapshot.exists():
            with open(snapshot, 'r') as f:
                self.data = json.load(f)
        # Replay WAL
        if self.path.exists():
            with open(self.path, 'r') as f:
                for line in f:
                    if not line.strip():
                        continue
                    record = json.loads(line)
                    if record['op'] == 'set':
                        self.data[record['key']] = record['value']
                    elif record['op'] == 'delete':
                        self.data.pop(record['key'], None)

    def _open_wal(self):
        self.wal_fd = open(self.path, 'a')

    def _append(self, record: dict):
        line = json.dumps(record) + '\n'
        self.wal_fd.write(line)
        self.wal_fd.flush()
        os.fsync(self.wal_fd.fileno())

    def set(self, key: str, value):
        with self.lock:
            self.data[key] = value
            self._append({'op': 'set', 'key': key, 'value': value})

    def get(self, key: str):
        with self.lock:
            return self.data.get(key)

    def delete(self, key: str):
        with self.lock:
            if key in self.data:
                del self.data[key]
                self._append({'op': 'delete', 'key': key})

    def snapshot(self):
        with self.lock:
            snapshot_path = self.path.with_suffix('.snapshot')
            tmp = snapshot_path.with_suffix('.tmp')
            with open(tmp, 'w') as f:
                json.dump(self.data, f)
                f.flush()
                os.fsync(f.fileno())
            os.replace(tmp, snapshot_path)
            # Truncate WAL
            self.wal_fd.close()
            self.path.unlink()
            self._open_wal()

    def close(self):
        if self.wal_fd:
            self.wal_fd.close()

This store does three things: it keeps the state in a Python dict, it appends every mutation to a WAL file, and it can snapshot the state to a compact file. On startup, it loads the snapshot and replays the WAL to reconstruct the state.

The fsync on every write ensures durability at the cost of performance. If you need higher write throughput, you can batch fsyncs or make it configurable. For agent state, durability is usually not critical—if the process crashes, you can often reconstruct the state from the agent's conversation history or other sources. So you might want to skip the fsync and rely on the OS buffer.

Integrating with an Agent Loop

Now let's see how you'd use this in an agent loop. Suppose you have an agent that maintains a goal, a plan, and a history. You can store all of that under a single key, or split it into multiple keys. Here's a simplified example:

import time
from kvstore import KVStore

class Agent:
    def __init__(self, agent_id: str, store: KVStore):
        self.id = agent_id
        self.store = store
        self.state_key = f'agent:{agent_id}'
        self.state = self.store.get(self.state_key) or {'goal': None, 'plan': [], 'history': []}

    def update_goal(self, new_goal: str):
        self.state['goal'] = new_goal
        self.store.set(self.state_key, self.state)

    def add_plan_step(self, step: str):
        self.state['plan'].append(step)
        self.store.set(self.state_key, self.state)

    def run(self):
        while True:
            # ... do some work ...
            self.update_goal('fetch data')
            self.add_plan_step('query API')
            time.sleep(1)

# Usage
store = KVStore('/var/lib/agent-state/wal.log')
agent = Agent('agent-1', store)
agent.run()

In this example, the agent loads its state at startup, modifies it, and persists it with each change. The WAL ensures that if the process crashes, the state is not lost (up to the last fsync).

Handling Concurrency and Atomicity

One of the advantages of an in-process store is that you can easily make composite operations atomic. In Redis, you'd need a transaction or Lua script. Here, you can just hold the lock:

with store.lock:
    # read-modify-write
    state = store.get(key)
    state['counter'] += 1
    store.set(key, state)

But in practice, if only one agent writes to its own state, you might not even need the lock. In our case, we have a supervisor that occasionally reads the state of all agents. That read is thread-safe because the GIL protects individual dict operations, but for consistency you might want to use a lock.

Beyond the Basics: Compaction and Recovery

Over time, the WAL grows unboundedly. To keep it from eating disk space, you need to compact it. The snapshot method above does that: it writes the entire state to a snapshot file and truncates the WAL. You can trigger this periodically, or when the WAL reaches a certain size.

Recovery is straightforward: load the snapshot, replay the WAL. But what if the WAL is corrupted? A common technique is to use a checksum per record. If a record fails to validate, you can stop replay and discard the rest. For agent state, losing the tail is often acceptable.

Real-World Lessons

We've been running this pattern for a while, and a few things stand out:

• The simplicity is liberating. No Redis connection management, no timeouts, no reconnection logic. The state is just there.
• Latency is predictable. There's no network jitter. The only variable is disk I/O on the WAL append, which we've made asynchronous with a background thread that batches writes.
• You need to be disciplined about state size. If you store large objects, the WAL grows quickly. We keep each agent's state under a few hundred kilobytes.
• Snapshots are your friend. We snapshot every hour and after each major agent transition. This keeps the WAL small and recovery fast.

One thing we didn't anticipate was how much easier it became to debug. With Redis, you'd have to connect to the Redis CLI and inspect keys. Now, we can just dump the state file to see what an agent was thinking.

Alternatives and Variations

If you don't want to roll your own, there are existing libraries. SQLite is a great choice: it's a battle-tested, single-file database with WAL mode built in. You can use it as a KV store with a simple table. The overhead is slightly higher than a custom in-memory map, but it gives you ACID transactions and a query language.

Another option is to use a library like diskcache (Python) which provides a disk-backed dict with a WAL. Or you can use a more sophisticated embedded database like RocksDB, which is designed for high write throughput.

In our stack, we actually use Postgres for permanent data, Qdrant for vector search, and Neo4j for graph relationships. But for ephemeral agent state, the in-process store is the right tool.

When to Revisit Redis

There are signs that you've outgrown the in-process store:

• Multiple processes need to share the same state in real-time.
• You need to scale horizontally and have agents on different machines that must coordinate.
• You need pub/sub to notify other components of state changes.
• You need to query the state with complex patterns (e.g., find all agents with a certain goal).

In those cases, Redis is a reasonable choice. But for the common case—one agent, one process, one machine—the in-process store is simpler, faster, and easier to reason about.

Conclusion

We didn't ditch Redis because Redis is bad. We ditched it because it was the wrong tool for the job. Agent state is local, ephemeral, and owned by a single process. An in-process KV store with a WAL provides exactly the right level of durability and performance, with zero operational overhead. If you're building an agent swarm, consider whether you really need a network service to hold your agents' memories. Often, a file on disk is enough.