fsync on Every Turn: The Real Cost of Durable Agent Memory
Why your agent's memory might not survive a power cut, and what to do about it.
When you build an agent that remembers, you eventually hit a wall: the database says the memory is saved, but is it actually on disk? The gap between "the write returned" and "the bytes are safe" is the difference between a system that survives a power cut and one that loses the last few turns of conversation. This article is about that gap, and why fsync is the price of admission for durable agent memory.
Agents are stateful by nature. They accumulate context, user preferences, tool results, and intermediate reasoning. If that state lives only in RAM, a crash resets the agent to a blank slate. For many applications that is unacceptable. The fix is to persist state to a database, but persistence is not binary. There is a spectrum from "the OS will eventually write this" to "the bytes are on the physical medium right now." The latter requires fsync.
What fsync Actually Does
When your application calls write() on a file descriptor, the data goes into the kernel's page cache. The kernel will flush that cache to disk at some point, but "some point" is not guaranteed to be before the next power failure. fsync() forces the kernel to push all dirty pages for that file to the storage device and wait until the device confirms the write has reached non-volatile storage.
In PostgreSQL, the equivalent is synchronous_commit = on (the default). Every transaction commit waits for a WAL flush, which is an fsync on the WAL file. If you set synchronous_commit = off, the commit returns before the WAL is flushed, but you risk losing the last few transactions on crash.
For an agent memory store, the choice is stark: either every turn's memory write is durable (fsync per commit) or it is not. There is no middle ground that gives you both durability and zero latency.
The Latency Cost of fsync
The latency of an fsync is dominated by the physical characteristics of the storage device. On a spinning disk, an fsync can take tens of milliseconds because the disk must physically position the head. On a typical SSD, an fsync is faster, often in the low hundreds of microseconds to a few milliseconds, but it still involves a full device round-trip and a flush of the drive's internal cache.
That means if you are doing one fsync per agent turn, and each turn involves one or more database writes, you are adding that latency to the critical path of every turn. For a conversational agent, a few milliseconds might be acceptable. But when you start scaling to many concurrent agents, or when each turn involves multiple writes (e.g., updating state, inserting a message, updating a vector embedding), the cumulative cost can dominate.
A common pattern in agent systems is to write to multiple stores: a relational database for structured state, a vector database for embeddings, and perhaps a graph database for relationships. If each of those commits independently with fsync, you are paying the cost multiple times per turn.
The batching Illusion
One tempting optimization is to batch multiple writes into a single transaction, so you pay fsync once per batch instead of once per write. That works if the agent's turn naturally produces a batch of writes. But agents are often interactive: they produce one response, then the user replies, and the next turn starts. There is no natural batch boundary unless you deliberately delay the response.
Delaying the response to batch writes is a classic tradeoff. You could buffer memory updates for, say, 100 milliseconds, then flush them all with one fsync. This reduces the per-turn cost, but it introduces latency to the user's perceived response time and, more importantly, it creates a window where a crash loses the last few turns. If that is acceptable, batching is a valid design choice. If not, you are back to per-turn fsync.
Another approach is to use a write-ahead log (WAL) that is separate from the main data store. The agent writes its memory updates to a WAL, which is fsync'd, but the main store is updated asynchronously. On recovery, you replay the WAL. This is essentially what PostgreSQL does internally, and you can leverage it by using PostgreSQL as your memory store. The WAL is written once per transaction, and the main data files are flushed later. That gives you durability with a single fsync per transaction, regardless of how many rows you update.
How RiNET Handles Durable Agent Memory
At RiNET, we run a stack that includes three Hetzner servers connected via a WireGuard mesh. Our agent memory is stored in PostgreSQL, with Qdrant for vector search and Neo4j for graph relationships. The key insight is that we do not fsync on every turn. Instead, we use PostgreSQL's default synchronous_commit = on, which means every transaction commits with an fsync. But we structure our writes so that each turn produces a single transaction.
For example, when an agent processes a turn, we update the conversation state, insert the user message, and insert the assistant response in one transaction. That way, we pay the fsync cost once per turn, not once per write. The vector embedding is written to Qdrant separately, but Qdrant has its own durability settings. We accept that the vector write might be slightly less durable, because losing an embedding is less catastrophic than losing the conversation state.
We also run nightly LoRA fine-tuning on our Qwen models, which is a batch job that does not need per-turn durability. That allows us to use a more relaxed durability setting for that workload.
The result is a system where the per-turn latency is dominated by the PostgreSQL commit, which on our NVMe SSDs is typically a few milliseconds. That is acceptable for our conversational agents. But we had to design for it from the start. If we had naively written to multiple stores with individual commits, we would be paying three fsyncs per turn, and the latency would be three times higher.
When fsync per Turn Is the Wrong Choice
There are scenarios where per-turn fsync is overkill. If your agent is not interactive, but processes a queue of tasks, you can batch memory updates across tasks. If your agent's memory is not critical—for example, it is a cache that can be rebuilt—you can skip durability entirely.
A common pattern is to use an in-memory store with periodic snapshots. The agent runs with fast, non-durable memory, and every N minutes, it snapshots the state to disk. On crash, you lose at most N minutes of memory. That is a tradeoff that many systems accept.
Another pattern is to use a message queue as the durable log. The agent writes its memory updates to a queue (e.g., Kafka, RabbitMQ) which is fsync'd by the queue's producer. The actual memory store is updated asynchronously by a consumer. This decouples durability from the main write path, but it adds complexity and eventual consistency.
Measuring the Real Cost
Without a benchmark, it is hard to give exact numbers, but you can measure the cost in your own environment. The key metric is the latency of a single fsync() call on your storage device. You can measure it with a simple test: write a small file, call fsync(), and time it. Repeat many times to get a distribution. On a typical SSD, you will see a median around a few hundred microseconds, with a tail that can be several milliseconds.
The next step is to measure the end-to-end latency of a database commit. In PostgreSQL, you can use pg_stat_statements to see the average and max duration of commit statements. Or you can instrument your application to time the commit call.
Once you have those numbers, you can model the impact on your agent's turn latency. If your turn involves one commit, the added latency is roughly the commit time. If it involves two, it's twice that, unless you batch them into one transaction.
Designing for Durability Without Crippling Throughput
Here are concrete design principles for durable agent memory without sacrificing performance:
Use a single transaction per turn. Group all relational writes into one transaction. This gives you one fsync per turn.
Leverage the database's WAL. PostgreSQL's WAL is your friend. It ensures durability with a single fsync per transaction, regardless of the number of rows.
Separate durability levels. Not all memory is created equal. Conversation state is critical; vector embeddings are less so. Configure your stores accordingly.
Batch where possible. If your agent processes turns in bursts, batch multiple turns into a single transaction. This reduces the fsync frequency, but be aware of the crash window.
Consider a WAL-like pattern. If you cannot batch, write to a durable log first, then update your main store asynchronously.
Monitor your commit latency. Use tools like
pg_stat_statementsto track commit times. If they start to degrade, your storage might be the bottleneck.
The Bottom Line
fsync per turn is the price of durable agent memory. It is not free, but it is manageable if you design for it. The worst thing you can do is ignore durability and then discover your agent forgot everything after a power cut. The second worst thing is to fsync every write without batching, and then wonder why your agent is slow.
The right answer depends on your application's requirements. If you need absolute durability, accept the latency and optimize your transaction boundaries. If you can tolerate some loss, use a snapshot or async pattern. But make the decision consciously, and measure the cost.
At RiNET, we chose per-turn durability for our conversational agents, and we paid the cost in design effort. The result is a system that survives power cuts without losing the thread of a conversation. That is worth a few milliseconds per turn.