Ext4 vs XFS for Agent Memory Journals: A 90-Day Corruption Study

When you run an autonomous agent swarm, the memory journal is the single most critical piece of state. It records every decision, every tool call, every retrieved context. Lose it, and the agent loses its identity. Lose it partially, and the agent starts hallucinating its own history. So the filesystem that backs that journal is not a minor detail — it's a durability decision.

Over the past few years, teams building long-running agent systems have gravitated toward append-only JSONL or SQLite-backed journals. Both are simple, but both rely on the underlying filesystem to honor fsync and to survive power loss without silent corruption. The two mainstream contenders on Linux are ext4 and XFS. Both are battle-tested, but they take very different approaches to metadata journaling, allocation, and recovery.

This article is about a 90-day corruption benchmark I ran on NVMe drives, comparing ext4 and XFS under repeated crash-recovery cycles. The goal was not to measure throughput — that's been done to death — but to answer a specific question: which filesystem is more trustworthy for an append-only journal that must survive sudden power cuts?

Why the Journal Matters

An agent memory journal is not a database. It's an event log. Each entry is a JSON object with a timestamp, an action, a result, and a pointer to the embedding in Qdrant or the graph node in Neo4j. The journal is the source of truth for the agent's short-term memory, and it's replayed on startup to reconstruct the conversation state.

If a journal entry is truncated or corrupted, the agent may fail to resume, or worse, resume with a false memory. That's not just a bug — it's a safety issue. An agent that thinks it already called a destructive API when it didn't could cause real damage.

The filesystem's job is to ensure that once we write a journal entry and call fsync, that entry is either fully present or fully absent after a crash. Partial writes are not acceptable. Both ext4 and XFS have mechanisms for this, but they differ in how they handle the journal itself.

The Test Setup

I used three identical NVMe drives (consumer-grade, 1TB) on a single server. The server ran Ubuntu 22.04 LTS with kernel 5.15. Each drive was partitioned and formatted once — one with ext4, one with XFS, and one as a control with ext4 but with the journal on a separate partition (not that you'd do that in production, but it isolates variables).

For each filesystem, I created a directory structure mirroring a typical agent runtime: a journal/ directory with rolling JSONL files, a state/ directory with a SQLite database (WAL mode), and a tmp/ directory for staging writes. The journal writer was a simple Python script that appended a random-sized JSON line (1KB to 8KB) every 100ms, then called fsync. The SQLite database was updated every 5 seconds with a checkpoint.

The crash was simulated by pulling the power on the server — not a clean shutdown, not a reboot, but a hard kill. I used a smart power strip that I could toggle via a script. After each crash, I rebooted and ran a verification script that:

• Checked the journal for valid JSON lines (parseable, correct schema)
• Checked for any gaps or duplicates in sequence numbers
• Verified the SQLite database integrity with PRAGMA integrity_check
• Measured the time to remount and the time to replay the journal

Each cycle was: run for 30 minutes, crash, reboot, verify, run again. I did this for 90 days, which gave roughly 4,320 crash cycles per filesystem. That's a lot of power cuts, but it's the kind of abuse a journal might see in a year of autonomous operation, especially if the agent is running on edge hardware with flaky power.

What the Benchmark Showed

After 90 days, the results were surprisingly close, but there were meaningful differences in the failure modes.

Ext4 showed zero full-file corruptions. The journal files were always either fully written or not written at all. However, I did observe a few instances where the file size was correct but the last line was truncated — the fsync had not fully flushed the page cache before the power cut. This happened in about 0.1% of crashes (roughly 4 out of 4,320). The truncation was always at the very end of the file, and the JSON parser could detect it and discard the partial line. Recovery was straightforward: replay the journal from the last valid entry.

The SQLite database on ext4 never failed integrity_check, but there were two instances where the WAL file was not fully replayed, causing the database to roll back to the last checkpoint. That's acceptable for our use case, but it meant losing up to 5 seconds of state.

XFS also showed zero full-file corruptions. The journal files were always consistent. Truncated last lines occurred at a similar rate — about 0.1% of crashes. However, XFS's recovery was faster: the filesystem remounted in about half the time of ext4 (roughly 200ms vs 400ms on this hardware). That might not sound like much, but for an agent that needs to resume quickly after a power blip, it matters.

What surprised me was the behavior under heavy load. When the journal was being written at a high rate (every 10ms instead of 100ms), XFS occasionally delayed fsync completion due to its allocation group locking. This didn't cause corruption, but it increased the latency of individual writes, which could cause timeouts in the agent's decision loop. Ext4 was more consistent in this regard.

The Subtle Differences

Why the difference in recovery time? Ext4 uses a journaling block device (JBD2) that is inherently ordered. It groups commits and replays them in order, which is safe but slower. XFS uses a more modern approach with metadata journaling that allows for more parallelism during recovery, hence the faster remount.

Another difference is in how the two filesystems handle delayed allocation. Ext4 by default uses delalloc, which buffers writes in memory and flushes them later. If a crash happens before the flush, you lose data — but that's expected. XFS also uses delayed allocation, but it has a different heuristic for when to flush, which can lead to more aggressive writeback under memory pressure.

For an append-only journal, the critical property is that the file size and the data are consistent after a crash. Both filesystems guarantee that the data that was fsynced is durable, but they interpret 'fsynced' differently. On ext4, fsync on a file flushes the file data and the journal. On XFS, fsync flushes the file data but the metadata (size, mtime) may be logged separately. In practice, this didn't cause visible corruption, but it's a reason to be careful when using XFS with sync-heavy workloads.

Practical Recommendations

If you're running an agent memory journal on a single node, both ext4 and XFS are viable. The choice should be driven by your recovery time requirements and your write pattern.

• If you need fast recovery after a crash (e.g., for a trading agent that must resume within milliseconds), XFS is the better choice due to its faster remount times.
• If you have a high write rate and need consistent fsync latency, ext4 is more predictable.
• If you're using SQLite as part of the state, I'd lean toward ext4 because its ordered journaling works well with SQLite's WAL mode. XFS also works, but the occasional WAL rollback is more likely.
• If you're running on a RAID controller or a network filesystem, the filesystem choice matters less because the controller or network stack introduces its own durability semantics.

One more thing: always use barrier=1 for ext4 (it's default) and nobarrier is not recommended for XFS. Barriers ensure that the storage device's write cache is flushed before the filesystem considers a write durable. On NVMe, this is less of an issue because the drive's power-loss protection is usually good, but it's still a safety net.

The Bottom Line

After 90 days of brutal crash testing, I found no reason to avoid either filesystem for agent memory journals. The corruption rate was essentially zero for both. The differences are in recovery time and write latency, not in data integrity.

But the real lesson is not about ext4 vs XFS. It's that your journal must be designed to tolerate the rare partial write. Use a format that includes a length prefix or a checksum, so that a truncated line can be detected and discarded. And always test your crash recovery — not with a clean reboot, but with a power cut. That's the only way to know if your agent will survive the real world.

For the RiNET stack, which runs on three Hetzner servers with a WireGuard mesh, we've standardized on ext4 for the journal directories. It's not because it's better in every way, but because it's simpler to reason about, and the recovery time difference is negligible for our workloads. The nightly LoRA training runs on a separate XFS partition, where the sequential write pattern benefits from XFS's allocation behavior. But that's a different story.