fsync on Every Token: A Power-Loss Journal for GPU Inference

You're mid-generation, the GPU has produced 800 tokens of a 2,000-token response, and the datacenter loses power. When the node comes back, that state is gone. The client retries, the model regenerates, and you've burned GPU-hours and user patience. For any inference service that matters, you need a journal. Not a buffer, not a cache—a write-ahead log that records every token before it leaves the GPU, and survives a hard kill.

This article walks through the design of a token journal for a multi-GPU inference node, focusing on the storage layer. We'll cover the WAL structure, the fsync discipline, crash recovery, and the performance tradeoffs. This is infrastructure engineering, not product talk. If you run inference at scale, you already know the pain.

Why a Journal?

Inference is stateful. A generation is a sequence of tokens, each dependent on the previous. If you lose 500 tokens in the middle, you can't just resume from the last checkpoint—the model's hidden state is gone. You have to restart from the prompt, which is expensive and often produces different output. A journal gives you the ability to replay the token stream and reconstruct the state, or at least know exactly where you stopped.

A journal is not a database. It's an append-only log of events. For inference, each event is a token, along with its metadata: the request ID, the sequence position, the timestamp, and the model version. You write to the journal as you generate, and you fsync at strategic points. The journal's job is to survive power loss, not to serve queries.

The WAL Structure

A write-ahead log (WAL) is the classic solution. The idea: before you mutate any durable state, you append a record describing the mutation to a log. The log is the source of truth. On recovery, you replay the log to reconstruct the state.

For token streams, the WAL is simple. Each record is a token event. You can group records into segments—say, 64 KB per segment—to reduce fsync overhead. But the principle is the same: append, fsync, then acknowledge.

Here's a minimal record format:

{
  "request_id": "req_1234",
  "seq": 0,
  "token": 5123,
  "model": "qwen-14b",
  "timestamp": 1710000000
}

In binary, you'd pack this into a fixed-size header plus payload, but the idea stands. The journal is just a sequence of these records.

fsync Discipline

fsync is the system call that forces data to physical storage. Without it, writes sit in the page cache, and a power loss wipes them. The discipline is: when do you call fsync, and how often?

Too often, and you serialize the GPU. Every fsync can take milliseconds—on spinning disks, tens of milliseconds. The GPU can generate tokens in microseconds. If you fsync after every token, you'll bottleneck at the disk.

Too rarely, and you lose data. If you fsync every 1,000 tokens, a power loss loses up to 1,000 tokens. The question is: what's your tolerance?

For many services, a few lost tokens are acceptable. The client can retry. But if you're building a system that must be deterministic, or if retries are expensive, you need tighter control.

A common pattern is to fsync at a fixed interval, say every 100 ms, or after a batch of tokens. Batching amortizes the fsync cost. But you still have a window where tokens are at risk.

An alternative is to fsync only when the client asks for a checkpoint. But that's not a journal—that's a checkpoint.

The Power-Loss Scenario

Let's walk through a power loss. The node is running, the GPU is generating, and the journal is appending. When power drops, the OS stops. The page cache is lost, but anything that was fsynced is on disk.

On reboot, the journal is on disk. You need to replay it. The replay process:

1. Open the journal file.
2. Read records from the last known good offset.
3. For each record, reconstruct the token stream.
4. Determine which requests are incomplete.
5. Decide: resume or notify the client.

Resuming is hard. The model's KV cache is gone. You'd need to re-run the prompt and all previous tokens through the model. That's expensive. For most systems, you don't resume—you just know exactly where you stopped, and you tell the client to retry from that point.

The journal gives you durability of the token stream, not the model state. That's still valuable: you can avoid re-generating the entire response, and you can provide a precise error to the client.

Journaling with NVMe vs. SATA

Storage hardware matters. NVMe drives have low latency and high throughput, making fsync cheap. SATA SSDs are slower. Spinning disks are a nightmare.

If you're running a GPU node, you likely have NVMe. That's good. But even with NVMe, fsync isn't free. The drive's write cache and power-loss protection play a role.

Some drives have a power-loss protection (PLP) capacitor that ensures in-flight writes are flushed on power loss. That can reduce the need for fsync, but you can't rely on it. The OS still has a page cache.

A common pattern is to use a dedicated journal file on a separate disk or partition, to avoid contention with the model weights and other I/O.

Implementation Sketch

Let's sketch a simple journal in Python. You wouldn't use Python for performance, but the logic is clear.

import os
import json

class TokenJournal:
    def __init__(self, path):
        self.path = path
        self.fd = os.open(path, os.O_RDWR | os.O_CREAT)
        self.offset = 0

    def append(self, record):
        data = json.dumps(record).encode() + b'\n'
        os.write(self.fd, data)
        self.offset += len(data)

    def fsync(self):
        os.fsync(self.fd)

    def recover(self):
        self.fd.seek(0)
        data = self.fd.read()
        for line in data.splitlines():
            yield json.loads(line)

This is naive—no batching, no checksums—but it shows the core. In production, you'd use a binary format, add CRC32 checksums, and batch writes.

Batching and Group Commit

To reduce fsync frequency, you can batch multiple tokens into a single fsync. This is group commit, borrowed from database design. The idea: collect tokens for a short window (say 10 ms), write them all, then fsync once. This amortizes the fsync cost over many tokens.

The tradeoff is durability: you can lose up to 10 ms of tokens on power loss. Tune the window to your tolerance.

Another trick is to use fdatasync instead of fsync. fdatasync doesn't flush metadata (like file size), which can be faster. But if the file size changes, you need fsync. Pre-allocate the file to avoid size changes.

Crash Recovery and Validation

On recovery, you need to validate the journal. A power loss can leave a partially written record. Use checksums to detect corruption.

def recover(self):
    self.fd.seek(0)
    data = self.fd.read()
    offset = 0
    while offset < len(data):
        # read a fixed-size header
        header = data[offset:offset+HEADER_SIZE]
        if len(header) < HEADER_SIZE:
            break
        # parse length and checksum
        length, crc = struct.unpack('II', header)
        payload = data[offset+HEADER_SIZE:offset+HEADER_SIZE+length]
        if len(payload) < length:
            break
        if crc32(payload) != crc:
            break
        yield json.loads(payload)
        offset += HEADER_SIZE + length

You stop at the first bad record. Everything after is garbage.

Performance Tradeoffs

Designing a journal is a series of tradeoffs. Here are the key ones:

• fsync frequency vs. durability: More fsyncs mean less data loss but lower throughput.
• Batch size vs. latency: Larger batches reduce fsync overhead but add latency to the acknowledgment.
• Checksums vs. overhead: Checksums add CPU and storage overhead but catch corruption.
• Separate disk vs. shared: A dedicated journal disk avoids contention but adds cost.

A typical pattern is to write to a memory-mapped file and periodically msync. msync is similar to fsync but works on mapped memory. It's often faster for large writes.

Real-World Considerations

In practice, you don't need to reinvent the wheel. Libraries like SQLite's WAL mode or Postgres's WAL are battle-tested. You could use SQLite to store tokens, but SQLite's WAL is designed for database transactions, not high-throughput token streams. It's doable, but you'll pay for overhead.

A dedicated journal is simpler. Just a file, a format, and a recovery routine. You can even use a message queue like Kafka, but that's a heavy dependency.

For a 4-GPU node, you might have multiple generation processes. Each writes to its own journal, or you coordinate with a single journal. A single journal serializes writes, which could be a bottleneck. Multiple journals complicate recovery.

A common approach is to shard by request ID, so each request has its own journal file. That way, recovery is per-request, and you don't need to scan a giant log.

Conclusion

A token journal is a critical piece of infrastructure for any inference service that must survive power loss. It's not glamorous, but it's the difference between a graceful recovery and a lost generation. The design is straightforward: append-only log, fsync discipline, crash recovery. The hard part is tuning the tradeoffs to your workload.

Remember, the journal is not a checkpoint. It's a record of what happened. Use it to know where you stopped, not to resume. That's a crucial distinction.

Now go fsync your tokens.