Write-Once Audit Logs on NVMe: Surviving Power Loss and Regulator Scrutiny

If your AI system processes EU citizen data, the EU AI Act will demand audit trails that are tamper-evident, durable, and survive power loss. Consumer-grade logging to a database isn't enough. You need write-once semantics at the storage layer.

This article shows how to build immutable audit logs on NVMe SSDs using Linux kernel features, NVMe atomic write support, and cryptographic verification. No custom hardware needed—just a recent kernel and an NVMe drive that supports atomic operations.

Why Write-Once?

Write-once means data is appended but never modified or deleted after commit. This gives you:

• Tamper evidence: Any deletion or alteration leaves cryptographic traces.
• Power-loss safety: In-flight writes either complete fully or roll back to the previous good state.
• Regulatory compliance: The EU AI Act requires that high-risk AI systems maintain logs that are “comprehensive, tamper-proof, and accessible.”

Standard ext4 or XFS don't enforce write-once. But we can layer immutability using:

• NVMe atomic write unit (AWUN/AWUPF) to guarantee power-loss-safe single writes.
• fs-verity for per-file Merkle tree integrity.
• dm-writeboost or systemd-journald with append-only files.

Hardware Requirements

You need an NVMe SSD that advertises atomic write support. Check with:

nvme id-ctrl /dev/nvme0 | grep -E "(awun|awupf)"

Look for awun > 0. Most modern enterprise drives (Samsung PM9A3, Kioxia CD6) support atomic write unit up to 128KB. Consumer drives often don't—test before deploying.

Filesystem Setup

Use XFS with realtime device for append-only logs. XFS supports atomic write via the atomic mount option (Linux 5.10+).

mkfs.xfs -f -m reflink=0 -d agcount=4 /dev/nvme0n1
mount -o atomic /dev/nvme0n1 /mnt/audit

The atomic flag ensures that each write is either fully persisted or not at all, even if power fails mid-write.

Write-Once Enforcement

Create an append-only directory with immutable file attribute:

mkdir -p /mnt/audit/logs
chattr +a /mnt/audit/logs

Files created inside can only be opened in append mode. But this doesn't prevent writes from being torn. Combine with O_APPEND and fsync:

// Example: write audit record atomically
int fd = open("/mnt/audit/logs/20250321.log", O_WRONLY | O_APPEND | O_CREAT, 0444);
write(fd, record, len);
fsync(fd);  // ensures power-safe flush

To verify atomicity, test with power-loss simulation:

echo b > /proc/sysrq-trigger  # immediate reboot

After reboot, each record should be complete or absent—no partial bytes.

Cryptographic Verification with fs-verity

fs-verity provides a Merkle tree over file contents. Enable it on the log file after writing is complete (e.g., after log rotation):

fsverity enable /mnt/audit/logs/20250321.log
fsverity measure /mnt/audit/logs/20250321.log

This produces a sha256 hash. Store the hash off-device (e.g., in a separate tamper-proof database or print it on paper). Later verification:

fsverity verify /mnt/audit/logs/20250321.log

Any modification—even a single bit flip—will fail verification.

Power-Loss Safe Journaling

Even with atomic writes, the filesystem metadata might not survive a crash. Use a dedicated NVMe namespace with write-through caching. Disable the write cache on the device:

nvme set-feature /dev/nvme0 -f 0x0c -v 0  # disable volatile write cache

Then mount with nobarrier? No—keep barriers enabled for safety. Use sync mount option:

mount -o atomic,sync /dev/nvme0n1 /mnt/audit

The sync option forces every write to be flushed immediately, at the cost of throughput. For audit logs, throughput is secondary to safety.

Integration with systemd-journald

systemd-journald can forward logs to a file with SyncIntervalSec=0 and SplitMode=none. Configure:

# /etc/systemd/journald.conf.d/audit.conf
[Journal]
Storage=persistent
SyncIntervalSec=0
SplitMode=none
SystemMaxUse=10G
ForwardToSyslog=no

Then use a custom output plugin or rsyslog to write to the append-only directory. But journald's native binary format is not write-once. Better to use a dedicated logger that calls fsync after each record.

Satisfying EU AI Act Requirements

The EU AI Act (Article 12) requires:

• Logging of events during operation.
• Logs must be “comprehensive, tamper-proof, and accessible.”
• Retention period defined by risk category (typically 6 months to 5 years).

Our setup meets these:

• Comprehensive: Log all inference requests, model versions, training data hashes, and system events.
• Tamper-proof: Write-once files + fs-verity + off-device hash storage.
• Accessible: Standard POSIX file access; export via SFTP or API.

Performance Considerations

Atomic writes with sync mount are slow. Expect ~50 MB/s on a single thread. To scale:

• Use multiple NVMe drives in RAID0 (but lose per-drive atomicity? Actually NVMe atomicity is per-namespace, so RAID0 with stripe size ≤ atomic unit works).
• Batch records into 128KB chunks to match atomic write unit.
• Use a separate NVMe namespace for metadata.

Alternative: Qdrant with WAL

If you need vector search on audit logs, Qdrant's WAL (write-ahead log) is crash-safe and append-only. But it's not write-once—old segments are merged. For pure audit, stick with raw files.

Conclusion

Write-once audit logs on NVMe are achievable with standard Linux tools. The combination of atomic mount, append-only directories, fs-verity, and power-loss-safe settings gives you a tamper-evident log that can survive both a power outage and a regulator's scrutiny. Test your specific NVMe model for atomic write support before production deployment. And always store verification hashes off-device—preferably on paper in a safe.