Keeping Inference Logs on Local NVMe: A Storage Pattern for EU AI Act Audit Readiness

If you run self-hosted LLM inference for a European business, the EU AI Act is coming for your logs. Article 12 requires high-risk AI systems to log events for traceability. Article 13 demands transparency. And GDPR already says you can't ship personal data to US cloud object stores without safeguards.

The obvious answer: store logs locally. But local disk is slow, noisy, and finite. NVMe changes the calculus. With 3+ GB/s sequential writes and sub-millisecond latency, you can write every inference prompt, response, and metadata to local NVMe without bottlenecking your model.

Here's a concrete pattern I've used with vLLM and llama.cpp to keep inference logs on local NVMe, rotate them efficiently, and export to cold storage only when required.

Why Local NVMe for Logs?

Cloud object stores (S3, GCS, Azure Blob) are convenient but violate data sovereignty if you're processing EU personal data. Even if you use EU regions, the provider's access policies and jurisdictional reach may not satisfy GDPR Article 28 (processor) or the AI Act's audit requirements (Article 12).

Local NVMe gives you:

• Physical control: The data never leaves your server's chassis.
• Performance: 3-5 GB/s writes per drive. A single vLLM instance doing 100 requests/second with 10KB logs per request writes ~1 MB/s. You have headroom.
• Cost: NVMe drives are cheap per GB compared to cloud egress and storage fees.
• Audit simplicity: Logs are on a filesystem. You can hash, sign, and export them without network dependencies.

The Storage Pattern

Directory Layout

/var/log/inference/
├── prompts/          # raw prompt text (if not PII-heavy)
│   └── 2025-03-21.log
├── responses/        # model outputs
│   └── 2025-03-21.log
├── metadata/         # JSON lines: model, version, latency, token count
│   └── 2025-03-21.log
└── archive/          # compressed, signed, ready for export
    └── 2025-03-21.parquet.gpg

Each log file is JSON Lines (one JSON object per line) because it's append-only, easy to parse, and works with jq for ad-hoc queries.

Writing Logs from vLLM

vLLM allows custom logging via environment variables. Set:

export VLLM_LOGGING_CONFIG_PATH=/etc/vllm/logging.conf

In logging.conf, configure a file handler pointing to /var/log/inference/metadata/. Example:

[loggers]
keys=root,inference

[handlers]
keys=fileHandler

[formatters]
keys=jsonFormatter

[logger_inference]
level=INFO
handlers=fileHandler
qualname=vllm.inference

[handler_fileHandler]
class=handlers.RotatingFileHandler
args=('/var/log/inference/metadata/vllm.log','a',10000000,5)
formatter=jsonFormatter

[formatter_jsonFormatter]
format=%(asctime)s %(levelname)s %(message)s

But rotating file handlers are not ideal for AI Act audit. You need immutable, time-stamped files. Better: write your own logging plugin or use a sidecar process.

Using a Sidecar Writer

I run a small Go daemon (logwriter) that reads from a Unix domain socket. vLLM sends JSON log lines to the socket. The daemon appends to daily files, compresses previous day, and signs with GPG.

# systemd unit for logwriter
[Unit]
Description=Inference log writer for vLLM
After=local-fs.target

[Service]
ExecStart=/usr/local/bin/logwriter --dir /var/log/inference --rotate daily --sign-key 0xDEADBEEF
Restart=always
User=inference
Group=inference

[Install]
WantedBy=multi-user.target

Rotation and Compression

Logrotate is fine for simple rotation, but for audit readiness you want atomic moves and compression. Use logrotate with copytruncate for zero-downtime, but beware of data loss. I prefer the sidecar approach: at midnight, the daemon closes the current file, renames it with date, compresses with zstd, and signs with GPG.

# Example cron job (or built into logwriter)
0 0 * * * /usr/local/bin/rotate-logs --dir /var/log/inference --compress zstd --sign-key 0xDEADBEEF

Storage Calculation

Assume 1000 requests/second, each log entry 1KB (prompt+response+metadata). That's ~1 MB/s, ~86 GB/day. A 2TB NVMe holds ~23 days. For a high-traffic system, plan for 30 days on fast storage, then archive to HDD or encrypted object storage.

Audit Readiness

Immutability

After a log file is closed (e.g., next day), it should never be modified. Use chmod 0444 and chattr +i on the directory. The sidecar sets immutable attribute after compression.

Signing

Sign each daily archive with GPG. The public key can be held by an external auditor. This proves the logs haven't been tampered with since rotation.

gpg --detach-sign --armor --default-key 0xDEADBEEF /var/log/inference/archive/2025-03-21.parquet.zst

Export

When the auditor requests logs, you decrypt and convert to Parquet for easy analysis. The export script:

#!/bin/bash
# export-logs.sh --date 2025-03-21 --output /tmp/audit
set -euo pipefail

DATE="$1"
OUTPUT="$2"

# Decrypt and decompress
gpg --decrypt /var/log/inference/archive/${DATE}.parquet.zst.gpg | zstd -d > ${OUTPUT}/${DATE}.parquet

# Verify signature
gpg --verify /var/log/inference/archive/${DATE}.parquet.zst.sig /var/log/inference/archive/${DATE}.parquet.zst

Querying Without Moving Data

Use DuckDB for on-the-fly queries on the compressed archives:

SELECT * FROM read_parquet('/var/log/inference/archive/2025-03-21.parquet')
WHERE prompt LIKE '%GDPR%'
LIMIT 10;

No need to copy to a database. DuckDB reads directly from the filesystem.

Pitfalls

NVMe Wear

Logging is write-heavy. Consumer NVMe drives have limited endurance (e.g., 300 TBW for a 1TB drive). At 86 GB/day, that's ~10 years. Enterprise drives (e.g., Samsung PM9A3) have 10x endurance. Use them.

Filesystem Choice

Ext4 is fine. XFS is better for concurrent writes. Avoid ZFS for pure log storage due to overhead. Use noatime mount option.

Clock Skew

If your server's clock jumps, log timestamps become unreliable. Run chronyd and monitor with timedatectl. Consider writing a monotonic clock counter alongside wall clock time.

Alternatives

• Postgres with pgvector: You can store logs in a table and use SQL for audit queries. But write throughput is lower (10k-50k inserts/s vs 1M+ lines/s with flat files).
• Prometheus + Loki: Good for metrics, but logs are not immutable by default. You'd need to configure retention and signing.
• Syslog with TLS: Works, but parsing structured JSON from syslog is painful.

Local NVMe is the simplest, most sovereign, and most performant option for inference logs. It's not fancy. It's just files on a fast disk.

Conclusion

The EU AI Act doesn't specify how you store logs, only that you must. By keeping inference logs on local NVMe with daily rotation, compression, and GPG signing, you satisfy audit requirements without shipping data to the cloud. This pattern works today with vLLM, llama.cpp, or any inference server that can output structured logs. Your auditor will thank you.

No emoji. No marketing. Just storage.