Provenance Chains for Every Token: Linking Inference Logs to Training Data Lineage

The EU AI Act does not require you to link an output token back to the training examples that shaped it. Article 12 asks for considerably less: a logging capability scoped to three purposes, namely identifying risk or substantial modification, feeding post-market monitoring under Article 72, and supporting deployer oversight under Article 26(5), with a six-month retention floor set by Articles 19 and 26(6). Token-level provenance is not mandated by that provision. You should build it anyway. For anyone running real inference infrastructure, this is not a paperwork exercise. It is an engineering problem: how do you record, store, and query the chain from a token in a response all the way to the training examples that influenced it? This article outlines a practical architecture for building provenance chains at the token level, using techniques that fit into existing stacks without requiring a rewrite of your serving layer.

Why Token-Level Provenance Matters

Most logging pipelines stop at request and response payloads. That is enough for debugging, but not for regulatory traceability. The training-data obligations in the Act sit elsewhere and are documentary rather than forensic: Article 10(2) on data collection processes and the origin of data, Article 11 with Annex IV on documenting the datasets used, and Article 53(1)(c) and (d) for general-purpose model providers. None of them asks you to resolve a specific output back to specific training examples. A single response from a language model may be influenced by thousands of training examples, but the chain of influence is not directly observable from the weights. Provenance chains bridge that gap by recording explicit links between inference events and the training data lineage.

Token-level provenance is the granularity that makes this workable. Instead of logging a whole response as a blob, you record metadata per token: which model version, which LoRA adapter, which retrieval context, and which training data segments are implicated. This is not about storing the actual training data in your logs—that would be impractical. It is about storing references and hashes that can be resolved back to a lineage database.

The Core Components

A provenance chain has three layers: the inference log, the lineage index, and the training data manifest. Each layer serves a distinct purpose and has its own storage and query characteristics.

Inference Log

The inference log is the entry point. Every request that hits your model should produce a structured log entry that includes:

• Request ID and timestamp
• Model identifier (including base model and any LoRA adapters)
• Input tokens (or a hash of them, if privacy is a concern)
• Output tokens, each with a token ID and position
• Retrieval context: which documents or embeddings were used, with their IDs
• Sampling parameters (temperature, top-p, etc.)

In practice, you do not need to log every token individually in a separate row. A single JSON document per request can contain an array of token objects, each with its own provenance metadata. This keeps the log volume manageable while still allowing per-token queries.

Lineage Index

The lineage index is a structured store that maps model artifacts and retrieval context to training data segments. For a model like Qwen, which we run on vLLM, the lineage index would contain entries for each checkpoint and each nightly LoRA update. Each entry lists the training dataset versions, the exact data points used, and the hashes of those data points.

The lineage index is not a log; it is a reference database. It should be queryable by model version, by data segment, and by date. A typical schema might look like this:

CREATE TABLE model_artifacts (
    artifact_id UUID PRIMARY KEY,
    model_name TEXT,
    version TEXT,
    base_model TEXT,
    lora_adapter TEXT,
    trained_on JSONB,  -- list of dataset versions
    created_at TIMESTAMPTZ
);

CREATE TABLE data_segments (
    segment_id UUID PRIMARY KEY,
    dataset_name TEXT,
    version TEXT,
    data_hash TEXT,
    source_path TEXT,
    metadata JSONB
);

CREATE TABLE artifact_segment_link (
    artifact_id UUID REFERENCES model_artifacts(artifact_id),
    segment_id UUID REFERENCES data_segments(segment_id),
    weight FLOAT,  -- optional, if you track influence
    PRIMARY KEY (artifact_id, segment_id)
);

This is a simplified version. In production, you might add sharding, partitioning by time, and indexes on the hash columns. The key point is that the lineage index is a separate system from the inference logs, because the access patterns are different: logs are append-heavy and time-ordered; the lineage index is read-heavy and relationship-oriented.

Training Data Manifest

The training data manifest is the ground truth. It is a versioned, immutable record of every dataset used for training or fine-tuning. This is where you store the actual data points, or at least their hashes and metadata. The manifest should be generated at training time, not retroactively, because retroactive generation is error-prone and may miss data that was deleted or modified.

A manifest entry for a single data point might look like:

{
    "id": "dp-2024-001",
    "dataset": "fineweb",
    "version": "2024-01",
    "hash": "sha256:abc123...",
    "content": "The full text or a pointer to it",
    "metadata": {
        "source_url": "https://example.com/article",
        "language": "en",
        "license": "cc-by"
    }
}

In practice, you do not store the full content in the manifest for large datasets; you store the hash and a pointer to the storage location. The manifest is what you would produce as evidence in an audit.

Building the Chain at Inference Time

When a request comes in, the inference service needs to capture the necessary context to build the provenance chain. This is not a post-processing step; it must happen in the request path, because you need the actual retrieval context and model version at that moment.

Here is a pseudocode flow for a typical request:

async def handle_request(request):
    request_id = generate_uuid()
    model = get_current_model()  # includes LoRA adapter info
    
    # Retrieve context if using RAG
    retrieval_context = await retrieve(request.query)
    
    # Generate response with vLLM
    response = await vllm.generate(
        model=model,
        prompt=request.query,
        sampling_params=request.sampling_params
    )
    
    # Build provenance log
    log_entry = {
        "request_id": request_id,
        "timestamp": now(),
        "model": model.identifier,
        "input_hash": hash(request.query),
        "retrieval_context": [
            {"doc_id": doc.id, "score": doc.score} for doc in retrieval_context
        ],
        "tokens": [
            {
                "token_id": token.id,
                "position": i,
                "probabilities": token.probs  # optional
            } for i, token in enumerate(response.tokens)
        ]
    }
    
    # Write to log store (e.g., Postgres or a time-series DB)
    await log_store.append(log_entry)
    
    return response

This is straightforward, but it has performance implications. Adding a log write per request can add latency, especially if the log store is remote. Common patterns to mitigate this include:

• Asynchronous logging: write to a local buffer and flush in batches.
• Using a separate log sink that does not block the response path.
• Sampling: for high-throughput systems, you might log only a percentage of requests, but that does not meet audit requirements. For audit, you need every request.

In our stack, we use Postgres for structured logs, but for high-volume inference you might consider a columnar store or a dedicated log aggregator. The tradeoff is between write speed and query flexibility.

Resolving a Provenance Chain

Suppose an auditor asks: "Why did the model output token X at position Y for request Z?" You need to resolve the chain from the inference log to the training data.

The resolution process is:

1. Look up the request in the inference log to get the model identifier and retrieval context.
2. Query the lineage index to find which data segments were used for that model version.
3. Query the training data manifest to get the actual data points.

This can be done with a few SQL joins or with a graph query. We use Neo4j for the lineage index because it naturally models the relationships between artifacts, segments, and data points. A Cypher query might look like:

MATCH (a:Artifact {id: $model_id})-[:TRAINED_ON]->(s:Segment)-[:CONTAINS]->(d:DataPoint)
RETURN d

But you can also do this in Postgres with recursive CTEs. The choice depends on your existing infrastructure and the complexity of your lineage graph.

Handling LoRA and Fine-Tuning

One of the tricky parts is that models are not static. We run nightly LoRA training on our stack, which means the model version changes frequently. Each LoRA update is a new artifact in the lineage index, and it must reference the base model and the new training data.

When a LoRA update is deployed, the inference service must immediately start logging the new model version. This is a configuration issue: the model identifier should be part of the deployment metadata, and the logging code should read it from there, not hardcode it.

A common mistake is to log only the base model name and forget the LoRA adapter. That breaks the provenance chain because you cannot tell which fine-tuning influenced the output.

Storage and Retention

Provenance data grows quickly. Each request generates a log entry, and each log entry can be several kilobytes if you include token-level details. For a system serving thousands of requests per second, that is gigabytes per day. You need a retention policy that balances audit requirements with storage costs.

The Act does set a floor: Articles 19 and 26(6) require automatically generated logs to be kept for a period appropriate to the intended purpose, and at least six months, longer where other Union or national law applies. That is a floor, not a target, and it is wise to keep provenance data for at least the lifecycle of the model. For high-risk systems, you might keep logs for several years. In practice, you can tier storage: hot logs for recent data, warm storage for the last year, and cold archival for older data.

For the lineage index and manifest, they are small and immutable, so you can keep them indefinitely. But the inference logs are the bulk.

Privacy and Security

Provenance logs contain sensitive information: user queries, retrieval contexts, and possibly personal data. You need to protect them. Encryption at rest is mandatory, and access should be restricted to authorized personnel. Also, consider hashing input text to avoid storing raw queries if that is not necessary.

However, hashing can break the ability to audit the exact input. A compromise is to store a hash for routine logs and a separate, access-controlled store for full inputs when needed for audit.

Tooling and Implementation

We have implemented this pattern in our own stack, which consists of three Hetzner servers connected via WireGuard mesh. We run Qwen on vLLM, use BGE-M3 for embeddings, and store data in Postgres, Qdrant, and Neo4j. The provenance chain fits naturally:

• Postgres stores the inference logs (with JSONB for token arrays).
• Qdrant stores the retrieval context, and we include document IDs in the logs.
• Neo4j holds the lineage index, connecting model artifacts to data segments.
• The training data manifest is a set of JSON files in an object store (or a simple directory on the servers).

We do not claim this is a turnkey solution; it requires careful design. But the components are standard and the patterns are proven.

Practical Considerations

• Log volume: Use compression and batching. For example, log in NDJSON format and compress with gzip before writing to disk.
• Query performance: Index the request ID and timestamp in the log store. For lineage queries, index the artifact ID and segment ID.
• Time sync: Ensure all servers have synchronized clocks (NTP) so that logs from different components can be correlated.
• Testing: Write integration tests that simulate a request and verify that the provenance chain resolves correctly.

The Bottom Line

Token-level provenance is not a regulatory requirement, and I would rather say so plainly than sell it as one. It is an operational one. When a bad output ships, the first question is which adapter, which retrieval chunk, which dataset version, and that question cannot be answered retroactively. The architecture I described is modular and can be adapted to your existing stack. The key is to start logging the right metadata now, because retrofitting provenance after the fact is painful and often incomplete.

We have been running this pattern for a while, and the main lesson is that provenance is not a single feature; it is a cross-cutting concern that affects logging, storage, and even the model serving configuration. Get the metadata right at the source, and the rest is just querying.

If you are building for the EU market, do not wait for the regulators to knock. Build the chain now, and make it a habit.

Correction, 25 August 2026: an earlier version of this article stated that the EU AI Act requires linking model output back to training data. It does not. Article 12 is a logging-capability provision with a narrower scope, and the training-data obligations in Articles 10, 11 and 53 are documentary. The framing has been corrected throughout, and the retention paragraph now cites the six-month floor in Articles 19 and 26(6). Note also that Regulation (EU) 2026/1744 moved the Annex III high-risk application date to 2 December 2027. Thanks to Marius Laurusevicius for the correction.