Pseudonymizing Activations at the Tokenizer: EU AI Act Compliance Pattern

When you run a large language model in production, the tokenizer is the first place where raw text becomes numbers. It is also the last place where you can still see the original string before it gets embedded into high-dimensional vectors. For anyone operating under the EU AI Act, that boundary is where you want to build a pseudonymization layer. The Act's transparency and data governance obligations push you to minimize personal data in logs, training traces, and inference telemetry. This article describes a concrete pattern: intercept the tokenizer output, replace sensitive tokens with deterministic pseudonyms, and keep the mapping outside the model's reach.

The idea is not to anonymize the whole sequence—that would destroy the model's ability to generate coherent responses. Instead, you pseudonymize only the tokens that carry high-risk personal data, such as names, email addresses, phone numbers, and national identifiers. The rest of the sequence flows through unchanged. The mapping from original tokens to pseudonyms is stored in a separate, encrypted store, so that if a log or audit trail is leaked, the pseudonymized tokens cannot be reversed without access to that store.

Why the Tokenizer Is the Right Choke Point

The tokenizer sits between raw text and the model's embedding layer. It converts a string into a list of token IDs, and often also produces a mapping from token IDs back to subword strings. This is the only stage where you have a lossless representation of the input. After embedding, you get vectors that are not human-readable, but they still encode the original information in a way that can be partially reconstructed with model inversion techniques. If you pseudonymize at the tokenizer, you prevent the original text from ever entering the model's context, which means it never appears in attention matrices, KV caches, or any downstream telemetry.

A common pattern is to run a lightweight named-entity recognition (NER) model right after tokenization, but before the embedding layer. That NER pass identifies spans that correspond to person names, locations, organizations, and other categories. For each sensitive span, you replace the tokens with a special pseudonym token, such as [PSEUDO:1], and store the mapping in a sidecar database. The model sees only the pseudonym tokens, and its output is generated based on those. After generation, you can optionally replace the pseudonyms back with the original values in the final response, but that step is outside the model's inference path.

The Pseudonymization Service

To make this work, you need a service that can map a sensitive span to a pseudonym deterministically. Deterministic means the same input always produces the same pseudonym, so that references to the same entity are consistent across a conversation. This is important for multi-turn dialogues. If you used random pseudonyms, the model would lose track of which entity is which, breaking coherence.

The mapping service can be a simple key-value store, but it needs to be fast because it is on the critical path of inference. A common setup is a Redis or Memcached instance with a TTL, but for audit purposes you may want a persistent store like Postgres with an encrypted column. The key is the original sensitive string (or its hash), and the value is the pseudonym token ID. The service should expose an API that the tokenizer wrapper calls.

Here is a simplified Python sketch of the tokenizer wrapper:

class PseudonymizingTokenizer:
    def __init__(self, base_tokenizer, ner_model, mapping_service):
        self.base = base_tokenizer
        self.ner = ner_model
        self.mapping = mapping_service

    def encode(self, text: str) -> list[int]:
        # Run NER to find sensitive spans
        spans = self.ner(text)
        # Replace sensitive spans with placeholder tokens
        pseudonymized_text = self._replace_spans(text, spans)
        # Tokenize the pseudonymized text
        return self.base.encode(pseudonymized_text)

    def _replace_spans(self, text, spans):
        # Sort spans by start, then build new string
        parts = []
        last_end = 0
        for start, end, label in spans:
            parts.append(text[last_end:start])
            original = text[start:end]
            pseudo = self.mapping.get_or_create(original)
            parts.append(pseudo)
            last_end = end
        parts.append(text[last_end:])
        return ''.join(parts)

This is a high-level sketch. In practice, you need to handle overlapping spans, case sensitivity, and the fact that the NER model itself might be a transformer that runs on GPU. That adds latency, so you need to decide whether to run NER on CPU or GPU, and whether to batch it.

Handling the Mapping Store

The mapping store is the linchpin. If it leaks, the pseudonymization is useless. So you need to encrypt the values at rest, and ideally use a separate key management system. The mapping should be scoped per tenant or per deployment, so that one tenant's data is not accessible to another.

A typical schema for the mapping table looks like this:

CREATE TABLE pseudonym_mapping (
    id BIGSERIAL PRIMARY KEY,
    original_hash CHAR(64) NOT NULL UNIQUE, -- SHA-256 of original string
    original_encrypted BYTEA NOT NULL,      -- AES-256-GCM encrypted
    pseudonym_token VARCHAR(20) NOT NULL UNIQUE,
    created_at TIMESTAMPTZ DEFAULT now(),
    tenant_id UUID NOT NULL
);

The original_hash is used for fast lookups without exposing the original in the index. The original_encrypted stores the actual value, but you never query by it. The pseudonym_token is a unique string like [PSEUDO:123] that the tokenizer will use.

When the tokenizer encounters a sensitive span, it first hashes the original string and looks up the mapping. If not found, it generates a new pseudonym token and inserts the mapping. To avoid race conditions, you need a unique constraint on original_hash and handle conflicts gracefully, perhaps by retrying the lookup after a conflict.

Impact on Model Quality

Replacing tokens with pseudonyms changes the distribution of token IDs the model sees. The model has never seen [PSEUDO:123] during training, so it will treat it as an unknown token. This can degrade performance, especially if the model relies on the semantic content of names. For example, if the input is "Alice is talking to Bob", and you replace both names with pseudonyms, the model loses the gender and cultural cues, which might affect pronoun resolution. However, for many tasks like summarization or question answering, the impact is minimal because the model can still infer the structure.

Teams that have adopted this pattern typically observe a small drop in perplexity on held-out data, but the trade-off is acceptable for compliance. To mitigate the quality loss, you can use a special token that the model has been fine-tuned to understand, such as [PERSON] or [ORG]. You can add these tokens to the tokenizer's vocabulary and fine-tune the model on a small dataset where sensitive spans are replaced with these tokens. That way, the model learns to treat them as placeholders.

Integration with vLLM and Other Inference Engines

If you are running a model with vLLM, you can implement the pseudonymizing tokenizer as a custom tokenizer class that vLLM uses. vLLM allows you to pass a tokenizer object, so you can wrap the original tokenizer. Here is an example of how you might integrate it with vLLM's LLM class:

from vllm import LLM, SamplingParams

# Assume PseudonymizingTokenizer is defined elsewhere
pseudo_tokenizer = PseudonymizingTokenizer(base_tokenizer, ner_model, mapping_service)
llm = LLM(model="qwen/Qwen2.5-7B", tokenizer=pseudo_tokenizer)

output = llm.generate(["Call Alice at 555-1234"], SamplingParams(temperature=0.8))

This way, the pseudonymization happens before the model sees the text. The logs and telemetry will only contain pseudonymized tokens, which reduces the risk of personal data exposure in audit trails.

EU AI Act Compliance Considerations

The EU AI Act, as of the knowledge cutoff in 2024, is not fully finalized, but it is clear that high-risk AI systems must have robust data governance. Pseudonymization is not explicitly required, but it is a strong measure to demonstrate that you have taken steps to minimize personal data processing. The Act's transparency obligations may require you to log inputs and outputs for auditability. If those logs contain raw personal data, you are processing that data, and you need a legal basis. Pseudonymization reduces the risk by making the logs less sensitive, but it is not a silver bullet. You still need to consider whether the pseudonymized data can be re-identified by an adversary with access to the mapping store.

A key point is that pseudonymization is not anonymization. The GDPR defines pseudonymization as processing personal data in such a way that it can no longer be attributed to a specific data subject without additional information. That additional information (the mapping) must be kept separately and subject to technical and organisational measures. The EU AI Act likely will reference GDPR concepts, so you should align your approach with GDPR's Article 4(5) and Recital 26.

Operational Concerns

The pseudonymization layer adds latency and operational complexity. The NER model runs on every input, which could double the inference time if not optimized. You can mitigate this by running NER on a smaller model, like a distilled version, or by using a rule-based fallback for common patterns like email addresses and phone numbers.

Another concern is the mapping store's availability. If it goes down, the tokenizer cannot generate new pseudonyms, which would block inference. You can implement a fallback that uses a local cache or generates temporary pseudonyms that are not persisted, but that would break consistency across requests. A better approach is to run the mapping store in the same region as the inference service and use a highly available setup like a replicated Postgres cluster.

Example: Pseudonymizing Email Addresses

Let's look at a concrete example. Suppose you have a customer support bot that receives emails. You want to log the conversation for quality assurance, but you don't want to store the customer's email address in plaintext. With the tokenizer-level pseudonymization, you can replace the email address with a pseudonym before the model processes it.

Original input: "My email is john.doe@example.com, please send the invoice."

After NER, the span "john.doe@example.com" is detected as an email. The tokenizer replaces it with [PSEUDO:1001]. The model generates a response based on that. The log entry looks like:

input: "My email is [PSEUDO:1001], please send the invoice."
output: "We will send the invoice to [PSEUDO:1001]."

If you need to send the actual invoice, you can replace the pseudonym with the original email after the model response, but that step is done outside the model's critical path and can be logged separately with access control.

Conclusion

Pseudonymizing at the tokenizer is a practical pattern for reducing personal data exposure in LLM deployments. It is not a complete compliance solution, but it is a strong technical measure that aligns with the EU AI Act's emphasis on data governance and privacy. By intercepting the tokenizer output, you prevent raw personal data from entering the model's context, which reduces the risk of leakage through logs, telemetry, or model inversion attacks. The pattern is implementable with existing tools and can be integrated into inference engines like vLLM. The main trade-offs are added latency and potential quality degradation, but those can be mitigated with careful design and fine-tuning.

If you are building sovereign AI infrastructure, this pattern is a good addition to your stack. It shows that you are taking privacy seriously, and it gives you a clear audit trail that does not expose personal data. Start with a small set of sensitive categories, measure the impact on your use case, and iterate.