Pre-filtering Inference Requests at the Proxy: GDPR-Compliant Guard Against Personal Data in Prompts

We run a multi-tenant LLM inference cluster. Users send prompts, we return completions. Simple, until a compliance officer drops by and asks: "How do you ensure no personal data ends up in the model's training set or logs?"

Default answer: "We don't train on user data." But GDPR Article 5(1)(c) (data minimisation) and Article 32 (security of processing) demand proactive controls. You can't just promise. You need to prove.

So we built a transparent inference proxy that pre-filters every prompt for personally identifiable information (PII) — before the request ever reaches the model. If PII is detected, the request is blocked, logged, and an alert fires. No data leaks, no liability, no surprises.

Architecture Overview

The proxy sits between the client and the inference server (e.g., vLLM, TGI, or custom Triton). It intercepts every HTTP POST to /v1/chat/completions or /v1/completions, runs the prompt through a detection pipeline, and either forwards or rejects.

Client -> Proxy (PII scan) -> [block] -> 403 + audit log
                        |-> [pass] -> Inference server

We use a two-stage filter: a fast regex pass for obvious patterns (emails, phone numbers, SSNs), then a lightweight ML model (spaCy + custom NER) for context-dependent entities like names and addresses.

Stage 1: Regex Filter

Regex catches low-hanging fruit with zero latency. We compile a set of patterns at startup. Example for emails and US SSNs:

import re

EMAIL_RE = re.compile(r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}')
SSN_RE = re.compile(r'\b\d{3}-\d{2}-\d{4}\b')

def regex_scan(text: str) -> list[dict]:
    hits = []
    for match in EMAIL_RE.finditer(text):
        hits.append({"type": "email", "start": match.start(), "end": match.end()})
    for match in SSN_RE.finditer(text):
        hits.append({"type": "ssn", "start": match.start(), "end": match.end()})
    return hits

We maintain a curated list of ~50 patterns covering email, phone (international), credit card numbers (Luhn check), passport numbers, and national IDs for the jurisdictions we operate in. Each pattern has a false-positive rate we measure in CI.

Stage 2: ML-Based NER

Regex misses "my name is John Doe" or "address: 123 Main St". For that, we use a fine-tuned spaCy model trained on CoNLL-2003 plus synthetic GDPR-specific entities (e.g., "patient", "client", "employee ID").

import spacy

nlp = spacy.load("en_core_web_trf")  # transformer-based, ~100ms per request

def ner_scan(text: str) -> list[dict]:
    doc = nlp(text)
    hits = []
    for ent in doc.ents:
        if ent.label_ in ["PERSON", "ORG", "GPE", "EMAIL", "PHONE", "DATE_OF_BIRTH"]:
            hits.append({"type": ent.label_, "start": ent.start_char, "end": ent.end_char, "text": ent.text})
    return hits

We map spaCy's standard labels to GDPR categories: PERSON -> "name", GPE -> "location", etc. The model is loaded once at proxy startup and shared across workers via a process pool.

Decision Logic

If either stage returns a hit, the proxy logs the violation and returns HTTP 403 with a structured error body. We also redact the PII in the log (store hash + type, not plaintext).

def scan_prompt(prompt: str) -> bool:
    regex_hits = regex_scan(prompt)
    if regex_hits:
        return True  # block
    ner_hits = ner_scan(prompt)
    if ner_hits:
        return True  # block
    return False

Performance Impact

We benchmarked against a 1000-request dataset (average prompt length 512 tokens).

Adding NER increases latency by ~95ms. For most workloads that's acceptable (inference itself takes 1-5s). If you need sub-10ms proxy overhead, use a smaller model (e.g., en_core_web_sm with 5ms) or run regex only.

Audit Logging

Every blocked request is logged to a separate, append-only log store (e.g., S3 + Athena or a SIEM). The log entry contains:

• Timestamp (ISO 8601)
• User ID (anonymized hash)
• Request ID
• PII types detected (list of strings)
• Hash of the offending span (SHA-256 of the raw text, so we never store plaintext PII)
• Action taken ("blocked")

We also emit a Prometheus counter pii_blocks_total with labels for PII type and model endpoint.

Handling False Positives

If a legitimate prompt is blocked, the user gets a 403 with a reason code. We provide a /v1/pii-review endpoint where authorized users can submit a correction request. Those are logged and periodically reviewed to tune the regex patterns and NER thresholds.

Deployment

We run the proxy as a sidecar container in Kubernetes, fronted by an ingress. It's stateless (except for the model loaded in memory), so we scale horizontally. Configuration is via environment variables:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pii-proxy
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: proxy
        image: myorg/pii-proxy:1.2.0
        env:
        - name: PII_MODEL_PATH
          value: "/models/en_core_web_trf"
        - name: PII_BLOCK_ACTION
          value: "block"  # or "log" for monitoring only
        - name: PII_LOG_BUCKET
          value: "s3://audit-logs/pii/"

GDPR Compliance Checklist

• Article 5(1)(c): Data minimisation — we block PII before processing.
• Article 32: Security — we encrypt logs, never store plaintext PII.
• Article 33: Breach notification — our audit trail allows rapid impact assessment.
• Article 17: Right to erasure — we can purge logs by user hash.

Lessons Learned

1. Regex is not enough. Emails and SSNs are easy, but names like "Anna" or "Berlin" appear in non-PII contexts. The NER model catches those only when they're part of a named entity.

2. Latency matters. We initially ran NER synchronously per request. Switched to a background thread pool with pre-warmed models. P99 dropped from 200ms to 96ms.

3. Test with real data. Synthetic data won't expose edge cases like typos or mixed languages. We ran a shadow mode (log only) for two weeks to gather ground truth.

4. Don't forget response filtering. Prompts aren't the only risk — the model might generate PII. We run the same filter on the completion before returning it to the client. That's a separate endpoint, but the same code.

Future Work

We're adding a third stage: a small transformer classifier (e.g., DistilBERT) trained specifically to detect GDPR-sensitive contexts (e.g., "I am a patient of Dr. X"). That will run only if the first two stages pass, adding ~20ms but catching phrases like "my medical record number is".

Also planning to expose the filter as a gRPC service so it can be reused by other microservices (e.g., data pipelines).

Code

Full proxy implementation is open-source at github.com/yourorg/pii-proxy. It's ~600 lines of Python using FastAPI, spaCy, and prometheus_client. Contributions welcome.