Why Our 6-Token Retrieval Beat a Full-Context LLM: A RAG Optimization Case Study

We shipped a RAG pipeline where the retrieved context is exactly six tokens. Not six sentences. Six tokens. And it beat our previous 128K-context-window LLM on every metric that mattered: latency, cost, and answer accuracy.

This is the story of that optimization. No fluff, just the architecture, the benchmarks, and the trade-offs.

The Naive Baseline

Our first production RAG system was standard: embed documents into pgvector (Postgres 15 + pgvector 0.5.2), retrieve top-5 chunks (~2000 tokens each), and stuff them into a 128K-context Mistral 7B instruct model (vLLM 0.3.2, bfloat16). The LLM would then reason over the concatenated chunks.

Results were okay. Latency: 8.2 seconds per query (p95). Cost: $0.03 per query (GPU compute). Accuracy (F1 on a held-out QA set of 500 domain questions): 0.74.

But we noticed something: the LLM was ignoring most of the context. We logged the attention patterns. On average, the model attended to only 11% of the input tokens. The rest was noise—boilerplate headers, redundant explanations, irrelevant details.

We asked: what if we don't need the LLM to read at all? What if retrieval itself becomes the inference?

The 6-Token Hypothesis

Our domain is legal contract analysis. Each contract clause has a specific label (e.g., "indemnification", "termination for convenience"). The question is always: "Does this clause exist?" The answer is binary or a short phrase.

We realized that for this task, the LLM was doing two things:

1. Retrieval: finding relevant clauses.
2. Reasoning: verifying the clause matches the question.

But the reasoning step was trivial: if the retrieved chunk contains the exact clause, answer yes. If not, no. The LLM's ability to "reason" was wasted on a lookup problem.

So we replaced the LLM with a deterministic match: after embedding-based retrieval, we extracted the exact clause label from the chunk using a regex. The output was a single token: "yes" or "no".

But even that felt like overkill. The real optimization came when we compressed the entire retrieval context into a single embedding lookup.

Architecture: Tokenizing the Retrieval

We built a two-stage pipeline:

1. Document embedding: Each contract clause (average length: 6 tokens) is stored in pgvector with its label. We used BGE-M3 (BAAI/bge-m3, 1024 dimensions) for embeddings. Index: IVFFlat with 100 lists.

2. Query embedding: The user question is embedded with the same model. Then we run a nearest-neighbor search (cosine distance) over the clause embeddings, returning the top-1 label.

That's it. The entire "reasoning" is: return the label of the nearest neighbor. The output is a single token (the label).

We call this "6-token retrieval" because the average clause is 6 tokens long. The system never sees more than those 6 tokens.

Benchmark Results

We compared the old pipeline (128K LLM) with the new pipeline (6-token retrieval) on three axes:

Accuracy improved because the LLM was sometimes confused by noisy context. The deterministic nearest-neighbor approach is immune to that: it only sees the most relevant clause.

When This Works (and When It Doesn't)

This optimization works when:

• The task is retrieval-heavy, reasoning-light. If the answer is directly present in the document (fact extraction, clause matching, entity resolution), you don't need an LLM.
• The granularity of retrieval matches the answer granularity. Our clauses were atomic. If your answers span multiple chunks, you'll need composition.
• The embedding space is well-separated. BGE-M3 gave us tight clusters per clause type. If your data is noisy, accuracy may drop.

It fails when:

• Questions require synthesis across multiple documents. E.g., "Summarize the payment terms across all contracts." Then you need the LLM.
• The answer is implicit. E.g., "Is there a risk of liability?" That needs reasoning.
• The label space is large and overlapping. 10,000 similar clauses will choke a simple nearest-neighbor.

Implementation Details

We used the following stack:

• Embedding model: BGE-M3 via llama.cpp (commit 3b7e8f4) on CPU. Batch size 1 for queries, batch size 256 for indexing.
• Vector store: pgvector 0.5.2 on Postgres 15. Table schema:

CREATE TABLE clauses (
  id SERIAL PRIMARY KEY,
  label TEXT NOT NULL,
  embedding vector(1024) NOT NULL
);
CREATE INDEX idx_clauses_embedding ON clauses USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);

• Query logic (Python):

import psycopg2
from llama_cpp import Llama

model = Llama("BAAI/bge-m3-Q4_K_M.gguf", embedding=True)

def query_clause(question):
    q_emb = model.create_embedding(question)["data"][0]["embedding"]
    conn = psycopg2.connect("dbname=contracts")
    cur = conn.cursor()
    cur.execute("""
        SELECT label FROM clauses
        ORDER BY embedding <=> %s::vector
        LIMIT 1
    """, (q_emb,))
    return cur.fetchone()[0]

• Deployment: systemd service with a uvicorn server. No GPU needed. Total RAM usage: 2 GB for the embedding model (Q4_K_M).

Lessons Learned

1. Context is not always your friend. More tokens means more noise. For many tasks, the optimal context size is exactly the size of the answer.
2. LLMs are overkill for retrieval. If you can express your problem as a nearest-neighbor search, do it. You'll get faster, cheaper, and often more accurate results.
3. Measure attention. We wouldn't have found this optimization without logging attention patterns. Don't assume the LLM uses the context you give it.
4. pgvector is production-ready. We handle 10M clauses on a single 8GB RAM instance. Queries run in under 50ms.

The Trade-Off

We lost flexibility. The old system could answer arbitrary questions. The new one only answers "which clause is this?" But for our use case, that's 95% of queries. The remaining 5% we route to a small LLM (phi-3-mini) on CPU.

Total system cost dropped from $0.03/query to $0.0015/query (including the fallback). Latency dropped from 8s to 0.8s (p95). User satisfaction went up.

Conclusion

This is not a general solution. It's a case study for a specific class of problems: lookup tasks where the answer is a single atomic fact. If that's your domain, stop throwing context at the problem. Start by retrieving the exact answer.

Our 6-token retrieval pipeline proves that sometimes, less is more. Much more.