Timeouts, Not Knowledge: When Legal-RAG Evaluators Measure Latency

The evaluator said the answers were wrong. The citations were right, but the evaluator never saw them. The system was a legal question-answering pipeline built on a retrieval-augmented generation (RAG) stack: a vector store for semantic search, a graph database for entity relationships, and a large language model for synthesis. Every answer was supposed to include deterministic citations—exact paragraph numbers, statute references, and case law links—pulled from the retrieved context. The evaluation harness measured response correctness by checking those citations against a gold standard. But the harness had a timeout. And the timeout was killing the pipeline before the citations could be delivered.

This is a post-mortem about a class of failure that is common in production RAG systems but rarely discussed: the evaluator measures timeouts instead of knowledge. When your evaluation harness sets a hard latency budget, and your system has a slow path for generating deterministic citations, you will see false negatives. The solution is not to make the model smarter. It is to understand where the latency comes from, and how the proxy layer between the client and the model can silently truncate your responses.

The Pipeline and the Hidden Slow Path

The system under discussion is a legal-RAG service that answers questions like "What is the statute of limitations for breach of contract in New York?" The pipeline works in stages:

1. Query understanding: Parse the question, extract entities, and classify the legal domain.
2. Retrieval: Query a vector store (e.g., Qdrant) for semantically similar passages, and a graph database (e.g., Neo4j) for entity relationships and case law links.
3. Generation: Feed the top-k passages into a large language model (e.g., Qwen) via vLLM, with a prompt that instructs the model to produce a structured answer.
4. Deterministic citation extraction: After the model generates a draft, a post-processing step extracts citations from the draft and validates them against the retrieved context. This step is deliberately deterministic—it uses regex patterns and exact string matching to identify paragraph numbers, statute IDs, and case citations. The idea is to avoid hallucinated citations.

The deterministic citation extraction is the slow path. It runs after the model returns, and it can add seconds to the response time, especially when the context is large and the regex patterns are complex. In a typical setup, the model might return in 1-2 seconds, but the citation extraction can take another 1-3 seconds, depending on the number of passages and the complexity of the legal references.

The Evaluator and Its Timeout

The evaluation harness sends a question to the service, waits for a response, and then checks the response against a gold standard. The harness has a timeout—commonly set to 10 seconds or 15 seconds. If the service does not respond within that time, the harness marks the answer as incorrect, regardless of whether the eventual response would have been correct.

In this post-mortem, the timeout was set to 10 seconds. The service, when running locally, typically responded in 5-6 seconds. But when deployed behind nginx as a reverse proxy, the response time ballooned. The evaluator saw timeouts and marked everything as a failure. The team initially assumed the model was underperforming, but the real issue was in the proxy layer.

nginx proxy_read_timeout: The Silent Killer

nginx is a common reverse proxy for web services. It has several timeout directives that control how long it waits for upstream responses. The critical one is proxy_read_timeout, which sets the timeout for reading a response from the upstream server. The default value is 60 seconds, but many teams set it to something lower, like 10 seconds, to fail fast.

In this case, the nginx configuration had proxy_read_timeout 10s;. The upstream service (the RAG pipeline) took 5-6 seconds to generate the response, but the response was streamed in chunks. The model generates tokens one by one, and vLLM streams them to the client. However, the deterministic citation extraction runs after the model finishes generating, and it requires the full response to be buffered. So the service does not send the final response until the citation extraction is complete. In the meantime, nginx has already started the clock on proxy_read_timeout. If the upstream does not send any data within 10 seconds, nginx closes the connection with a 504 Gateway Timeout.

Here is a simplified nginx configuration that exhibits the problem:

server {
    listen 80;
    server_name rag.example.com;

    location / {
        proxy_pass http://rag_backend;
        proxy_read_timeout 10s;
        proxy_connect_timeout 5s;
        proxy_send_timeout 10s;
    }
}

In this setup, the backend service is a FastAPI app that waits for the full response before sending anything. The model generation takes 5 seconds, but the citation extraction takes another 3 seconds. The total is 8 seconds, which is under the 10-second timeout. But there is a subtlety: the timeout is not measured from the start of the request; it is measured from the last read from the upstream. If the backend does not send any data for 10 seconds, nginx times out. In practice, the backend sends nothing until the response is fully ready, so the clock starts immediately. If the total time exceeds 10 seconds, the request fails.

But why would the total time exceed 10 seconds when the pipeline takes 8 seconds? Because the pipeline time is not constant. The vector store and graph database queries can be slow under load. The model can be slow due to queueing. And the citation extraction can be slow when the context is large. In this post-mortem, the team observed that the 95th percentile response time was 12 seconds, which exceeded the timeout.

The Deterministic Citation Latency

Why is deterministic citation extraction so slow? It is not because of the regex matching itself—that is usually fast. The slowness comes from the validation step. The extraction process does the following:

1. Parse the model output to identify citation candidates (e.g., "Section 2-201" or "Roe v. Wade").
2. Look up each candidate in the retrieved context to verify that the citation actually appears in the source documents.
3. Resolve ambiguity—for example, "Section 2" might appear in multiple statutes, so the system needs to disambiguate using the surrounding text.
4. Format the citation in a standard legal format.

Step 2 is the bottleneck. If the retrieved context contains 20 passages, and each passage has multiple potential citations, the system might perform dozens of lookups. Each lookup involves a string search or a database query. In a naive implementation, this can take seconds.

A common optimization is to pre-index the retrieved context into a hash map of citation strings to passage IDs. Then the lookup is O(1). But even with that, the parsing and disambiguation can be slow if the model output is noisy.

The Evaluation Failure

The evaluation harness reported that the system had a 40% accuracy rate. The team was puzzled because the model seemed to produce good answers when they tested manually. They dug into the logs and found that most of the failures were timeouts. The harness was not measuring knowledge; it was measuring latency. The system was failing because it could not respond within the timeout, not because the answers were wrong.

This is a critical lesson: evaluation metrics are only as good as the conditions under which they are measured. If your evaluator has a hard timeout, you need to ensure that your system can respond within that timeout consistently. Otherwise, you are not evaluating the system's ability to answer questions; you are evaluating its ability to fit within a latency budget.

Fixing the Proxy Layer

The first fix is to increase proxy_read_timeout to a value that accommodates the worst-case response time. In this case, setting it to 30 seconds would solve the immediate problem. But that is a band-aid. The real issue is that the system is too slow.

A better approach is to make the citation extraction faster. Here are some strategies:

• Cache the extraction results: If the same question is asked multiple times, the citations are likely the same. A cache can avoid recomputing the extraction.
• Parallelize the validation: Instead of checking citations sequentially, use concurrent lookups. In Python, you can use asyncio.gather or a thread pool.
• Reduce the context size: If the retrieval returns too many passages, the citation extraction has to scan more text. Tighten the retrieval to return only the most relevant passages.
• Use a faster model: The model generation time dominates the total latency. If the model is too slow, consider using a smaller model or quantizing it.

But there is a more fundamental design change: stream the response and run citation extraction on the fly. Instead of waiting for the full model output, you can start extracting citations as soon as the model produces a sentence. This is more complex, but it reduces the perceived latency.

The Streaming Alternative

One approach is to have the backend stream the response to the client, and have the client (or a middleware) perform the citation extraction in a separate pass. This changes the architecture:

• The model generates a response and streams it to the client immediately.
• The client displays the response, but the citations are not yet validated.
• A background task runs the citation extraction and updates the response with validated citations.

This way, the evaluator sees a response within the timeout, but the citations may be incomplete. The evaluator could be configured to accept incomplete citations, or the system could send a second response with the final citations.

However, this approach has trade-offs. The evaluator might expect the citations to be present in the first response. If the evaluator is strict, it will mark the answer as incorrect because the citations are missing.

The Latency Budget Approach

A more systematic solution is to define a latency budget for each stage of the pipeline. For example:

• Query understanding: 200 ms
• Retrieval: 500 ms
• Model generation: 2 seconds
• Citation extraction: 1 second
• Total: 3.7 seconds

Then you can measure each stage and identify the bottleneck. In this case, the model generation and citation extraction are the main contributors. You can then optimize those stages.

The Post-Mortem Conclusion

The post-mortem revealed that the system was not as bad as the evaluator suggested. The real problem was a misconfiguration in the proxy layer and a lack of attention to the latency of the deterministic citation extraction. The team fixed the nginx timeout, optimized the extraction code, and re-ran the evaluation. The accuracy improved dramatically, not because the model changed, but because the evaluator could now receive the responses.

This is a common story in RAG systems: the evaluation harness is often the most demanding client, and it will expose any latency issue in the system. If you are building a RAG system, you should test it with your evaluation harness early and often. Do not wait until the end to discover that your proxy is dropping requests.

Key Takeaways

• Evaluators measure latency, not just knowledge. If your system is slow, the evaluator will mark it as incorrect.
• nginx proxy timeouts can silently kill your requests. Always check proxy_read_timeout and related directives.
• Deterministic citation extraction is a hidden latency cost. Profile it and optimize it.
• Streaming can help, but it may not satisfy strict evaluators. Consider the trade-offs.
• Define a latency budget and measure each stage. This helps you identify bottlenecks.

In the end, the system was fixed. But the lesson remains: when your evaluator measures timeouts instead of knowledge, you need to fix your latency, not your model.