Refusing to Answer: Calibrating Confidence Scores to Cut Hallucination

Hallucinations are not a bug you fix; they are a property of the sampling distribution. No amount of prompting eliminates them, because the model does not know what it does not know. What you can do is make the model say "I don't know" instead of fabricating. The trick is not to ask the model whether it knows — that is unreliable — but to measure the confidence of its own output and refuse when that confidence is low.

This article walks through a practical methodology for calibrating confidence scores on LLM outputs, setting thresholds, and building a refusal layer that catches hallucinations before they reach the user. The approach is model-agnostic and works with any transformer-based LLM served via vLLM or similar inference engines.

Why raw token probabilities are not enough

Every LLM returns token-level logits. The softmax over the vocabulary gives a probability for each token, and the product of those probabilities is the model's joint probability of the generated sequence. That number feels like confidence, but it is not calibrated. A model can assign high probability to a sequence that is factually wrong, especially for rare facts or long-tail knowledge.

Calibration means the confidence score matches the empirical accuracy. If the model says 90% confident, it should be right 90% of the time. Raw token probabilities are typically overconfident, especially for models trained with RLHF or DPO, which push probabilities toward extremes.

A common pattern is to use the sequence log-probability normalized by length as a crude confidence. But that still suffers from overconfidence. You need to calibrate the score against a validation set where you know ground truth.

What confidence signals are available

Depending on your serving stack, you can extract several signals:

• Token logprobs: vLLM can return logprobs for each generated token. Sum or average them.
• Semantic entropy: Generate multiple samples (e.g., 5) for the same prompt, cluster them by meaning, and compute entropy over the clusters. High entropy means the model is unsure.
• Self-consistency: Ask the same question multiple times and check if the answers agree. Disagreement is a strong signal of uncertainty.
• Embedding distance: Encode the generated answer and the question, measure similarity. Low similarity can indicate hallucination (though not always).

Each signal has tradeoffs. Token logprobs are cheap but need calibration. Semantic entropy is more robust but requires multiple generations, which increases latency and cost. Self-consistency is similar. Embedding distance is fast but can be misleading.

For a production system, a hybrid approach works best: use token logprobs as the primary signal, and optionally trigger a more expensive consistency check only when confidence is borderline.

The calibration procedure

The goal is to map the raw confidence signal to a calibrated probability of correctness. You need a labeled dataset: a set of prompts with known correct answers. For each prompt, generate the model's answer and compute the raw confidence. Then label whether the answer is correct (including partial credit if you're generous).

With that data, you can build a calibration curve. A common method is histogram binning: sort the confidence scores, divide into bins (e.g., 10 bins), and for each bin compute the empirical accuracy. Then you can interpolate to get a calibrated confidence for any new score.

Here's a Python sketch:

import numpy as np
from sklearn.isotonic import IsotonicRegression

def calibrate(scores, labels):
    # scores: array of raw confidence values
    # labels: array of 0/1 correctness
    iso = IsotonicRegression(out_of_bounds='clip')
    iso.fit(scores, labels)
    return iso

# Later, for a new score:
calibrated = iso.predict([new_score])[0]

Isotonic regression is non-parametric and works well when you have enough data. For smaller datasets, Platt scaling (a logistic regression on the logit of the score) is simpler and less prone to overfitting.

Setting the refusal threshold

The threshold is a business decision. You trade recall (catching hallucinations) against precision (not refusing correct answers). A common approach is to set the threshold at the point where the calibration curve crosses a target accuracy, say 95%. If the calibrated confidence is below that, refuse.

But the threshold should be per-domain. For a legal Q&A, you might want 99% accuracy and refuse more. For a creative writing assistant, you might accept 80% and refuse rarely.

You also need to consider the cost of a wrong answer. If a hallucination in your domain is expensive (medical, financial), set the threshold high. If it's cheap (entertainment), set it low.

Implementing the refusal layer

Once you have a calibrated score, you wrap the generation call. Here's a pseudo-code example using vLLM's API:

from vllm import LLM, SamplingParams

llm = LLM(model="your-model")

def generate_with_guard(prompt, threshold=0.9):
    sampling_params = SamplingParams(temperature=0.7, logprobs=1)
    output = llm.generate([prompt], sampling_params)[0]
    # Extract raw confidence: average logprob of generated tokens
    token_logprobs = output.outputs[0].logprobs
    avg_logprob = sum(lp for lp in token_logprobs) / len(token_logprobs)
    raw_conf = np.exp(avg_logprob)  # convert to probability-like
    calibrated = calibrator.predict([raw_conf])[0]
    if calibrated < threshold:
        return "I don't know.", calibrated
    return output.outputs[0].text, calibrated

In practice, you might also want to return the confidence to the caller so the UI can display it.

Tradeoffs and pitfalls

• Latency: Multiple generations for semantic entropy add latency. For real-time systems, use a single pass and rely on logprobs.
• Over-refusal: If the threshold is too high, the system becomes useless. You need to tune on real user traffic.
• Domain shift: Calibration degrades if you deploy on a different distribution than your validation set. Re-calibrate periodically.
• Model updates: Every time you fine-tune or change the model, recalibrate.

A real-world pattern from the RiNET stack

At RiNET, we run a stack of three Hetzner servers connected via a WireGuard mesh. We serve a Qwen model on vLLM, use BGE-M3 for embeddings, and store data in Postgres, Qdrant, and Neo4j. We also do nightly LoRA fine-tuning.

We have implemented a confidence-based refusal layer for our agent swarms. The calibration set is built from a curated set of prompts where we know the correct answers. We use isotonic regression on the average logprob. The threshold is set to 0.85, which balances recall and precision for our use case.

A common pattern in the industry is to combine confidence with a retrieval check: if the answer is low-confidence, query the vector database and see if a retrieved passage supports the answer. If not, refuse. This adds a second layer of guardrail.

Beyond refusal: using confidence for routing

Confidence scores are not only for refusal. You can route low-confidence queries to a human, or to a more expensive model (e.g., a larger model) for a second opinion. This is a cost-saving strategy: use the cheap model when confident, escalate when not.

Conclusion

Calibrating confidence scores is a practical, measurable way to reduce hallucinations. It requires a labeled dataset, a calibration method, and a threshold. The implementation is straightforward with vLLM's logprobs. The key is to treat confidence as a first-class signal and act on it.

Start by collecting a small validation set, compute raw logprobs, fit a calibrator, and set a threshold. You'll find that a significant fraction of hallucinations are low-confidence — and now you can catch them.