Claim Tickets: Verifiable Evidence for Every Agent Assertion

A methodology for making LLM agent claims auditable through evidence pointers and verify-before-claim discipline.

by

An agent that says "the config was updated" without a pointer is not reporting; it is guessing. Claim tickets fix that by making every assertion carry a verifiable evidence pointer. The pattern is simple: no claim ships without a ticket, and no ticket ships without a pointer that a verifier can resolve.

This is a methodology piece. It describes a pattern teams converge on when they need to audit agent output after the fact, not a specific product.

What is a claim ticket, exactly?

A claim ticket is a small structured record that binds an assertion to its evidence. It has four fields that matter:

  • claim: the atomic assertion, one sentence, no hedging.
  • pointer: a resolvable reference to the source (file path plus line range, URL plus selector, message ID, row key).
  • extractor: the function that pulled the evidence from the pointer.
  • verifier: the check that confirms the extractor's output actually supports the claim.

Everything else (timestamps, agent ID, run ID) is bookkeeping. The four fields above are the contract. If any is missing, the claim is not a claim; it is a hypothesis.

Why do agents need tickets instead of citations?

Citations are prose. Tickets are data. A citation like "see the docs" cannot be machine-checked. A pointer like {"path": "src/config.py", "lines": [42, 58]} can be resolved, hashed, and diffed on the next run.

The practical difference shows up in forensics. When an agent's output is challenged a week later, a citation forces a human to re-read the source. A ticket lets a script re-resolve the pointer and confirm the evidence still exists and still says what the agent claimed.

A common pattern is to store tickets alongside the agent's output in the same record. The output is the human-facing artifact; the tickets are the audit trail.

How should a pointer be shaped?

Pointers should be narrow and typed. A pointer that names a whole document is nearly useless; a pointer that names a line range is checkable. Teams typically define a small pointer vocabulary:

{"kind": "file_range", "path": "src/config.py", "start": 42, "end": 58}
{"kind": "http_selector", "url": "https://example.com/pricing", "selector": "#tier-enterprise"}
{"kind": "message", "channel": "#incidents", "id": "C0123ABC"}
{"kind": "sql_row", "table": "orders", "pk": 918273}

Each kind has exactly one extractor. The extractor is deterministic: same pointer, same bytes out. That determinism is what makes verification cheap. If the extractor needs an LLM to interpret the pointer, the pointer is too vague.

Where does verify-before-claim fit in the loop?

The discipline is verify-before-claim, not claim-then-verify. The agent must resolve the pointer and run the verifier before the claim enters its output buffer. If verification fails, the claim is downgraded to a question or dropped.

A minimal harness looks like this:

def emit_claim(claim: str, pointer: dict, verifier) -> dict:
    evidence = extract(pointer)          # deterministic
    if not verifier(claim, evidence):
        raise ClaimRejected(claim, pointer)
    return {
        "claim": claim,
        "pointer": pointer,
        "evidence_hash": sha256(evidence),
    }

The evidence_hash is the forensic anchor. On a later audit, re-extract and compare hashes. If the hash changed, the source changed; the claim may no longer hold. That is exactly the signal an auditor needs.

Try this next: take one agent output from last week and ask which of its claims would survive a re-resolution today. The ones that would not are the ones that needed tickets.

What does a verifier actually check?

A verifier answers one question: does the extracted evidence support the claim as written? There are three tiers, and teams usually need all three:

  • Presence: the pointer resolves and the extractor returns non-empty output.
  • Containment: the claim's key terms appear in the evidence (string or token match).
  • Entailment: a small model or rule set judges whether the evidence entails the claim.

Presence is cheap and catches broken pointers. Containment catches paraphrase drift. Entailment catches the hard cases, where the evidence is present but does not actually support the assertion. Skipping entailment is the most common failure mode; teams ship presence-only verifiers and then wonder why confident wrong claims still get through.

How do tickets change the eval harness?

With tickets, the eval harness stops scoring prose and starts scoring ticket coverage. Two metrics matter:

  • Claim coverage: fraction of assertions in an output that carry a ticket.
  • Ticket resolution rate: fraction of tickets whose pointers resolve and whose verifiers pass on re-run.

These are cheap to compute and hard to game. An agent that emits fewer claims but tickets all of them is more useful than one that emits many unticketed claims. The harness should reward the former.

A pattern worth adopting: run the eval twice, once against the live sources and once against a frozen snapshot. Divergence between the two runs tells you which claims are sensitive to source drift. That is the difference between a stable eval and a flaky one.

What breaks first when you adopt this?

The pointer vocabulary is the first casualty. Agents will try to invent pointer kinds ("the general area of the codebase") that no extractor can resolve. Reject unknown kinds at the ticket boundary; do not let the agent negotiate.

The second failure is verifier scope creep. A verifier that tries to judge the whole output instead of one claim becomes a second hallucination surface. Keep verifiers atomic: one claim, one evidence blob, one boolean.

The third is storage. Tickets are small but numerous. A run that emits a thousand claims produces a thousand tickets. Plan for that in the schema from day one, or you will be backfilling later.

The trade-off you are actually making

Claim tickets cost latency and tokens. Every claim now requires an extraction and a verification step before it can be emitted. That is real overhead, and it is not free.

The trade is auditability. Without tickets, an agent's output is a story. With tickets, it is a set of assertions with receipts. For any agent whose output drives a decision, the receipts are worth more than the latency.

Start with one agent, one pointer kind, and one verifier. Ship it. Then ask yourself: which claim in your last agent run would you be unable to defend under audit? That is the one to ticket first.

#agent-swarms#eval-harness#forensics#llm-hallucination#verify-before-claim
Share — X / Twitter · LinkedIn · HN · Email
Damir Radulić
Founder of RiNET. On the Croatian internet since 1996 (Kvarner Net). In Amsterdam now, building autonomous AI infrastructure that runs on Monday morning when nobody's watching — sovereign stacks, agent swarms, LoRA fine-tuning, civic-intelligence platforms.