Deterministic Replay of Agent Decisions: Beyond Temperature 0

When an agent misbehaves, the first instinct is to reproduce the failure. You set temperature to 0, seed the RNG, and rerun. And yet the agent does something different. This is the hidden state trap: temperature 0 only controls one layer of stochasticity. The model, the sampling kernel, the batching logic, the vector store, even the network — each carries its own hidden state that can derail a supposedly deterministic replay.

This article is about designing agent systems for deterministic replay. Not just setting a seed, but understanding every source of nondeterminism between the prompt and the action. I'll cover the obvious and the subtle, with concrete patterns that have worked in practice.

Why Temperature 0 Isn't Enough

Temperature 0 is a common reflex. It makes the model pick the highest-probability token at each step. But the greedy path is not the only source of nondeterminism.

First, the model itself is not a pure function. The forward pass involves floating-point arithmetic that can vary across hardware, drivers, and even batch compositions. A GPU kernel that uses a different reduction order can produce slightly different logits, and those tiny differences can tip a token choice when the top two probabilities are close.

Second, the sampling kernel may still introduce randomness even at temperature 0. Some implementations use a stochastic rounding or a top-k filter that breaks ties randomly. Others might have a bug where temperature 0 is treated as a very small positive number, causing numerical instability.

Third, the agent loop is full of hidden state: the conversation history, the tool call results, the vector store query results, the order of items in a retrieved list. Any variation in these inputs changes the prompt, and the prompt changes everything.

The Hidden State Trap

The trap is that you think you've controlled the inputs, but you haven't. You set the seed, you set the temperature, you run the same script. But the script is not the whole story.

Consider a simple agent that retrieves documents from a vector store. The retrieval is often approximate: HNSW, IVF, or other ANN indexes are nondeterministic by design. The same query can return slightly different results depending on the index state, the number of threads, or the random seed used during index construction. If your agent uses the retrieved documents to craft its next prompt, the prompt changes, and the output changes.

Similarly, if your agent makes a tool call that hits an external API, the response might vary. Even a local tool that reads a file or queries a database can return different results if the underlying data changes. The agent's state is not just the model's hidden state; it's the entire environment.

Designing for Replay

The goal is to make a single run reproducible: given the same initial state and the same sequence of external events, the agent produces the same sequence of actions. This requires controlling every source of nondeterminism.

1. Control the Sampling Kernel

First, use a sampling kernel that is deterministic at temperature 0. Most frameworks allow you to set do_sample=False or greedy=True to force greedy decoding. This is better than relying on temperature 0 alone.

# Example: using Hugging Face Transformers with greedy decoding
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B")

inputs = tokenizer("The capital of France is", return_tensors="pt")
outputs = model.generate(
    **inputs,
    do_sample=False,  # greedy, not temperature 0
    max_new_tokens=10
)
print(tokenizer.decode(outputs[0]))

But greedy decoding is not enough if the model is running on a GPU with nondeterministic kernels. You might need to set environment variables or use deterministic algorithms.

# For PyTorch, you can enable deterministic mode
export CUBLAS_WORKSPACE_CONFIG=:4096:8

import torch
torch.use_deterministic_algorithms(True)

These settings force PyTorch to use deterministic kernels, at the cost of some performance. It's a tradeoff: deterministic replay vs. speed. For production, you might want to run in deterministic mode only during testing.

2. Freeze the Environment

The agent's environment includes the model weights, the tokenizer, the vector store, and any external data. To make replay possible, you need to freeze all of them.

• Model weights: Use a fixed checkpoint, not a continuously updated model. If you fine-tune nightly, as we do with LoRA, you need to pin the exact weights for a given replay.
• Tokenizer: Tokenizers can change between versions. Pin the tokenizer version.
• Vector store: If you use an approximate index, you need to either use a deterministic index (e.g., brute-force) or save the exact index state. For Qdrant, you can use the quantization and hnsw settings to control determinism, but the simplest is to use a brute-force search for replay.
• External APIs: You cannot control external APIs. Instead, you need to record the responses and replay them. This is called a mock or a VCR (video cassette recorder) pattern.

3. Record and Replay External Interactions

For any tool call that hits the network, a database, or a file system, you should record the exact request and response. Then, during replay, you intercept the call and return the recorded response.

# Pseudocode for a replay harness
class ReplayInterceptor:
    def __init__(self, recording):
        self.recording = recording
        self.calls = []

    def call_tool(self, tool_name, **kwargs):
        key = (tool_name, json.dumps(kwargs, sort_keys=True))
        if key in self.recording:
            return self.recording[key]
        else:
            # In live mode, call the real tool and record
            response = real_call(tool_name, **kwargs)
            self.recording[key] = response
            return response

This pattern is common in testing HTTP clients, and it works for agent tool calls too. You record a session, then replay it to verify the agent's behavior.

4. Control Batching and Ordering

If your agent processes multiple items in a batch, the order of the batch can affect the results because of padding and attention masks. For deterministic replay, you need to ensure the same batch composition and order.

For example, if you use vLLM to serve the model, the batching is dynamic. You need to set max_num_seqs and other parameters to a fixed value, and you might need to disable continuous batching for replay. Alternatively, you can process requests one at a time, which is slower but deterministic.

5. Snapshot the Hidden State

If you want to replay from an arbitrary point in the agent's execution, you need to snapshot the entire state: the model's KV cache, the conversation history, the vector store state, and any other variables. This is like a database checkpoint.

In practice, this is hard. The KV cache is large and tied to the GPU. But you can snapshot the conversation history and the retrieved documents, and re-run the model from scratch. That's often enough.

A Practical Replay Harness

Here's a design for a replay harness that works for most agent systems:

1. Log everything: Every input and output of every step: prompts, model outputs, tool calls, tool responses, retrieval results.
2. Record external calls: Use the interceptor pattern to record all non-deterministic external interactions.
3. Provide a replay mode: In replay mode, the agent uses the recorded data instead of real calls.
4. Compare outputs: Run the agent twice with the same recorded data and compare the final actions. If they differ, you have a nondeterminism bug.

# Example: logging and replaying a simple agent step
class Agent:
    def __init__(self, model, tools, replay_data=None):
        self.model = model
        self.tools = tools
        self.replay_data = replay_data or {}

    def step(self, prompt):
        # Log the prompt
        print(f"PROMPT: {prompt}")

        # Generate a response (deterministic greedy)
        response = self.model.generate(prompt, do_sample=False)
        print(f"RESPONSE: {response}")

        # If there's a tool call, execute it
        if "tool_call" in response:
            tool_name, args = parse_tool_call(response)
            # In replay mode, use recorded response
            if self.replay_data:
                tool_result = self.replay_data[f"{tool_name}:{args}"]
            else:
                tool_result = self.tools[tool_name](**args)
            print(f"TOOL RESULT: {tool_result}")
            # Append to conversation and continue...

The Tradeoff: Determinism vs. Performance

Enforcing determinism often costs performance. Deterministic GPU kernels are slower. Brute-force vector search is slower than HNSW. Recording and replaying external calls adds I/O overhead. You need to decide where determinism matters.

In our stack, we run three Hetzner servers with a WireGuard mesh. The nightly LoRA fine-tuning updates the model weights. If we want to reproduce an agent's behavior from last night, we need to pin the exact LoRA weights. That means storing the adapter file and loading it explicitly.

For testing, we run a separate deterministic environment with a fixed model and a recorded set of interactions. For production, we accept a bit of nondeterminism because the benefits of faster inference outweigh the need for exact replay.

Hidden State in Vector Stores

Vector stores are a common source of nondeterminism. Approximate nearest neighbor search (ANN) is inherently random. Even with a fixed seed, the index construction can produce different results on different runs.

To make vector search deterministic, you have a few options:

• Use exact search (brute-force) for replay. This is O(n) but deterministic.
• Use a deterministic ANN algorithm, like a tree-based method that doesn't rely on random initialization.
• Save the index state and reload it exactly.

In practice, exact search is fine for small datasets. For large datasets, you might need to accept nondeterminism and design your agent to be robust to small variations in retrieval results.

The Role of Seeds

Seeds are a necessary but insufficient condition for determinism. You need to seed every random generator in the system: the model's RNG, the sampling kernel's RNG, the data loader's RNG, the vector store's RNG, and any other library that uses randomness.

But seeds only work if the algorithm is deterministic given the same seed. If the algorithm itself is nondeterministic (e.g., due to floating-point non-associativity), a seed won't help.

Testing for Determinism

To test for determinism, run the same scenario twice and compare the outputs. This is a regression test. You can automate it in your CI pipeline.

# Test for determinism
def test_deterministic_replay():
    # Set up the agent with a fixed seed and recorded data
    agent1 = create_agent(seed=42, replay_data=recorded)
    agent2 = create_agent(seed=42, replay_data=recorded)

    # Run both agents on the same input
    output1 = agent1.run("What is the weather?")
    output2 = agent2.run("What is the weather?")

    # Assert they are identical
    assert output1 == output2

If this test fails, you have a nondeterminism source. You need to trace it down and fix it.

Conclusion

Deterministic replay is not about temperature 0. It's about controlling every source of nondeterminism in the system. This is hard, but it's necessary for testing and debugging agents. By recording external interactions, freezing the environment, and using deterministic kernels, you can make your agents reproducible.

The payoff is significant: you can debug failures, verify fixes, and build confidence in your agent's behavior. It's worth the engineering effort.

In the next article, I'll discuss how to design agent state machines that are easier to replay, and how to integrate deterministic replay into your CI/CD pipeline.

Until then, happy debugging.