Deterministic replay of LLM inference with temperature 0 and seed: what breaks in practice

You set temperature=0, you fix the seed, you expect the same output every time. That's the mental model most developers have when they move from OpenAI's API to self-hosted inference. It's a nice model. It's also wrong.

In practice, deterministic replay of LLM inference is a research problem, not a configuration option. I've spent the last year building a self-hosted inference stack for a regulated industry. We needed exact replay for audit trails. We got it working, but only after understanding every layer where determinism breaks.

This post is a field guide to those layers. If you're using vLLM, llama.cpp, or any other inference engine, and you expect bit-identical outputs from a fixed seed, read this before you ship.

The naive setup

Here's what most people start with. You're using vLLM, you pass --seed 42 and set temperature=0 in the sampling params. You run the same prompt twice. You get different outputs.

Why? Because temperature=0 doesn't mean "greedy" in most engines. It means "scale logits by 1/temperature" — and when temperature is exactly 0, the division is undefined. So engines treat temperature=0 as a special case, often mapping it to greedy decoding. But greedy decoding still has to pick the argmax token. That's deterministic in theory. The non-determinism creeps in from the hidden state computation, not the sampling.

Layer 1: GPU non-determinism

GPUs are not deterministic. Floating-point operations like addition and multiplication are not associative. The order of operations matters. In a matrix multiplication, the accumulation order can vary based on the kernel implementation, the tile size, and even the number of threads scheduled.

NVIDIA GPUs have a known issue: some operations are non-deterministic by default. For example, cudnn and cublas have deterministic modes, but they're not enabled by default because they're slower. In PyTorch, you can set torch.use_deterministic_algorithms(True), but that only affects certain operations. For inference, the forward pass of a transformer involves many operations that are not covered.

In practice, this means that even with the same weights, the same input, and the same seed, the hidden states can differ by tiny floating-point errors. These errors are amplified through the layers, and eventually they cause a different token to be sampled.

I've seen this happen with vLLM on A100s. We ran the same prompt 100 times with temperature=0 and seed=42. We got 3 different outputs. The difference appeared at token 15, and after that the sequences diverged completely.

Layer 2: Batching and sequence scheduling

Even if the GPU operations were deterministic, the order in which sequences are processed is not fixed. In vLLM, the scheduler batches multiple sequences together to improve throughput. The composition of the batch can change depending on the arrival order of requests, the current memory usage, and the scheduling policy.

The forward pass for a batch is a single matrix multiplication. If you have two sequences with different lengths, the padding changes the computation. More importantly, if you have a batch of size 2 vs a batch of size 4, the matrix multiplication kernel may use a different algorithm, which can produce different floating-point results.

So even if you run the same prompt with the same settings, if there are other requests in the system, the batch composition changes, and the computation for your prompt changes.

This is the biggest practical killer. In a production server, you can't guarantee that each request is processed in isolation. To get deterministic replay, you have to control the batching.

Layer 3: Kernel selection and hardware

The choice of kernel can change the result. For example, vLLM uses different kernels for different attention implementations (e.g., xformers, flash-attention, flashinfer). These kernels have different numerical properties. FlashAttention is faster but uses a different accumulation order than standard attention.

Even on the same GPU, the driver version or the CUDA toolkit version can change the kernels. We upgraded from CUDA 11.8 to 12.1, and our outputs changed for the same model and seed.

Also, the hardware itself matters. Different GPU architectures (e.g., A100 vs H100) have different instruction sets and different floating-point behavior. If you're replicating an inference on a different GPU, don't expect the same output.

Layer 4: Model loading and quantization

If you're using quantization (like 8-bit or 4-bit), the quantization process can introduce non-determinism. The scales and zero points are computed from the weights, and the computation may be done in a different order depending on the library and the hardware.

Even with the same model file, loading it in a different order (e.g., if the file is read in parallel) can produce different results if the loader uses any kind of reduction.

In practice, if you want deterministic replay, you need to load the model in a single-threaded manner, and you need to use the same quantization configuration every time.

Layer 5: Sampling implementation

Even with greedy decoding, the sampling step itself can be non-deterministic. Some engines use a random number generator to break ties when multiple tokens have the same logit. If the RNG is not seeded properly, or if it's used in a multi-threaded context, the tie-breaking can vary.

More importantly, if you set temperature=0 but the engine still uses a sampling distribution (like top-p or top-k), the filtering can be deterministic, but the actual draw is not. Some engines treat temperature=0 as a special case that forces greedy, but not all.

In vLLM, temperature=0 is handled as greedy decoding, but the implementation may still use the RNG for other purposes, like in speculative decoding.

Layer 6: Multi-threading and process scheduling

Even if all the above are fixed, the order of operations in a multi-threaded environment can vary. For example, if the engine uses multiple threads for the forward pass, the reduction order can change. This is especially true for CPU inference (llama.cpp) where the number of threads and the scheduling can affect the result.

We tested llama.cpp with --seed 42 and --temp 0 on the same machine, same prompt, and got different outputs across runs. The difference was due to the thread scheduling and the way the cache is used.

What actually works: practical strategies

After hitting these walls, we developed a set of strategies to achieve deterministic replay in our production environment. They are not perfect, but they work for our use case.

Strategy 1: Use a single request per process

For critical replay, we run a separate inference process with no other requests. This eliminates batching effects. We use vLLM in offline mode (LLM class) and set --max-num-seqs 1. This ensures that the batch size is always 1, and the computation is the same.

Strategy 2: Pin the GPU and use deterministic kernels

Set CUDA_LAUNCH_BLOCKING=1 to force synchronous execution. This makes debugging easier and can reduce some non-determinism. Also, use torch.use_deterministic_algorithms(True) if you're using PyTorch. For vLLM, you can set the environment variable VLLM_ATTENTION_BACKEND to a fixed backend (e.g., XFORMERS) and ensure the same kernel is used.

In our case, we use FlashAttention 2, but we set the flash_attn backend and ensure the same kernel version is installed. We also set --dtype float16 and avoid mixed-precision variations.

Strategy 3: Freeze the software stack

We containerize the entire inference stack with specific versions of CUDA, PyTorch, vLLM, and the model. We use a lock file for the dependencies. We also pin the GPU driver version in the base image.

Strategy 4: Use a custom sampling function

For the final token selection, we don't rely on the engine's sampling. We modify the code to always take the argmax, and we disable any RNG usage. In vLLM, we can do this by setting temperature=0 and top_p=1.0, and also setting seed to a fixed value. But we also ensure that the engine doesn't use speculative decoding.

Strategy 5: Validate with a hash

We compute a hash of the output tokens and compare it against a known-good hash. If the hash differs, we retry with a different configuration. This is not a fix, but it's a detection mechanism.

The alternative: accept non-determinism

For many use cases, deterministic replay is not necessary. If you're building a chatbot, you don't need bit-identical outputs. But for auditing, testing, and regression testing, you do.

If you're testing prompts, you can use a different approach: instead of comparing exact outputs, compare semantic similarity using embeddings. That's more robust and doesn't require deterministic inference.

Conclusion

Deterministic replay of LLM inference is not a matter of setting temperature=0 and a seed. It's a matter of controlling the entire software and hardware stack, and even then, you may not achieve perfect reproducibility.

If you need it, you have to invest in a controlled environment: single-request processes, pinning kernels, freezing versions, and possibly modifying the engine's sampling code. It's a lot of work, but it's possible.

I hope this post saves you some of the debugging time we spent. If you've solved this in a different way, I'd love to hear about it.