Why Setting temperature=0 Doesn’t Make an LLM’s Output Deterministic, and What Actually Does

Setting temperature=0 is the standard advice for getting consistent output from an LLM, and it does remove one source of randomness: sampling. What it doesn’t remove is the randomness introduced by how the model’s math actually runs on GPU hardware, which depends on things that have nothing to do with your prompt or your parameters — including what other requests happen to be sharing the server at that moment. Two calls with identical temperature, identical seed, and identical prompt can still come back with different text, and the reason is architectural, not a bug you can prompt your way around.

What temperature=0 actually removes

Temperature controls how the model samples from its probability distribution over the next token. At temperature 0, sampling is replaced with greedy decoding — always pick the single most likely next token — which removes the randomness that comes from that sampling step. That’s a real, meaningful reduction in variance. It is not, on its own, a guarantee of identical output, because sampling randomness was never the only source of variation in the first place.

What the seed parameter adds, and what it still doesn’t promise

OpenAI’s own cookbook on reproducible outputs describes the seed parameter as making the system “make a best effort to sample deterministically,” with the explicit caveat that determinism is “not guaranteed.” In the cookbook’s own test, five requests sent with the same seed and temperature 0 still produced varied text — the average embedding distance between responses dropped to about 0.045 with a seed, versus about 0.114 without one, but it did not reach zero. The page attributes any remaining difference partly to system_fingerprint, a value that identifies the backend configuration used to generate a completion and that can change when the provider updates its serving infrastructure, which the cookbook says happens a few times a year. Matching temperature and seed narrows the variance. It does not close it, and the cookbook says so directly, attributing the remainder to “the inherent non-determinism” of the models themselves.

The actual mechanism: it’s not just floating-point rounding

The common shorthand explanation is “floating-point addition isn’t associative, so GPUs give slightly different sums each time.” That’s true as a statement about floating-point math, but it doesn’t by itself explain why a fixed model, fixed input, and fixed decoding strategy would produce different results on different runs. A deep technical write-up from Spheron’s engineering blog on vLLM batch invariance makes the sharper point: a fixed sequence of floating-point operations on the same inputs produces the same output every time. What changes between runs isn’t the arithmetic rule — it’s which sequence of operations actually executes, and that sequence is chosen by the GPU kernel based on the shape of the batch it’s currently processing.

Production inference servers use continuous batching: the set of requests sharing a GPU at each decoding step is reformed constantly as requests arrive, finish, or get added mid-stream. Your request’s “batch neighbors” change from one decode step to the next, and that batch composition is exactly what several key kernels use to decide how to do their math:

• RMSNorm — how the normalization reduction is split across GPU threads depends on how many rows are being reduced together at once, and a different split changes the order values get summed in.
• Matrix multiplication (GEMM) — the tiling and accumulation strategy a GPU kernel picks is chosen based on the operation’s shape, which is partly a function of batch size, changing the order partial sums are combined.
• Attention with split-KV — the number and size of key-value cache chunks adapts to batch size and cache length, changing how the partial attention outputs from each chunk get combined.

None of these reorderings change what the “correct” mathematical answer is. They change the order floating-point additions happen in, and because floating-point addition genuinely isn’t associative, a different order produces a slightly different sum at the bit level. That tiny difference in one layer’s output becomes the input to the next layer, and compounds across the model’s depth — eventually large enough to flip which token is the single most likely next one, even under pure greedy decoding.

Why this means the same prompt can get different answers on different days

Put together, this explains a pattern a lot of people notice and can’t explain: the exact same prompt, temperature 0, same seed, gives a consistent answer for a while and then one day comes back different — with no change on your end. The batch your request landed in at each decode step is not something you control or can even observe; it depends on what other traffic the provider’s servers happened to be running at that instant. A low-traffic 2 a.m. request and a high-traffic midday request to the same endpoint are not guaranteed to pass through the same sequence of kernel operations, even though the model weights, your prompt, and your sampling parameters are all identical.

What actually fixes it — and what it costs

The real fix isn’t a request parameter at all — it’s forcing the serving infrastructure to use “batch invariant” kernels, meaning kernels whose output for a given row depends only on that row’s own data, never on the size or composition of the batch it happens to share the GPU with. vLLM exposes this as an environment flag, VLLM_BATCH_INVARIANT=1, which pins kernels to fixed, batch-independent execution strategies rather than letting them choose a strategy based on current load. It isn’t free or universal: the flag requires NVIDIA compute capability 8.0+ or an Intel XPU with the Triton attention backend, and in tensor-parallel deployments it also disables custom all-reduce, asynchronous tensor parallelism, and sequence parallelism — optimizations that exist specifically to make inference faster. Fused mixture-of-experts kernels have also been reported as not fully batch-invariant even with the flag enabled, so it narrows the problem rather than eliminating every instance of it.

That narrowing has a measured cost. Reported slowdowns for deterministic serving modes range from roughly 24% to 46% depending on workload, averaging around 34% in one measurement, with CUDA graph optimizations recovering a meaningful fraction — about 2.8x — of the lost throughput in that same test. A separate test of bitwise-consistent reinforcement-learning training on a small model reported training running at roughly 2.4x the wall-clock time of the non-bitwise baseline. Determinism here is a real trade against latency and cost, not a free correctness fix, which is exactly why it isn’t the default and why most API providers don’t expose it as a toggle you can set.

A quick way to check whether you’re actually hitting this

Before redesigning anything around this, confirm it’s actually happening to you and not something else, like a prompt template that silently includes a timestamp or a non-deterministic retrieval step upstream of the model call. Send the identical request — same prompt string, same temperature 0, same seed if your provider supports one — ten to twenty times in a loop, spaced a few seconds apart, and diff the raw responses rather than eyeballing them. Two things to look for: whether the responses cluster into a small number of distinct variants (consistent with the batching explanation above, where a handful of batch-composition patterns recur) versus drifting continuously with no repeats, and whether the variation is concentrated in places where multiple tokens were genuinely close in probability — a borderline word choice, a number that could reasonably round either way — rather than being evenly spread across the whole response. If your provider returns a system_fingerprint or an equivalent build/version identifier, log it alongside each response; a fingerprint change lines up with provider-side infrastructure updates, while variation at a constant fingerprint points more specifically at the batching behavior described above. This costs a handful of API calls and turns “it feels inconsistent” into something you can actually point to.

What this means if you’re building something that depends on repeatable output

If a workflow — a test suite, a regression check, a compliance audit trail — depends on an LLM returning the literal same text for the literal same input, temperature=0 and a fixed seed reduce how often it drifts, but they don’t make that assumption safe to build on through a standard hosted API. Three more realistic approaches:

• Test for semantic equivalence, not string equality. Compare structured fields, extracted values, or embedding similarity rather than asserting the raw text is byte-for-byte identical across runs.
• Pin the model version explicitly and expect to re-validate after any provider-side update — the system_fingerprint changing is a signal that the serving configuration moved, not evidence that something in your request is wrong.
• If strict determinism is a hard requirement — a regulated audit trail, a reproducible research benchmark — that’s a case for self-hosting inference with a batch-invariant configuration (vLLM’s flag, accepting the throughput cost) rather than relying on a parameter in a hosted API that was never designed to promise it.

The deeper point generalizes past this one parameter: a setting that reduces one source of variance is not the same claim as a setting that eliminates all sources of variance, and large inference systems have more moving parts than the parameters exposed in a request body. The same gap between “what I can set” and “what actually controls the behavior” shows up in how quantized model weights change output quality in ways a model card’s listed parameters don’t capture, and it’s worth the same skepticism anywhere a single toggle is described as guaranteeing a result that depends on the whole serving stack underneath it.

Featured image: “Arduino Due” by Simon Waldherr, licensed CC BY-SA 3.0, via Wikimedia Commons.