Nondeterminism

Set the seed, set temperature to zero, run the same prompt twice, and get two different completions. This surprises people, and the surprise is the problem — it leads to bug reports filed against sampling code that is working correctly.

The seed controls the sampler. Nothing else in the stack consulted it.

Addition is not associative

In exact arithmetic these are equal. In floating point each addition rounds, and rounding depends on the magnitudes of the operands, so a different grouping gives a different answer. The difference is one unit in the last place per operation — utterly negligible, and it does not stay negligible.

A GPU reduction is a tree, and the shape of the tree is chosen by the kernel from the problem size, the launch configuration and sometimes an autotuner. Change any of them and the grouping changes.

Batch size changes the answer

This is the one that catches people in production. cuBLAS and its successors pick a matmul kernel from the shapes involved, and a batch of 1 gets a different kernel — different tiling, different split-K, different reduction order — than a batch of 32. Same weights, same input row, different logits in the last few decimal places.

With continuous batching, your request is grouped with whatever else arrived in the same window. The batch composition is a function of other people’s traffic, so the numerics of your own request are too. This is why a served model gives different output than the same model run locally, and why it gives different output at 3 a.m. than at noon.

Where 1e−3 becomes a different sentence

Greedy decoding takes an argmax. Almost always the top two logits are far enough apart that a $10^{-3}$ perturbation cannot reorder them — and then a token arrives where they are not.

Past that token the two runs have different context, and they diverge completely rather than gradually. Ten to a hundred tokens is the usual distance to a visible split for a 7B model; a chain-of-thought trace with hundreds of near-ties reaches it faster.

Mixture of experts

MoE routing makes it structural rather than numerical. Experts have finite capacity per batch; when more tokens route to an expert than it can take, the surplus is dropped. Which tokens are dropped depends on what else is in the batch — so a token’s computation, not merely its rounding, depends on its neighbours.

import os

import torch

# Must be set before the CUDA context exists. cuBLAS otherwise picks
# split-K kernels whose reduction order varies with workspace availability.
os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"


def make_deterministic(seed: int = 0) -> None:
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)

    # Raises on any op with no deterministic implementation, rather than
    # quietly using the nondeterministic one.
    torch.use_deterministic_algorithms(True)

    # cuDNN autotuning benchmarks kernels at runtime and picks the fastest,
    # which is not stable across machines or even across runs.
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

What this buys is run-to-run reproducibility on one machine, at one batch size, with one library version, typically at a 10–20% throughput cost. What it does not buy is reproducibility across GPU architectures, across batch sizes, or against a served endpoint — none of which are under your control from inside the process.

Mostly, stop requiring bitwise equality. It is the wrong test for a system built on floating point, and pinning it forces an expensive configuration on production in order to satisfy CI.

Better tests: assert on distributions rather than samples — mean and variance of a metric over $n$ runs, with an interval; compare logits with a tolerance rather than an equality; and pin the batch size in any evaluation whose numbers will be compared against numbers from another day. For anything comparative, the statistical power question comes first anyway, and seed variance is usually smaller than the effect being argued about — which makes it a reason to run more seeds, not a reason to eliminate them.