F
Member
【Editorial Discussion】
A “deterministic” decoding setting (temperature zero, a fixed seed, even greedy search) can still produce different text on another GPU type, another batch size, or another kernel. Evaluation then disagrees with itself, and a release is blamed on the model when the run was not pinned. Determinism is a property of the whole stack, not of one sampling flag.
1) Pin what you can: model revision, tokenizer, precision, inference engine and version, GPU type, batch size, and whether graph capture or kernel autotune is allowed to change. If any of these differ, do not call the two runs a failed reproducibility test of the model.
2) Know which operations you have not pinned. Reduction order, non-deterministic attention kernels, and dynamic batching can change logits at the margin and, with long outputs, the token path. If you need bit-level or string-level agreement, you may have to disable those and accept a slower run.
3) For quality scores, decide whether you need exact string match or a stable score distribution. Many evals only need the latter, and should not be presented as “broken determinism” when a single token differs.
4) Record the hardware and the engine in the result file. A score without that row is not a regression baseline.
Open question: how strict is “deterministic” in your evals — exact text, or stable scores? What was the last stack difference that moved an answer despite temperature zero? Please share what you now pin.
A “deterministic” decoding setting (temperature zero, a fixed seed, even greedy search) can still produce different text on another GPU type, another batch size, or another kernel. Evaluation then disagrees with itself, and a release is blamed on the model when the run was not pinned. Determinism is a property of the whole stack, not of one sampling flag.
1) Pin what you can: model revision, tokenizer, precision, inference engine and version, GPU type, batch size, and whether graph capture or kernel autotune is allowed to change. If any of these differ, do not call the two runs a failed reproducibility test of the model.
2) Know which operations you have not pinned. Reduction order, non-deterministic attention kernels, and dynamic batching can change logits at the margin and, with long outputs, the token path. If you need bit-level or string-level agreement, you may have to disable those and accept a slower run.
3) For quality scores, decide whether you need exact string match or a stable score distribution. Many evals only need the latter, and should not be presented as “broken determinism” when a single token differs.
4) Record the hardware and the engine in the result file. A score without that row is not a regression baseline.
Open question: how strict is “deterministic” in your evals — exact text, or stable scores? What was the last stack difference that moved an answer despite temperature zero? Please share what you now pin.

