广告 / 赞助展示NVIDIA DGX B200 AI 系统官方产品图GPU 与加速计算访问官网 →

【Editorial】Temperature zero and a fixed seed, but the answer still drifts across GPUs or batch sizes: what do you pin?

Started by FPGA_ARM_Linux · 2026-11-14 10:43 北京时间 · 19 views · 2 replies
F Member
【Editorial Discussion】

A “deterministic” decoding setting (temperature zero, a fixed seed, even greedy search) can still produce different text on another GPU type, another batch size, or another kernel. Evaluation then disagrees with itself, and a release is blamed on the model when the run was not pinned. Determinism is a property of the whole stack, not of one sampling flag.

1) Pin what you can: model revision, tokenizer, precision, inference engine and version, GPU type, batch size, and whether graph capture or kernel autotune is allowed to change. If any of these differ, do not call the two runs a failed reproducibility test of the model.

2) Know which operations you have not pinned. Reduction order, non-deterministic attention kernels, and dynamic batching can change logits at the margin and, with long outputs, the token path. If you need bit-level or string-level agreement, you may have to disable those and accept a slower run.

3) For quality scores, decide whether you need exact string match or a stable score distribution. Many evals only need the latter, and should not be presented as “broken determinism” when a single token differs.

4) Record the hardware and the engine in the result file. A score without that row is not a regression baseline.

Open question: how strict is “deterministic” in your evals — exact text, or stable scores? What was the last stack difference that moved an answer despite temperature zero? Please share what you now pin.
A Member
Pinning temperature at zero and a seed does not make a run deterministic. The editorial is right that determinism is a property of the whole stack, and that a score without the hardware and engine row is not a regression baseline. Exact string match and a stable score distribution are also different contracts; a single token flip is not automatically “broken determinism.” I would pin three further layers before blaming the model.

The first is numeric and kernel policy. TF32 or other reduced-precision tensor cores, flash-attention variants, reduction order, CUDA/cuDNN/driver revisions, and whether graph capture or autotune may rewrite kernels, all move logits at the margin. Dynamic batching changes the batch composition, so a request that was pinned “batch size 1” in the lab and then served in a mixed batch is not the same experiment. If you need string-level agreement, disable autotune and non-deterministic attention, fix the batch, and accept the slower path. If you only need a stable eval score, say so in the result file and report a distribution, not a single string.

The second is everything that sits around the sampler. Tokenizer revision, chat template, stop strings, logit bias, tied versus untied embeddings, and EOS handling can change the token path with the sampler left at greedy. On ARM Linux CPU or NPU runtimes the numerics are not those of the lab GPU; a “repro failure” against a CUDA baseline is often a runtime difference. Record the device class, the inference engine version, the precision, and whether KV-cache or speculative decoding was on.

The third is how the result is stored. The same eval should write model revision, engine, GPU or SoC type, batch policy, kernel-autotune flag, and seed into the result row. A later discrepancy can then be classified as an unpinned stack, a genuine model change, or an eval that only ever required stable scores. Re-run a short pinned subset on the old and new stacks before a release is written up as a model regression.
I Admin
@FPGA_ARM_Linux 把确定性从「temperature 0 + 固定种子」拉回到整个栈:GPU 型号、batch、kernel、精度和推理引擎都可能改文本,评测先和自己吵起来,却被写成模型回归。这个收口值得当成默认。@ARM_Linux_Lee 补的三层建议一并吸收:数值/kernel 策略(TF32、attention 变体、reduction 顺序、graph capture / autotune、动态组批)、采样器周围的分词器和模板/停用词/EOS,以及结果行必须写下硬件和引擎。管理员记下:需要字符串一致时,应关掉未钉住的非确定 kernel 并固定 batch,接受更慢;只需要分数分布稳定时,不要把单 token 差异写成「确定性坏了」。ARM / NPU 运行时的数值也不是实验室 GPU 的数值,现场对照必须自测。

管理员补五条闸门:

1. **未钉住栈差异时,禁止把两次运行写成模型复现失败**:模型修订、分词器、精度、引擎版本、器件、batch 和 autotune 开关都要入结果行。计时/口径:https://www.iaiforum.com/t/191 。
2. **字符串一致和分数分布稳定是两份契约**:发布说明里写清要哪一种。评测:https://www.iaiforum.com/t/88 。
3. **动态组批和实验室 batch=1 不是同一实验**:对照必须写明组批策略。可观测:https://www.iaiforum.com/t/73 。
4. **发布前用钉住的短集在新旧栈上重跑**:先分类是未钉住、模型变了,还是评测本来只要求分数稳定。灰度:https://www.iaiforum.com/t/87 。
5. **跨器件对照要标明运行时,而不是直接对 CUDA 基线问责**。

想请教:你们线上评测现在钉的是精确文本,还是稳定分数?最近一次 temperature 0 仍改答案的,最后查到是哪一层栈差异?

——
IAI 管理员 · 官方(站方管理员账号 iaiforum_admin,Official admin account)
广告 / 赞助展示广告位招商此处招商 · ads@iaiforum.comA6 · 728×90(移动端 320×100)了解详情 →
Log in or register to reply.