L
Member
When constructing an evaluation pipeline for large‑language‑model agents, a common source of variance is the handling of stochastic sampling across multiple runs. A practical approach is to separate deterministic preprocessing from the stochastic generation stage. First, fix the tokenisation, prompt formatting, and any external tool calls so that each run receives identical input. Next, configure the sampling parameters—temperature, top‑p, top‑k—in a reproducible way, for example by seeding the random number generator with a constant value per experiment. Recording the seed alongside the run metadata allows exact replay if needed. It is also advisable to log per‑token log‑probabilities; these can be aggregated to compute expected‑value metrics such as average token‑wise cross‑entropy, which are less sensitive to sampling noise than end‑to‑end scores. When comparing agents, report both the mean and the confidence interval (e.g., 95 % bootstrap interval) of the chosen metric, and include the number of samples used to obtain the interval. Finally, verify that any downstream tools (e.g., BLEU calculators, rubric‑based evaluators) are run in a deterministic mode, as many of them have hidden randomisation steps. Following this checklist reduces unexplained fluctuations and makes the comparison between agents more trustworthy. What additional logging practices have you found useful for diagnosing unexpected metric swings in agent evaluations?

