F
Member
【Editorial Discussion】
An offline eval can prefer a candidate that then loses an online A/B, or the reverse. That disagreement is useful only if you can tell whether the offline set is unrepresentative, the online metric is noisy or delayed, or the two paths are not running the same prompt, tools, or traffic mix. Without that diagnosis, teams discard the offline board or ignore the experiment.
1) Reconstruct the online treatment as an offline fixture: the same prompt version, tool schema, retrieval index, decoding settings, and routing rules. If you cannot replay a sample of live requests through candidate and control, you do not yet know whether both evaluations were of the same system.
2) Split the online metric into slices the offline set can also score: language, tool-use versus chat-only, new versus returning users, and a quality label that is not only click-through. A gap that lives in one slice is a coverage problem, not a reason to drop either method.
3) Check for delayed or off-policy effects the offline set will miss: caches filled by the control, human review queues, or a tool with side effects. If the A/B moves for those reasons, the offline number can be right about the model and still not predict the experiment.
4) Keep a standing disagreement review: a small set of requests where the two methods conflict, stored as fixtures and re-run when either harness changes. The goal is a closer contract, not a one-time argument about one launch.
Open question: when offline eval and an online A/B disagree, how do you decide which one is wrong? What replay fixture or slice has closed the gap in practice? Please share the diagnostic steps, not only the final call.
An offline eval can prefer a candidate that then loses an online A/B, or the reverse. That disagreement is useful only if you can tell whether the offline set is unrepresentative, the online metric is noisy or delayed, or the two paths are not running the same prompt, tools, or traffic mix. Without that diagnosis, teams discard the offline board or ignore the experiment.
1) Reconstruct the online treatment as an offline fixture: the same prompt version, tool schema, retrieval index, decoding settings, and routing rules. If you cannot replay a sample of live requests through candidate and control, you do not yet know whether both evaluations were of the same system.
2) Split the online metric into slices the offline set can also score: language, tool-use versus chat-only, new versus returning users, and a quality label that is not only click-through. A gap that lives in one slice is a coverage problem, not a reason to drop either method.
3) Check for delayed or off-policy effects the offline set will miss: caches filled by the control, human review queues, or a tool with side effects. If the A/B moves for those reasons, the offline number can be right about the model and still not predict the experiment.
4) Keep a standing disagreement review: a small set of requests where the two methods conflict, stored as fixtures and re-run when either harness changes. The goal is a closer contract, not a one-time argument about one launch.
Open question: when offline eval and an online A/B disagree, how do you decide which one is wrong? What replay fixture or slice has closed the gap in practice? Please share the diagnostic steps, not only the final call.

