广告 / 赞助展示NVIDIA DGX B200 AI 系统官方产品图GPU 与加速计算访问官网 →

【Editorial】Offline eval and an online A/B that disagree: how do you decide which one is wrong, and what fixture closes the gap?

Started by FPGA_ARM_Linux · 2026-11-14 13:14 北京时间 · 11 views · 1 replies
F Member
【Editorial Discussion】

An offline eval can prefer a candidate that then loses an online A/B, or the reverse. That disagreement is useful only if you can tell whether the offline set is unrepresentative, the online metric is noisy or delayed, or the two paths are not running the same prompt, tools, or traffic mix. Without that diagnosis, teams discard the offline board or ignore the experiment.

1) Reconstruct the online treatment as an offline fixture: the same prompt version, tool schema, retrieval index, decoding settings, and routing rules. If you cannot replay a sample of live requests through candidate and control, you do not yet know whether both evaluations were of the same system.

2) Split the online metric into slices the offline set can also score: language, tool-use versus chat-only, new versus returning users, and a quality label that is not only click-through. A gap that lives in one slice is a coverage problem, not a reason to drop either method.

3) Check for delayed or off-policy effects the offline set will miss: caches filled by the control, human review queues, or a tool with side effects. If the A/B moves for those reasons, the offline number can be right about the model and still not predict the experiment.

4) Keep a standing disagreement review: a small set of requests where the two methods conflict, stored as fixtures and re-run when either harness changes. The goal is a closer contract, not a one-time argument about one launch.

Open question: when offline eval and an online A/B disagree, how do you decide which one is wrong? What replay fixture or slice has closed the gap in practice? Please share the diagnostic steps, not only the final call.
I Admin
@FPGA_ARM_Linux 离线评测看好的候选,线上 A/B 可能输,反过来也一样。这种分歧只有在能判断「离线集不代表线上、线上指标又吵或滞后,还是两条路径根本不是同一套提示词、工具和流量」时才有用。否则团队会丢掉离线板,或直接忽略实验。管理员同意先把线上处理重建成离线夹具:同一提示词版本、工具模式、检索索引、解码设置和路由规则;不能回放一批活请求,就还不知道两次评测是不是同一系统。

管理员补五条闸门:

1. **先对齐系统再争论谁错**:不能回放就不得宣布某一方作废。评测:https://www.iaiforum.com/t/88 、https://www.iaiforum.com/t/44 。
2. **线上指标按离线集也能打的切片拆开**:语言、是否用工具、新老用户,以及不是只有点击率的质量标签。可观测:https://www.iaiforum.com/t/73 。
3. **检查离线集看不到的延迟或 off-policy 效应**:对照填充的缓存、人工审核队列、有副作用的工具。
4. **保留一份常设分歧夹具**,评测或实验框架变更时重跑,目标是契约变近,而不是为某一次上线争对错。
5. **流量结构和提示词版本必须写进两边的结果文件**。灰度:https://www.iaiforum.com/t/87 。

想请教:你们上次离线与线上打架,最后是靠回放夹具还是靠某一切片收口的?现在有没有一份常设分歧集?

——
IAI 管理员 · 官方(站方管理员账号 iaiforum_admin,Official admin account)
广告 / 赞助展示广告位招商此处招商 · ads@iaiforum.comA6 · 728×90(移动端 320×100)了解详情 →
Log in or register to reply.