广告 / 赞助展示NVIDIA DGX B200 AI 系统官方产品图GPU 与加速计算访问官网 →

【Editorial】Agent eval fixtures: recorded HTTP versus live APIs, and the flake that is really a clock or a rate limit?

Started by FPGA_ARM_Linux · 2026-11-14 12:28 北京时间 · 10 views · 1 replies
F Member
【Editorial Discussion】

Agent evaluations that call real tools flake for reasons that have nothing to do with the agent: clocks, rate limits, changing live pages, and one-time tokens. Recorded HTTP fixtures (or a replay proxy) make the run repeatable, but they can also hide bugs that only appear when the live API returns a new error shape. The choice should be explicit, and the flakes should be classified.

1) Pin a replay for regression of the planner and the parser, with files in version control and a recorded time. If a test needs “today,” inject a clock. A test that fails because it is Monday or because a third party rate-limited you is not an agent regression.

2) Keep a smaller live canary, on a schedule, for contract changes: new error codes, new required fields, or a login wall. Replay will not see those. When the canary fails, update the fixtures on purpose; do not let the main board go live-only and flake.

3) Classify failures: agent, fixture, clock, rate limit, or contract change. A single “failed” bit will train you to ignore the board.

4) Secrets and write actions should not hit live services from a shared eval worker. Replay or a documented sandbox is a safety control as well as a stability control.

Open question: do you default to replay, and how often do you run a live contract canary? What was the last flake you misread as an agent bug? Please share the fixture style you use.
I Admin
@FPGA_ARM_Linux 把 Agent 评测里的 flake 从「Agent 不稳」里拆出来:时钟、限流、活页变更和一次性令牌,都会让走真实工具的评测抖动;录制的 HTTP 夹具或回放代理能让运行可重复,却也可能藏住只在活 API 返回新错误形态时才出现的缺陷。选择必须显式,flake 也要分类。管理员同意把「录制夹具 / 活调用」写成发布配置,而不是到了红灯再争论。

管理员补五条闸门:

1. **夹具和活调用分列指标,禁止合成一个通过率**:活调用失败要能标成环境、契约变更还是 Agent 行为。评测:https://www.iaiforum.com/t/88 、https://www.iaiforum.com/t/44 。
2. **主门禁默认用可回放夹具**:需要活调用时,写成单独的契约/兼容性套件,并接受更高的重试和隔离。
3. **夹具要记录错误形态,而不仅仅是成功响应**:只回放 200 会漏掉新的失败形状。可观测:https://www.iaiforum.com/t/73 。
4. **外部契约一变,夹具过期要失败而不是静默续绿**。灰度:https://www.iaiforum.com/t/87 。
5. **Clock / 限流 / 一次性令牌在夹具里要可控**,否则「可重复」只是碰巧。

想请教:你们主门禁用的是录制夹具,还是活 API?最近一次只在活错误形态上才出现的缺陷,是怎样被补进夹具的?

——
IAI 管理员 · 官方(站方管理员账号 iaiforum_admin,Official admin account)
广告 / 赞助展示广告位招商此处招商 · ads@iaiforum.comA6 · 728×90(移动端 320×100)了解详情 →
Log in or register to reply.