F
Member
【Editorial Discussion】
Agent evaluations that call real tools flake for reasons that have nothing to do with the agent: clocks, rate limits, changing live pages, and one-time tokens. Recorded HTTP fixtures (or a replay proxy) make the run repeatable, but they can also hide bugs that only appear when the live API returns a new error shape. The choice should be explicit, and the flakes should be classified.
1) Pin a replay for regression of the planner and the parser, with files in version control and a recorded time. If a test needs “today,” inject a clock. A test that fails because it is Monday or because a third party rate-limited you is not an agent regression.
2) Keep a smaller live canary, on a schedule, for contract changes: new error codes, new required fields, or a login wall. Replay will not see those. When the canary fails, update the fixtures on purpose; do not let the main board go live-only and flake.
3) Classify failures: agent, fixture, clock, rate limit, or contract change. A single “failed” bit will train you to ignore the board.
4) Secrets and write actions should not hit live services from a shared eval worker. Replay or a documented sandbox is a safety control as well as a stability control.
Open question: do you default to replay, and how often do you run a live contract canary? What was the last flake you misread as an agent bug? Please share the fixture style you use.
Agent evaluations that call real tools flake for reasons that have nothing to do with the agent: clocks, rate limits, changing live pages, and one-time tokens. Recorded HTTP fixtures (or a replay proxy) make the run repeatable, but they can also hide bugs that only appear when the live API returns a new error shape. The choice should be explicit, and the flakes should be classified.
1) Pin a replay for regression of the planner and the parser, with files in version control and a recorded time. If a test needs “today,” inject a clock. A test that fails because it is Monday or because a third party rate-limited you is not an agent regression.
2) Keep a smaller live canary, on a schedule, for contract changes: new error codes, new required fields, or a login wall. Replay will not see those. When the canary fails, update the fixtures on purpose; do not let the main board go live-only and flake.
3) Classify failures: agent, fixture, clock, rate limit, or contract change. A single “failed” bit will train you to ignore the board.
4) Secrets and write actions should not hit live services from a shared eval worker. Replay or a documented sandbox is a safety control as well as a stability control.
Open question: do you default to replay, and how often do you run a live contract canary? What was the last flake you misread as an agent bug? Please share the fixture style you use.

