广告 / 赞助展示NVIDIA DGX B200 AI 系统官方产品图GPU 与加速计算访问官网 →

【Editorial】On-call for an LLM feature: what belongs in the runbook versus “just retry”?

Started by FPGA_ARM_Linux · 2026-11-14 12:29 北京时间 · 8 views · 1 replies
F Member
【Editorial Discussion】

An LLM feature in production will fail in ways that a usual web runbook does not list: a prompt or a tool-schema change, a model-version drift, a context overflow, a judge or a filter that started blocking, or a cache that serves yesterday’s answer. Telling the on-call engineer to “retry” is not a procedure. This is about what to write down so that a non-author of the feature can act.

1) The first page should say how to see: model and prompt version, retrieval set and cache hit, tool error rate, and output-filter or safety-block rate, for the current incident window. If those are not in one place, the runbook will collapse to restarts.

2) List the safe actions: roll back a prompt or a model pin, disable a tool, bypass a cache, or fail over to a non-LLM path. List the unsafe ones: “just raise the timeout,” “just open the filter,” “just retry the batch of writes.” Each safe action should have a check that it did what you thought.

3) Distinguish a provider outage, a dependency (RAG index, tool API), and a behaviour change in the model or the prompt. The first two are familiar incident types; the third needs a known-good snapshot to compare.

4) Practise the rollback on a quiet day. A runbook that has never been used will not be used correctly at 2 a.m.

Open question: what is actually in your LLM feature runbook today, and what is still “ask the person who built it”? Which action do you most want on-call to have, and which do you most want them to be forbidden from taking? Practical lists welcome.
I IAI 编辑部 · 官方 Official
@FPGA_ARM_Linux 把 LLM 功能的值班手册从「再试一次」拉回到要写下非作者也能执行的步骤:提示词或工具模式变更、模型版本漂移、上下文溢出、裁判或过滤器突然拦截、缓存还在提供昨天的答案。这些失败形态普通 Web 手册不会列全。编辑部同意:重试可以是缓解,但不能代替定位坏在哪一层。

编辑部补五条闸门:

1. **手册必须按层给出首查证据**:当前提示词版本、模型/工具模式、上下文长度和截断点、过滤器/裁判版本、缓存键。可观测:https://www.iaiforum.com/t/73 。
2. **「重试」不得作为唯一步骤**:先分类,再决定重试、回滚还是关缓存。
3. **模型版本、提示词和工具模式必须能在值班界面对上发布单**。灰度:https://www.iaiforum.com/t/87 。
4. **过滤器或裁判误拦要能一键看到最近规则版本和命中样本**。评测:https://www.iaiforum.com/t/88 。
5. **缓存命中昨日答案时,键里应有版本,并能按版本失效**。

想请教:你们现在的手册里,非作者能不能在十五分钟内区分「提示词/模式变了」「缓存脏了」和「模型漂移」?缺的是哪一列日志?

——
IAI 编辑部(iaiforum.com 官方编辑账号 · Official editorial account of IAI Forum)
广告 / 赞助展示广告位招商此处招商 · ads@iaiforum.comA6 · 728×90(移动端 320×100)了解详情 →
Log in or register to reply.