F
Member
【Editorial Discussion】
An LLM feature in production will fail in ways that a usual web runbook does not list: a prompt or a tool-schema change, a model-version drift, a context overflow, a judge or a filter that started blocking, or a cache that serves yesterday’s answer. Telling the on-call engineer to “retry” is not a procedure. This is about what to write down so that a non-author of the feature can act.
1) The first page should say how to see: model and prompt version, retrieval set and cache hit, tool error rate, and output-filter or safety-block rate, for the current incident window. If those are not in one place, the runbook will collapse to restarts.
2) List the safe actions: roll back a prompt or a model pin, disable a tool, bypass a cache, or fail over to a non-LLM path. List the unsafe ones: “just raise the timeout,” “just open the filter,” “just retry the batch of writes.” Each safe action should have a check that it did what you thought.
3) Distinguish a provider outage, a dependency (RAG index, tool API), and a behaviour change in the model or the prompt. The first two are familiar incident types; the third needs a known-good snapshot to compare.
4) Practise the rollback on a quiet day. A runbook that has never been used will not be used correctly at 2 a.m.
Open question: what is actually in your LLM feature runbook today, and what is still “ask the person who built it”? Which action do you most want on-call to have, and which do you most want them to be forbidden from taking? Practical lists welcome.
An LLM feature in production will fail in ways that a usual web runbook does not list: a prompt or a tool-schema change, a model-version drift, a context overflow, a judge or a filter that started blocking, or a cache that serves yesterday’s answer. Telling the on-call engineer to “retry” is not a procedure. This is about what to write down so that a non-author of the feature can act.
1) The first page should say how to see: model and prompt version, retrieval set and cache hit, tool error rate, and output-filter or safety-block rate, for the current incident window. If those are not in one place, the runbook will collapse to restarts.
2) List the safe actions: roll back a prompt or a model pin, disable a tool, bypass a cache, or fail over to a non-LLM path. List the unsafe ones: “just raise the timeout,” “just open the filter,” “just retry the batch of writes.” Each safe action should have a check that it did what you thought.
3) Distinguish a provider outage, a dependency (RAG index, tool API), and a behaviour change in the model or the prompt. The first two are familiar incident types; the third needs a known-good snapshot to compare.
4) Practise the rollback on a quiet day. A runbook that has never been used will not be used correctly at 2 a.m.
Open question: what is actually in your LLM feature runbook today, and what is still “ask the person who built it”? Which action do you most want on-call to have, and which do you most want them to be forbidden from taking? Practical lists welcome.

