F
Member
【Editorial Discussion】
A quantized model that was calibrated on a convenient set (short prompts, a public benchmark, a clean internal corpus) can look accurate in the lab and then drop on production prompts: longer context, a different language mix, tool traces, or a heavier tail of numbers and code. The issue is often the calibration distribution, not “quantization does not work.”
1) Build a calibration and a hold-out set from the same distribution you will serve, including length and task mix. If you cannot use production text, say how the proxy differs. A lab set of short English questions is not a proxy for long, mixed-language tickets.
2) Watch per-slice accuracy or a task-level score, not only an average. A calibration that under-represents one class or one length will hide in the mean. Outlier activations that appear only on those prompts should be inspected, not only clipped.
3) Re-check when the prompt template, the tokenizer, or the traffic mix changes. A new system prompt can be a new distribution. Treat that as a reason to re-calibrate or to re-run the hold-out, not as an unexplained quality drop.
4) Keep a path back to a higher precision for the failing slice, so a bad calibration is a roll-forward decision rather than an all-or-nothing outage.
Open question: have you shipped a quantization that failed only on production-length or production-domain prompts? What do you now require in a calibration set, and which hold-out caught it? Practical recipes welcome.
A quantized model that was calibrated on a convenient set (short prompts, a public benchmark, a clean internal corpus) can look accurate in the lab and then drop on production prompts: longer context, a different language mix, tool traces, or a heavier tail of numbers and code. The issue is often the calibration distribution, not “quantization does not work.”
1) Build a calibration and a hold-out set from the same distribution you will serve, including length and task mix. If you cannot use production text, say how the proxy differs. A lab set of short English questions is not a proxy for long, mixed-language tickets.
2) Watch per-slice accuracy or a task-level score, not only an average. A calibration that under-represents one class or one length will hide in the mean. Outlier activations that appear only on those prompts should be inspected, not only clipped.
3) Re-check when the prompt template, the tokenizer, or the traffic mix changes. A new system prompt can be a new distribution. Treat that as a reason to re-calibrate or to re-run the hold-out, not as an unexplained quality drop.
4) Keep a path back to a higher precision for the failing slice, so a bad calibration is a roll-forward decision rather than an all-or-nothing outage.
Open question: have you shipped a quantization that failed only on production-length or production-domain prompts? What do you now require in a calibration set, and which hold-out caught it? Practical recipes welcome.

