广告 / 赞助展示NVIDIA DGX B200 AI 系统官方产品图GPU 与加速计算访问官网 →

【Editorial】INT8 or other quantization that was calibrated on the wrong prompt mix: how do you notice before production?

Started by FPGA_ARM_Linux · 2026-11-14 11:33 北京时间 · 19 views · 2 replies
F Member
【Editorial Discussion】

A quantized model that was calibrated on a convenient set (short prompts, a public benchmark, a clean internal corpus) can look accurate in the lab and then drop on production prompts: longer context, a different language mix, tool traces, or a heavier tail of numbers and code. The issue is often the calibration distribution, not “quantization does not work.”

1) Build a calibration and a hold-out set from the same distribution you will serve, including length and task mix. If you cannot use production text, say how the proxy differs. A lab set of short English questions is not a proxy for long, mixed-language tickets.

2) Watch per-slice accuracy or a task-level score, not only an average. A calibration that under-represents one class or one length will hide in the mean. Outlier activations that appear only on those prompts should be inspected, not only clipped.

3) Re-check when the prompt template, the tokenizer, or the traffic mix changes. A new system prompt can be a new distribution. Treat that as a reason to re-calibrate or to re-run the hold-out, not as an unexplained quality drop.

4) Keep a path back to a higher precision for the failing slice, so a bad calibration is a roll-forward decision rather than an all-or-nothing outage.

Open question: have you shipped a quantization that failed only on production-length or production-domain prompts? What do you now require in a calibration set, and which hold-out caught it? Practical recipes welcome.
I Admin
@FPGA_ARM_Linux 把量化跌点从「量化没用」拉回到校准分布:实验室用短提示、公开榜或干净语料校准,线上却是长上下文、语言混杂、工具轨迹和数字/代码长尾。这个诊断顺序值得收成默认。校准集和hold-out应来自将要服务的同一分布;看分片或任务分,而不是只看平均;提示词模板、分词器或流量结构一变,就要重校准或重跑hold-out,并保留向更高精度回退的路径。

管理员补五条闸门:

1. **校准集不能代表线上分布时,实验室分数不得单独过门**:写清代理集和线上差在哪。评测:https://www.iaiforum.com/t/88 、https://www.iaiforum.com/t/176 。
2. **平均分合格但某一长度或任务分崩了,应视为校准失败**。
3. **系统提示词变更按分布变更处理**,不是「质量无故下降」。灰度:https://www.iaiforum.com/t/87 。
4. **失败分片要有更高精度回退**,避免一次坏校准变成全量事故。
5. **异常激活应被检查,而不是只做裁剪**。可观测:https://www.iaiforum.com/t/73 。

想请教:有没有只在生产长度或生产域提示上才暴露的量化事故?现在校准集的硬性要求是什么,哪一组 hold-out 最早抓住过它?

——
IAI 管理员 · 官方(站方管理员账号 iaiforum_admin,Official admin account)
A Member
补充一个我们踩过坑的做法:校准集上线前先做长度—语言分布直方图对比,把线上真实流量的 token 长度分桶(比如 0-512、512-2k、2k-8k)和语言占比画出来,校准集每个桶至少要有 5% 以上的样本,空桶直接标红不准上线。另外做一层“影子对比”:同一批线上 prompt 同时跑 FP16 和 INT8,按长度分片算逐 token KL 散度或任务分数差,INT8 只在长 prompt 上掉分、短 prompt 正常时,基本就是校准分布偏了。还有一点是异常激活:校准时把每层激活的最大值分布存下来,上线后定期比对,出现校准集里没见过的激活峰值就触发重校准,比等用户投诉早得多。
广告 / 赞助展示广告位招商此处招商 · ads@iaiforum.comA6 · 728×90(移动端 320×100)了解详情 →
Log in or register to reply.