广告 / 赞助展示NVIDIA DGX B200 AI 系统官方产品图GPU 与加速计算访问官网 →

【Editorial】A gateway that retokenizes or strips special tokens before the model: how do you catch logprob or stop-sequence mismatches in CI?

Started by FPGA_ARM_Linux · 2026-11-14 13:12 北京时间 · 10 views · 1 replies
F Member
【Editorial Discussion】

A serving gateway may retokenize the prompt, drop special tokens, or rewrite a stop list before the request reaches the model. The application then sees logprobs, token counts, or a stop that do not match the model card or the local eval harness. The bug shows up as a “model change” or a broken stop, when the model never saw the same string. This is a contract problem between the gateway and the model, and it should be caught in CI.

1) Pin a fixture of raw prompts, including chat templates and special tokens, and assert the exact token IDs the model receives — not only the decoded text. If the gateway strips a BOS, an EOS, or a tool-call marker, the ID sequence will move first.

2) Compare logprobs and stop behavior for the same prompt through the gateway and through a direct path, with the same model version and decoding settings. A stop sequence that works in the harness but is stripped or re-encoded in the gateway will over-generate or cut early. Record both traces.

3) Include adversarial whitespace, multibyte characters, and tokens that sit on encoding boundaries. These are the usual places a second tokenizer disagrees with the first.

4) Version the gateway’s tokenizer, template, and stop configuration with the model. A model upgrade that leaves the gateway on the old tokenizer is a new fixture failure, not a silent production surprise.

Open question: do you assert token IDs and stop behavior through the real gateway in CI, or only against a direct model client? What mismatch — logprobs, a stripped special token, or a stop list — have you caught? Please share the fixture shape.
I Admin
@FPGA_ARM_Linux 网关可能在请求到达模型前重新分词、丢掉特殊 token 或改写停用词,于是应用侧看到的 logprobs、token 计数或停用行为,对不上模型卡和本地评测夹具。问题会被写成「模型变了」或「停用坏了」,其实模型从未看到同一串文本。这是网关与模型之间的契约,应该在 CI 里抓住。管理员同意:夹具要断言模型实际收到的 token ID,而不只是解码后的文本;同一提示词要同时走网关和直连路径对照。

管理员补五条闸门:

1. **CI 必须断言经网关后的 token ID 序列**,BOS/EOS 或工具标记被剥掉时,ID 会先动。评测:https://www.iaiforum.com/t/88 。
2. **停用行为和 logprobs 要双路径对照**,并保留两边轨迹。可观测:https://www.iaiforum.com/t/73 。
3. **夹具包含空白、多字节和编码边界 token**,这是两个分词器最容易分歧的地方。
4. **网关的分词器、模板和停用配置与模型一起版本化**:模型升级却留下旧分词器,应是夹具失败,不是线上惊喜。灰度:https://www.iaiforum.com/t/87 。
5. **评测分数缺少「是否经过网关」这一列时,不得当作回归基线**。计时/口径:https://www.iaiforum.com/t/191 。

想请教:你们现在是在真实网关上断言 token ID 和停用行为,还是只测直连客户端?最近抓住的是 logprobs、被剥掉的特殊 token,还是停用列表不一致?

——
IAI 管理员 · 官方(站方管理员账号 iaiforum_admin,Official admin account)
广告 / 赞助展示广告位招商此处招商 · ads@iaiforum.comA6 · 728×90(移动端 320×100)了解详情 →
Log in or register to reply.