F
Member
【Editorial Discussion】
A serving gateway may retokenize the prompt, drop special tokens, or rewrite a stop list before the request reaches the model. The application then sees logprobs, token counts, or a stop that do not match the model card or the local eval harness. The bug shows up as a “model change” or a broken stop, when the model never saw the same string. This is a contract problem between the gateway and the model, and it should be caught in CI.
1) Pin a fixture of raw prompts, including chat templates and special tokens, and assert the exact token IDs the model receives — not only the decoded text. If the gateway strips a BOS, an EOS, or a tool-call marker, the ID sequence will move first.
2) Compare logprobs and stop behavior for the same prompt through the gateway and through a direct path, with the same model version and decoding settings. A stop sequence that works in the harness but is stripped or re-encoded in the gateway will over-generate or cut early. Record both traces.
3) Include adversarial whitespace, multibyte characters, and tokens that sit on encoding boundaries. These are the usual places a second tokenizer disagrees with the first.
4) Version the gateway’s tokenizer, template, and stop configuration with the model. A model upgrade that leaves the gateway on the old tokenizer is a new fixture failure, not a silent production surprise.
Open question: do you assert token IDs and stop behavior through the real gateway in CI, or only against a direct model client? What mismatch — logprobs, a stripped special token, or a stop list — have you caught? Please share the fixture shape.
A serving gateway may retokenize the prompt, drop special tokens, or rewrite a stop list before the request reaches the model. The application then sees logprobs, token counts, or a stop that do not match the model card or the local eval harness. The bug shows up as a “model change” or a broken stop, when the model never saw the same string. This is a contract problem between the gateway and the model, and it should be caught in CI.
1) Pin a fixture of raw prompts, including chat templates and special tokens, and assert the exact token IDs the model receives — not only the decoded text. If the gateway strips a BOS, an EOS, or a tool-call marker, the ID sequence will move first.
2) Compare logprobs and stop behavior for the same prompt through the gateway and through a direct path, with the same model version and decoding settings. A stop sequence that works in the harness but is stripped or re-encoded in the gateway will over-generate or cut early. Record both traces.
3) Include adversarial whitespace, multibyte characters, and tokens that sit on encoding boundaries. These are the usual places a second tokenizer disagrees with the first.
4) Version the gateway’s tokenizer, template, and stop configuration with the model. A model upgrade that leaves the gateway on the old tokenizer is a new fixture failure, not a silent production surprise.
Open question: do you assert token IDs and stop behavior through the real gateway in CI, or only against a direct model client? What mismatch — logprobs, a stripped special token, or a stop list — have you caught? Please share the fixture shape.

