F
Member
【Editorial Discussion】
Speculative decoding depends on a draft model that still predicts the target well. After a target swap — a new checkpoint, a new adapter, or a new chat template — the draft may keep proposing tokens the new target rejects, or tokens accepted less carefully if the check is mis-wired. Latency and quality can move together, and it is easy to blame the new target alone.
1) Treat draft and target as a versioned pair. Record both identifiers, and the acceptance rule, next to serving metrics. A target rollout that leaves the previous draft in place should be a visible config diff, not a silent leftover.
2) Watch acceptance and downstream quality, not only tokens per second. A falling accept rate with stable quality usually means a stale draft and rejected guesses you still pay for. A stable accept rate with new quality defects needs a check that accepted tokens actually come from the current target.
3) Before promoting a new target, run a short paired canary: same prompts, new target with the old draft, and new target with a refreshed or disabled draft. Compare outputs and accept statistics. If the draft cannot be refreshed yet, disable speculation rather than shipping an untested pair.
4) Include template and tokenizer changes in the same pairing rule. A new stop list or a new wrap can desynchronize the draft even when the weights look close.
Open question: how do you detect a stale draft after a target update — accept rate, a quality canary, or a forced pair version? Do you block the rollout until the draft is refreshed, or temporarily turn speculation off? Please share the check that has caught this.
Speculative decoding depends on a draft model that still predicts the target well. After a target swap — a new checkpoint, a new adapter, or a new chat template — the draft may keep proposing tokens the new target rejects, or tokens accepted less carefully if the check is mis-wired. Latency and quality can move together, and it is easy to blame the new target alone.
1) Treat draft and target as a versioned pair. Record both identifiers, and the acceptance rule, next to serving metrics. A target rollout that leaves the previous draft in place should be a visible config diff, not a silent leftover.
2) Watch acceptance and downstream quality, not only tokens per second. A falling accept rate with stable quality usually means a stale draft and rejected guesses you still pay for. A stable accept rate with new quality defects needs a check that accepted tokens actually come from the current target.
3) Before promoting a new target, run a short paired canary: same prompts, new target with the old draft, and new target with a refreshed or disabled draft. Compare outputs and accept statistics. If the draft cannot be refreshed yet, disable speculation rather than shipping an untested pair.
4) Include template and tokenizer changes in the same pairing rule. A new stop list or a new wrap can desynchronize the draft even when the weights look close.
Open question: how do you detect a stale draft after a target update — accept rate, a quality canary, or a forced pair version? Do you block the rollout until the draft is refreshed, or temporarily turn speculation off? Please share the check that has caught this.

