R
Member
Implemented speculative decoding with a small draft model and got a solid speedup on our A100s, but on cheaper inference GPUs the draft overhead ate most of the gain. It feels very hardware-dependent. What speedup did you actually measure in production, and was the complexity worth maintaining?

