A Practical Investigation of Training-free Relaxed Speculative Decoding
2026-07-09 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors study a way to speed up large language models by first using a faster smaller model to guess multiple words, and then double-checking these guesses in batches. Normally, this method is exact and always matches the original model’s behavior, but the authors explore relaxing this to gain more speed or flexibility. They compare different relaxed versions without needing extra training and summarize what works well. They find that relaxed methods often need careful testing and work best when the smaller model is also good at language, which makes simple guessers less effective.
speculative decodingautoregressive modelslarge language modelssampling distributionmulti-token predictiondrafter modelspeed-accuracy trade-offparallel verificationlossless decoding
Authors
Guoxuan Xia, Luka Ribar, Paul Balanca
Abstract
Speculative decoding accelerates sampling from an autoregressive LLM by using a faster auxiliary model to draft tokens which are then verified in parallel by the LLM. Standard speculative decoding is lossless: its rejection and resampling steps exactly preserve the LLM's sampling distribution. Recent work argues that relaxing this strict guarantee can yield further speed-ups, controlled capability-speed trade-offs, or even capability gains. We practically investigate training-free relaxed speculative decoding techniques, unify existing approaches within a shared framework, benchmark them on contemporary settings, and distil takeaways and empirical findings for practitioners. Important takeaways include: relaxation can require considerable capability evaluation unlike lossless speculative decoding, and many relaxed approaches rely on a drafter that is a good language model, making them unsuited for lightweight dedicated multi-token-prediction drafters.