Speculative decoding is a way to make a large language model generate text faster: a much smaller draft model first guesses the next few tokens, and the large model then checks those guesses in a single parallel pass, keeping the correct ones and fixing the first mistake it finds. It changes the order of generation and how compute is scheduled, not the large model itself, so in a standard implementation the final output distribution matches what the large model would have produced token by token — faster, with no loss in quality.
Why token-by-token generation is slow
A large model writes one token at a time, and every token requires a full pass through the entire model. The catch is that for a single token, most of the time is not spent computing but on loading the huge set of parameters from memory. The hardware sits largely idle waiting for data. Verifying only one token per parameter load is where the waste comes from.
Who guesses, and who checks
The process has two steps. First, the draft model — often a few dozen times smaller than the target model — races ahead and proposes, say, four to eight candidate tokens. Second, the large model runs one parallel pass over those candidates together with the context and compares its own probability distribution at each position. Correct guesses are all accepted. At the first wrong position, the large model resamples a token from its own distribution, the remaining candidates are discarded, and the next round starts from there. Because verification happens in one parallel pass, a single loading of the parameters validates several tokens at once.
The actual speedup depends on how often the draft model guesses right. Predictable text — code completion, fixed-format replies, continuations that follow the context closely — yields high acceptance rates and commonly two to three times faster generation. Wildly unpredictable text gets rejected more often, and the gain shrinks.
Not the same as quantization or distillation
Speculative decoding is often confused with two other speed techniques. Quantization stores parameters at lower precision: the model gets smaller, but output quality shifts slightly. Distillation trains a small model to imitate a large one, and the student itself is what gets deployed, capability ceiling included. In speculative decoding the small model only drafts; the large model makes every final decision, which makes the speedup lossless. The cost is running two models at once: extra memory, and an alignment relationship to maintain.
When it does not help
It struggles when the draft model has very different habits from the target model — acceptance drops toward zero and nothing is gained, which is why practitioners use a smaller sibling from the same model family or a specially trained draft head. It also helps less under heavy concurrency: when many users hit the GPU at once, compute itself becomes the bottleneck rather than memory loading, so the benefit fades; it shines mainly in single-user or low-concurrency interactive use. And it does not shorten the wait for the very first token — it speeds up everything after that.
The draft-model and speculative-sampling options you see in local deployment tools, and the equivalent switches in serving frameworks such as vLLM, all run on this guess-then-verify mechanism. One rule of thumb when tuning: pick a draft model from the same family as the main model, do not overreach on guess length, and remember that acceptance rate matters more than stride.
