Speculative Decoding is a large model inference acceleration method: first, a lighter, faster draft model proposes several candidate tokens in succession, then the target large model performs a parallel check. When a candidate is accepted, the target model can advance multiple positions in a single forward calculation; Any unqualified parts are corrected by the target model. It optimizes generation wait time, not simply stitching together answers from two models.
Slow down is generated by each token
Autoregressive models usually need to be formed as the first token before the second can proceed. Even with strong GPU computing power, this serial dependency causes frequent short forward computations during decoding, and the hardware may not be fully utilized. It is speculated that decoding first drafts a possible follow-up content, then submits it to large models for batch grading, reducing the number of calls to the target model.
The process can be understood as four steps:
- The draft model proposes a short sequence of candidate sequences based on the current context.
- The target model calculates the probabilities of these positions in parallel.
- Candidates are retained according to acceptance rules, and if a position does not match, acceptance stops.
- The target model supplements the correct branches before starting the next round of drafting.
Standard algorithms use supporting acceptance and correction rules, so they can maintain the original sampling distribution of the target model; It does not secretly replace large models with small models, nor does it require retraining the target model.
Why isn't it always faster?
The key metric is candidate acceptance rate. If the draft model is too weak, it is often rejected within the first few tokens, and additional drafting becomes an overhead; If the draft model is too large, it loses the meaning of "cheap generation." Text predictability, candidate length per round, hardware parallelism, and batch size all affect revenue. Content with strong continuity, such as code and fixed formats, is usually more likely to accept longer drafts than highly divergent creation.
It is different from other acceleration methods that solve problems
Quantization mainly reduces weight size and computational cost, while KV Cache reuses calculated historical attention states; Speculative decoding focuses on how to enable a single target model to confirm more tokens. These methods can be combined, but they collectively consume memory and increase scheduling complexity.
To determine whether it's worth adopting, don't just look at the speed in the demo. Deployment teams should compare their prompt length, output length, concurrency volume, and hardware for first token time, per-token latency, throughput, and memory usage, and confirm that acceleration does not come at the expense of lower stability.
