Speculative Decoding: How AI Guesses Ahead to Write Faster

Speculative decoding is a way to make a large language model produce text faster by letting a smaller, cheaper process guess several words ahead while the big model checks the guesses all at once. The original paper, from Google researchers, reports a 2X to 3X speedup on a T5-XXL model with identical outputs. Later methods such as Medusa and EAGLE-3 report larger gains, though under their own test conditions.

Why do language models generate text one token at a time?

A language model writes by predicting a token (a word or a fragment of a word), adding it to the text, and predicting the next one. Each new token depends on all the earlier ones, so the steps are strictly sequential.

The expensive part is moving data. According to the Medusa paper, each decoding step has to move the full set of model parameters out of the GPU's high-bandwidth memory. The chip spends much of its time waiting for data to arrive, and one token per trip leaves most of its compute capacity idle.

Because the trip to memory is a large part of each step, scoring several candidate tokens in one pass is cheaper than running several separate steps, because the parameters are read once for the whole group of candidates. A longer reply means more trips, so the saving grows with length.

If you want background on what the model keeps in memory between steps, see our explainer on the KV cache.

What is speculative decoding and how does it work?

The idea was published in November 2022 by Yaniv Leviathan, Matan Kalman and Yossi Matias in Fast Inference from Transformers via Speculative Decoding. The paper builds on two observations: hard language tasks often contain easier pieces that a smaller model can approximate well, and a new sampling method lets the large model run in parallel on the small model's output.

In practice the loop has three steps:

1. A small, fast "draft" model proposes the next few tokens. 2. The large "target" model scores all of those proposals in a single forward pass. 3. The target keeps the proposals it agrees with, up to the first disagreement, and supplies its own token at that point.

When the draft is right, several tokens arrive for the price of one big-model step. When it is wrong, the loop falls back to roughly normal speed for that step. The more often the target agrees with the draft, the bigger the gain, which is why the quality of the guesses matters so much.

Does speculative decoding change the answers the model gives?

No, when it is implemented as described in the original paper. The abstract says the method samples "without any changes to the outputs" and without changing the distribution of results, and the T5-XXL speedup it reports comes with identical outputs.

The reason is that the large model has the final say. A draft token is only kept if it passes the target model's check, so the draft model affects only speed. Proposals past the first disagreement are discarded, which means the GPU computes them for nothing.

In the worst case the target rejects the very first guess and supplies its own token, so the text still follows the large model's own distribution.

The Medusa paper makes a similar claim for Medusa-1, which it describes as lossless acceleration. Medusa-2, which retrains the backbone model, is the variant where the authors warn that a special recipe is needed to protect the model's abilities.

How do Medusa and EAGLE-3 differ from the original approach?

The original method needs a separate small model. Medusa and EAGLE remove or reshape that requirement.

Medusa. According to the Medusa paper (Cai, Li, Geng, Peng, Lee, Chen and Dao), the technique attaches extra decoding "heads" to the main model so that each step proposes several future tokens. A tree-based attention mechanism lets it verify several candidate continuations together in one step. No separate draft model is needed.

The paper also describes self-distillation, a way to train the heads when the original training data is not available, and a "typical acceptance" scheme meant to raise how often candidates are accepted. Neither changes the basic idea of proposing several tokens and checking them together.

The authors describe two variants. Medusa-1 trains only the heads on a frozen backbone and reports over 2.2x speedup with no loss in generation quality. Medusa-2 trains the heads and the backbone together and reports 2.3x to 3.6x, but needs a special training recipe to preserve the model's abilities.

EAGLE-3. The EAGLE-3 paper (Li, Wei, Zhang and Zhang) says earlier EAGLE versions predicted internal "features" of the target model, and that this limited how much extra training data helped. EAGLE-3 predicts tokens directly and fuses features from several layers of the target instead of only the top layer.

The authors report up to 6.5x speedup across five tasks on chat and reasoning models, about 1.4x better than EAGLE-2. They also report a 1.38x throughput gain in the SGLang serving framework at batch size 64.

When does the speedup disappear?

Speed gains depend on how often the target accepts the draft's guesses. The result depends on how closely the draft matches the target, how many positions are drafted, and the workload. Acceptance drops at later draft positions, so a longer draft adds checking work that is often thrown away, and separate drafters take extra GPU memory. That is how one method can speed up a model in one test and slow down another model in the same test.

The vLLM documentation for version 0.8.2 says speculative decoding in vLLM "is not yet optimized and does not usually yield inter-token latency reductions" for all prompt datasets or sampling parameters. It also lists restrictions: it is not compatible with pipeline parallelism, and EAGLE-based draft models must run without tensor parallelism.

A 2026 vLLM blog post on AMD GPUs tested several methods against a baseline without speculation. Some results were strong: on gemma-4-26B-A4B-it, DFlash reached 2.87x throughput on MATH500 and EAGLE-3 ranged from 2.11x to 2.27x.

On Qwen3-8B, by contrast, EAGLE-3 with one drafted token ran at 0.44x of baseline on MATH500, meaning it slowed serving down. The post says results depend on the model, the draft checkpoint and the workload.

The authors recommend measuring acceptance and end-to-end throughput on your own workload before choosing settings.

Should you use it for your own AI workload?

If you run models yourself, draft models, EAGLE heads and Medusa heads each have trade-offs: a separate draft model adds memory, trained heads need training, and the best number of guessed tokens differed between models in the vLLM test.

Our guide to prefix caching in SGLang and vLLM covers another way serving engines save time.

What to watch next

• Serving-framework support: the vLLM docs describe speculative decoding as not yet optimized, so improvements there would change the numbers.
• New draft methods, such as the DFlash and DSpark approaches in the vLLM AMD post, and whether they hold up outside that one test setup.
• Built-in multi-token prediction (the "native MTP" in the same post) from model makers, which would remove the need for a separate draft.
• Independent benchmarks on real workloads, which the best-case figures in papers do not cover.