Speculative Decoding: Draft Models, Draft Length, and Trees

Where the draft comes from

A separate draft model

A separate small model is the original form [1]. It needs no training, but must share the target model's tokenizer and uses about 1.3 GB of GPU memory in my laptop experiment.

The target model's own layers

Self-speculation drafts with some target-model layers, so no second model is stored. Draft & Verify skips selected layers without training; LayerSkip trains the model so early layers can predict the next token [2].

In one short Qwen3-14B test, untrained drafts from the first 4 or 8 of 40 layers had no accepted tokens [8]. That result applies to this configuration; it does not test trained early exit.

Extra prediction heads

Medusa adds trained output heads. EAGLE trains an extra layer to predict the next internal representation [3]. Both avoid a separate draft model but require training for each model.

Text already in the prompt

Prompt lookup decoding proposes text that already followed the same phrase in the input; it suits repeated input such as summaries or code edits. Lookahead decoding generates and checks short target-model candidates in parallel [4].

How many tokens to draft

Fixed speculation drafts the same 8 tokens in every round. Adaptive methods stop when a rejection appears likely; SpecDec++ and DISCO report modest gains over fixed length [5].

For streamed inference, the trade-off differs: a wasted draft token costs milliseconds, while a target-model pass costs about 20 s. Stopping early can save little while giving up tokens that the pass could have checked.

My measurements agree. I tested three adaptive policies on Qwen3-14B against a fixed draft length [8]:

1. Policy 1A learning policy that adjusts k after each round cut k to 1 after a few rejections and was slower than the fixed draft: 7.7 against 7.2 s per token, on one prompt with 16 sampled tokens.
2. Policy 2A threshold policy tied with the fixed draft, within the run-to-run noise of the same test.
3. Policy 3A stopping policy that weighs rejection risk against drafting cost was slower on the four prompts: 9.8 against 9.2 s per token.

In these small streamed Qwen3-14B tests, adaptive length did not beat the fixed 8-token draft. That does not establish the result for other models or workloads.

One guess or a tree of guesses

A chain of draft tokens has one weak point: a single wrong token discards everything after it. A tree draft keeps several alternatives at each position, and the target model checks the whole tree in one pass, with each token seeing only its own branch [6]. The longest branch the target model agrees with is kept. Figure 1 compares the two with a simple example.

The variants side by side

Table 1 places the variants above by their draft source, the training they need, and the extra memory they use. A survey by Xia et al. compares many more [7].

What the tests show

For this streamed Qwen3-14B workload, the separate draft model with a fixed 8-token draft was the best option I tested. Untrained self-speculation had no accepted drafts, and the adaptive policies did not improve on the fixed length.

Tree drafts are a useful next experiment because one slow pass can check several alternatives. I have not measured trees locally; that discussion is literature context, not a laptop result.