How Speculative Decoding Speeds Up Inference

A small draft model proposes tokens before the large model verifies them in one pass. On my 8 GB laptop, fixed speculation reduced Qwen3-14B from 17.4 to 9.2 s per generated token at a similar GPU peak.

Oney ErgeOctober 14, 20266 min read

A streamed large model normally needs one full pass for each answer token. On this laptop, that pass takes about 20 s because most weights must be read from storage again.

Speculative decoding lets a smaller draft model guess ahead, then lets the large target model check those guesses in one pass [1]. The target model still chooses every kept token.

Drafting and checking tokens

The target here is Qwen3-14B. The draft is Qwen3-0.6B from the same family [4]; it stays on the GPU and uses about 1.3 GB.

For a fixed draft length k, one round is:

1The draft model proposes k tokens.
2The target model checks all k positions in one pass.
3Matching tokens are kept from the start of the draft. At the first mismatch, the target model's token replaces the draft and the rest is discarded.

Each pass produces between 1 and k + 1 tokens. Table 1 shows a four-token draft where one target pass adds three answer tokens.

Table 1. One round of speculative decoding with k = 4 draft tokens, simple example. The tokens are illustrative.

Position	Draft token	Target model's choice	Result
1	The	The	Accepted
2	capital	capital	Accepted
3	is	of	Replaced by "of"
4	Paris		Discarded
Tokens added by this pass			3: "The capital of"

Without speculation, those three tokens would need three target-model passes.

The same output as the large model alone

With greedy decoding, every kept token is the target model's own top choice for the same preceding text. The draft changes the number of target-model passes, not the intended output.

With sampling, a draft token cannot simply be compared with one choice. Speculative sampling accepts each draft token with a probability set by the scores of both models and, after a rejection, samples a replacement from a corrected distribution. Leviathan et al. and Chen et al. showed that the result follows the target model's own distribution exactly [1].

Parallel checking can change floating-point operation order, so matching output tokens remains a measured result rather than an assumption.

Tokens per pass and time per token

How many tokens one pass produces depends on how often the draft is right. If the target model accepts each draft token with the same probability, independently of the others, the expected number of tokens per pass is [1]:

E = \frac{1 - α^{k + 1}}{1 - α}

(1)

where α is the acceptance rate, the fraction of draft tokens that the target model accepts, and k is the number of draft tokens per round.

Figure 1 plots Eq. (1) for drafts of 4 and 8 tokens.

Figure 1. Expected tokens per pass of the target model from Eq. (1), for drafts of 4 and 8 tokens. Drafting 8 tokens instead of 4 pays off mainly when the draft is usually right.

Longer drafts pay off only when the draft is usually right. At a 0.5 acceptance rate, 8 tokens add little over 4; at 0.9, they add about 2 tokens per target pass.

The time per generated token follows from the cost of one pass and the cost of drafting:

t_{token} \approx \frac{t_{pass} + k t_{draft}}{E}

(2)

where t_pass is the time of one pass of the target model with the draft tokens included, and t_draft is the time the draft model needs for one token.

Here, t_pass is about 20 s and t_draft is tens of milliseconds [5]. Drafting is therefore small relative to one slow target pass. When the model fits on the GPU, drafting and verification are a larger fraction of the work.

Fixed speculation measured on my laptop

Afterimage [5] uses fixed speculation: the draft always proposes k = 8 tokens. I tested 4 prompts, 4 generated tokens each, greedy decoding, and a cleared page cache before each run.

The draft left 2.7 GB for the target model. The whole run reached a measured PyTorch peak of 3.8 GB.

Figure 2. Generating the 4 answer tokens for the gold prompt with Qwen3-14B on my laptop. Without speculation, Afterimage needs one pass per token. With fixed speculation, the draft model proposes 8 tokens and one pass keeps the 4 that were needed. Both totals are measured [5]; the equal passes and the length of the draft bar are simplified.

Table 2 shows all four prompts. Request time includes prompt prefill.

Table 2. Fixed speculation with Qwen3-14B as the target model and Qwen3-0.6B as the draft model, k = 8, on the RTX 3080 Laptop GPU. Each prompt generated 4 tokens with greedy decoding and a cleared page cache. Passes counts the passes of the target model; without speculation, every prompt needs 4. Source: Afterimage results [5].

Prompt	Answer	Passes	Request time
Chemical symbol for gold	Au	1	22 s
17 multiplied by 6	102	2	41 s
Missing expression in a Python function	x ** 2	3	61 s
Access code in a short note	7319	1	22 s

Two prompts finished in one pass; the code prompt needed three because the draft disagreed more often.

Across the four prompts, fixed speculation took 9.2 s per generated token at a 3.8 GB peak. Without speculation, the 4 GB Afterimage plan took 17 s at 3.9 GB. Every prompt produced the same output tokens in both runs.

In a later 8-token run, fixed speculation averaged about 4.6 s per token [5]; the longer answer spreads prefill across more tokens.

GPU memory required by the draft model

The draft model occupies about 1.3 GB throughout the run, leaving less space for target-model weights. Together with the target model's minimum, this configuration needs about 3 GB, so it cannot reach AirLLM's 1.6 GB operating point [5].

The gain depends on the prompt, and these answers are short. A fixed policy also proposes 8 tokens even when the draft is going poorly.

Why it helps here

In this streamed workload, a draft token took milliseconds while a target-model pass took about 20 s. Accepted guesses therefore remove expensive target-model passes.

Fixed 8-token drafts measured 9.2 s per generated token at a 3.8 GB peak, against 17 s at 3.9 GB without speculation. Next: the draft source, length, and verification shape change that trade-off.

References

Speculative decoding and speculative sampling: Y. Leviathan, M. Kalman, and Y. Matias. 2023. “Fast Inference from Transformers via Speculative Decoding,” ICML 2023; C. Chen et al. 2023. “Accelerating Large Language Model Decoding with Speculative Sampling,” arXiv:2302.01318.
M. Stern, N. Shazeer, and J. Uszkoreit. 2018. “Blockwise Parallel Decoding for Deep Autoregressive Models,” NeurIPS 2018.
R. Svirschevski et al. 2024. “SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices,” NeurIPS 2024.
Qwen Team. 2025. Qwen3 Technical Report. Model cards: Qwen/Qwen3-14B and Qwen/Qwen3-0.6B.
O. Erge, Afterimage. Method and results in docs/HOW_IT_WORKS.md; raw runs in results/2026-08-21_bounded_qwen3-14b_rtx3080_run1.json and the 8-token run.