Speculative Decoding

Audio version — Estimated duration: 5 min 6 sec

Speculative Decoding — Faster Inference

Speculative decoding is a technique that uses a small, fast “draft” model to generate candidate tokens, which the large “target” model then verifies in parallel. Since the small model is much faster, and verification of multiple tokens can be batched, this can significantly speed up generation.

How It Works

1. A tiny draft model (e.g., 100M-500M params) generates N candidate tokens.
2. The large target model verifies all N candidates in one forward pass.
3. Accepted tokens are returned immediately; rejected tokens are re-generated.

Result: 2-3× faster generation, identical output quality (verified outputs are mathematically the same as without speculation).

Requirements

• A draft model in GGUF format. Must share the same tokenizer/vocabulary as the target model.
• Good draft models:
  • Speculative-0.5B-GGUF — good for Llama-3 8B
  • SmolLM2-135M-GGUF — tiny, works with many models
  • Qwen2.5-0.5B-GGUF — for Qwen target models

Usage

./bin/Release/llama-cli.exe ^
  -m "C:\AI\models\llama-3.1-8b.gguf" ^
  --draft-model "C:\AI\models\Speculative-0.5B.gguf" ^
  -p "Explain black holes" ^
  -ngl 99

Flags

Flag	Description
--draft-model	Path to the small draft GGUF model
--draft-n	Number of candidate tokens to speculate (default: 5, range: 3-16)
--draft-p	Minimum probability threshold for draft tokens (default: 0.9)

Tuning --draft-n

Higher values = more tokens verified per step, but lower acceptance rate:

• 5 (default) — good balance
• 8-16 — better if draft model is very accurate
• 3-4 — better if draft model is less accurate

When to Use Speculative Decoding

Scenario	Benefit
Long responses (>200 tokens)	High — more tokens to speculate
Short responses	Low — overhead of loading draft model
Batch/API serving	High — throughput improvement
Memory-constrained	Low — draft model uses extra RAM
CPU-only inference	High — CPU benefits most from speculation

Compatibility

• Works with all backends (CUDA, HIP, SYCL, Vulkan, CPU).
• The draft model can run on CPU while the target model runs on GPU — useful when VRAM is tight.
• Not all model pairs work. If the draft and target use different tokenizers, speculation won’t work.

# Draft on CPU, target on GPU
./bin/Release/llama-cli.exe -m "C:\AI\models\llama-8b.gguf" -ngl 99 ^
  --draft-model "C:\AI\models\tiny-draft.gguf" --draft-n 5

---

Last Updated: 2026-07-19