Speculative Decoding: Double Local LLM Tokens/Sec

INFERENCE ACCELERATION, MEMORY BANDWIDTH & LOCAL LLMS

Speculative Decoding in Local LLMs: Double Your Tokens/Sec Without VRAM Overhead

# Test Methodology: Batch size = 1, Temperature = 0.0 (Greedy), Speculative Lookahead (K) = 5
# Workloads: Code completion, RAG summarization, JSON schema extraction
# Target Model: Llama-3.3-70B-Instruct (Q4_K_M) | Draft Model: Llama-3.2-1B-Instruct (Q4_K_M)

[1] Dual NVIDIA RTX 3090 (48GB VRAM Total, 936 GB/s Bus per card) — Baseline:
    • Standard Generation Speed: 14.2 tokens/sec
    • GPU Tensor Core Compute Utilization: 28%
    • Memory Bus Utilization: 98.4% (Hard Memory Bandwidth Wall)

[2] Dual NVIDIA RTX 3090 — Speculative Decoding Enabled:
    • Generation Speed: 29.8 tokens/sec (2.09x Speedup)
    • Mean Acceptance Rate (Alpha): 74.2% on Code, 68.1% on General Prose
    • Additional VRAM Allocated: +1.42 GB (Draft weights + draft KV cache)
    • Generation Perplexity Shift: 0.000 (Exact Distribution Match)

[3] Single NVIDIA RTX 4090 24GB — Llama-3.1-8B Target + 1B Draft:
    • Baseline Autoregressive: 58.4 tokens/sec
    • Speculative Enabled: 94.1 tokens/sec (1.61x Speedup)
    • Mean Acceptance Rate (Alpha): 79.5%

[4] Apple Mac Studio M2 Ultra (192GB Unified Memory, 800 GB/s Bandwidth):
    • Baseline 70B Generation: 11.8 tokens/sec
    • Speculative Enabled (llama.cpp -md): 22.4 tokens/sec (1.89x Speedup)
    • Thermal Delta: Negligible (+2.8°C over continuous 4k output buffer)

1. The Physics of Generation: Why Local LLMs Hit the Memory Bandwidth Wall

To understand why speculative decoding works, we must analyze the hardware limits of modern deep learning inference. When executing local LLMs, execution phases split into two distinct regimes: the prefill phase (processing the prompt) and the decode phase (generating text token-by-token).

The prefill phase is compute-bound. All prompt tokens are processed simultaneously using dense Matrix-Matrix multiplications (GEMM). The GPU’s Tensor Cores operate at near peak FLOP/s capacity.

However, the autoregressive decode phase is strictly memory-bandwidth bound. Because generating token N+1 requires the context of token N, the GPU must perform Matrix-Vector operations (GEMV):

Consider a 70B parameter model quantized to 4 bits (~38 GB in VRAM). On a workstation featuring an NVIDIA RTX 4090 with 1,008 GB/s of theoretical memory bandwidth, the maximum possible sequential token rate is:

1,008 GB/s ÷ 38 GB ≈ 26.5 tokens/sec (theoretical upper ceiling)

Your powerful Tensor Cores remain largely idle, starved of data while waiting for parameter transfers. This physical bottleneck is precisely what speculative decoding is engineered to bypass.

2. Algorithmic Mechanics: Draft Engines, Parallel Verification & Rejection Sampling

Speculative decoding (originally conceptualized by Leviathan et al. and Chen et al.) alters the autoregressive workflow by unlinking token proposal from token verification:

1. The Lightweight Draft Phase: A tiny, highly optimized draft model (e.g., Llama 3.2 1B) runs sequentially for K steps. Because the draft model is tiny, reading its weights takes minimal memory bandwidth, allowing it to generate 4 to 6 candidate tokens rapidly.
2. The Target Verification Phase: The heavy target model (e.g., Llama 3.3 70B) takes all K candidate tokens plus the original prompt context and evaluates them in a single parallel forward pass.
3. Mathematical Rejection Sampling: The target model compares the draft token probabilities against its own probability distribution. If a token is accepted, generation advances. If a token is rejected, the target model samples a replacement token from a modified distribution, discarding subsequent speculations.

Crucially, because evaluating K tokens in parallel leverages the target model's idle compute capacity via GEMM operations, verifying 5 tokens takes virtually the same elapsed time as generating 1 token autoregressively! To understand how attention compression impacts this balance, read our audit on How Grouped-Query Attention (GQA) Slashes VRAM.

3. Acceptance Rate Dynamics (Alpha): When Speculation Fails

The net generation speedup of speculative decoding is governed by the Acceptance Rate (α)—the statistical percentage of draft tokens approved by the target model:

Acceptance rates are heavily domain-dependent:

• Code Completion & Syntax (α ≈ 75%–85%): Code consists of predictable structures, brackets, import statements, and boilerplate tokens. Draft models speculate code tokens with remarkable accuracy, delivering massive 2.2x+ speedups.
• Structured JSON & RAG Formatting (α ≈ 70%–80%): Strict formatting schemas constrain token choices, yielding high acceptance.
• High-Entropy Creative Prose (α ≈ 40%–55%): Nuanced stylistic generation causes frequent divergence between draft and target models. If acceptance rates fall below 45%, speculative decoding can run slower than raw autoregressive baseline due to wasted draft overhead.

4. VRAM Architecture: Allocating Weights, KV Caches & Speculation Lookahead

A common misconception among developers is that speculative decoding demands a secondary high-end GPU. In reality, modern quantization formats (AWQ vs EXL2 vs GGUF) make running draft models surprisingly lightweight:

• Draft Model Weight Footprint: An 8-bit quantized 1B parameter model consumes approximately 1.1 GB of VRAM. A 4-bit version occupies merely 700 MB.
• Draft KV Cache Duplication: The draft model requires its own Key-Value (KV) cache to project forward during the speculation window. However, because its context length only needs to cover the active prompt plus a small sliding window, this footprint rarely exceeds 200–400 MB.
• Target KV Cache Modifications: The target model must maintain an expanded speculative KV buffer to store unverified branches during tensor verification.

For exact capacity planning on workstations running multiple GPUs or Apple unified memory, review our definitive telemetry guide on Exact VRAM & GPU Sizing for 8B, 14B, 32B & 70B LLMs.

5. Implementation Playbook: Deploying Speculative Decoding in vLLM & llama.cpp

Modern inference engines support speculative decoding natively with minimal configuration flags:

Method 1: Enterprise Deployment via vLLM

In vLLM, pass the --speculative-model flag alongside your primary model deployment. The engine automatically synchronizes tokenizers and manages parallel tensor execution:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.3-70B-Instruct \
  --speculative-model meta-llama/Llama-3.2-1B-Instruct \
  --num-speculative-tokens 5 \
  --gpu-memory-utilization 0.92 \
  --tensor-parallel-size 2

Method 2: Local Workstation Acceleration via llama.cpp

For cross-platform local development on Apple Silicon or Windows workstations, llama.cpp provides the -md (model draft) argument:

./llama-cli \
  -m models/Llama-3.3-70B-Instruct-Q4_K_M.gguf \
  -md models/Llama-3.2-1B-Instruct-Q4_K_M.gguf \
  -n 512 \
  --draft 5 \
  -t 16 -ngl 99

To verify operating system kernel tuning and driver performance before scaling your serving instances, consult our benchmark on Linux vs Windows for Local AI: CUDA Audit.

For high-performance hardware comparisons across consumer GPUs and unified memory Mac rigs, examine our breakdown in Mac Studio Ultra vs RTX 4090 for Local LLMs.

7. Frequently Asked Technical Questions

What is speculative decoding in local LLMs?

Speculative decoding is an inference optimization algorithm where a fast, lightweight draft model (such as a 1B model) predicts multiple tokens sequentially, which are then verified simultaneously in a single forward pass by a larger target model (such as a 70B model). Because the target verification is compute-bound rather than memory-bandwidth bound, accepted tokens drastically increase generation speed (tokens per second) without modifying output probability distributions.

Does speculative decoding reduce model quality or perplexity?

No. Speculative decoding employs modified rejection sampling that mathematically guarantees the final output tokens match the exact probability distribution of the large target model. There is zero loss in generation quality, reasoning capability, or perplexity.

How much additional VRAM does speculative decoding require?

Speculative decoding requires additional VRAM proportional to the draft model's weight footprint plus its independent KV cache. For instance, pairing an 8-bit quantized 1B draft model alongside a 70B target typically consumes an extra 1.2 GB to 2.4 GB of VRAM, leaving most of your GPU capacity intact.

Can speculative decoding make local LLM inference slower?

Yes. If the draft model and target model have low token alignment (a low acceptance rate, typically under 45–50%), the time spent generating rejected draft tokens exceeds the savings of parallel verification. This commonly occurs when draft models are trained on completely different tokenizers or domains than the target model.

✓

Abdulrahman Maslmany

Lead Productivity & Systems Architect

"Memory bandwidth bounds arithmetic throughput. I analyze CUDA kernel scheduling, speculative draft-target alignments, and memory hierarchies to extract maximal token throughput from local AI silicon."

Connect with Abdulrahman on Dev.to, inspect models on Hugging Face, or read our complete Editorial Standards.