Speculative Decoding Explained: How It Speeds Up LLM Inference

Speculative Decoding is a technique that accelerates large language model inference by generating several candidate tokens with a lightweight draft mechanism and verifying them together using the main model. Instead of requiring a separate expensive model execution for every output token, it can produce multiple accepted tokens in a single verification step.

The advantage comes from reducing sequential decoding work, not replacing the main model with a less accurate one. When implemented with exact verification, speculative decoding preserves the target model's output distribution. Understanding when that acceleration outweighs the cost of drafting and verification is an important part of LLM Inference Optimization.

How Speculative Decoding Works

A standard autoregressive language model generates one token, adds it to the context, and runs another decoding step to produce the next token. Every additional token depends on the preceding sequence.

Speculative decoding changes the execution process by separating token prediction from final verification.

A lightweight Draft Model proposes several tokens. The larger Target Model evaluates those proposals together and determines which tokens can become part of the final response.

The process becomes easier to understand through a concrete example.

Draft Generation and Token Verification

Consider an LLM completing this sentence:

"The server processes incoming requests"

A draft model proposes four additional tokens that would produce the continuation:

"using available GPU resources efficiently"

The proposed continuation is only a prediction. None of those tokens is accepted as final output until verification is complete.

The target model evaluates the drafted sequence in one parallel verification pass.

For this example, assume greedy decoding is enabled and the target model agrees with the first two proposed tokens but selects a different third token.

The result follows this sequence:

1. The draft model proposes four tokens.

2. The target model verifies the proposed continuation.

3. The first two tokens match the target model's greedy choices and are accepted.

4. The third proposal is rejected and replaced with the target model's selected token.

5. The remaining draft tokens are discarded because their context no longer matches the accepted continuation.

The next drafting step begins from the verified output.

Notice what happened: the target model advanced the response through multiple token positions without needing a separate sequential target execution for each accepted token.

That is the computational advantage behind speculative decoding.

The example uses greedy decoding to make the verification decision easy to follow. Probabilistic sampling uses a different acceptance mechanism, which is essential for preserving the target model's original distribution.

Draft Model and Target Model Roles

The Draft Model is responsible for producing candidate tokens quickly. It must be inexpensive enough that generating those candidates costs less than the target computation it helps avoid.

The Target Model remains responsible for determining the final output.

This distinction matters because a draft model does not need the same reasoning ability or parameter count as the target. Its job is to predict likely continuations that the target can verify efficiently.

A larger draft model can produce better predictions but requires more computation. A smaller draft model reduces drafting overhead but can generate more proposals that the target rejects.

The best combination is determined by both proposal quality and execution cost.

Speculative Decoding vs Autoregressive Decoding

Traditional autoregressive decoding processes output positions sequentially.

For a response containing 100 generated tokens, the baseline decoder advances through 100 sequential token generation steps, excluding prompt processing.

Speculative decoding attempts to reduce the number of expensive target model steps by accepting several tokens during each verification pass.

This does not eliminate the sequential dependency between tokens. The draft mechanism still proposes a continuation using information available at each position.

The improvement comes from allowing the target model to evaluate several positions in parallel.

Modern GPUs can perform substantial parallel computation, but decoding with a large model also requires repeated access to model weights and cached attention information.

When a decoding workload is limited by memory bandwidth, verifying several proposed positions together can use the hardware more efficiently than running separate target passes.

The practical result depends on how much additional work verification introduces.

If the target accepts most proposals, the system advances through more output tokens per expensive target execution.

If proposals are repeatedly rejected, drafting adds work without eliminating enough target executions to justify its cost.

Speculative decoding is faster only when the time saved through accepted tokens exceeds the additional cost of generating and checking proposals.

Token Acceptance and Output Quality

One of the most important technical distinctions is the difference between matching a target model's preferred token and preserving its complete probability distribution.

In greedy decoding, the model selects the token with the highest probability. Verification can compare proposed tokens against the target model's greedy selections and accept the longest valid prefix.

Sampling is different.

A model using probabilistic sampling assigns probabilities to possible next tokens. Two different tokens can both be valid samples from the same distribution.

Rejecting every proposal that differs from an independently sampled target token would waste useful proposals.

Exact speculative sampling solves this problem with a probability based acceptance rule.

Exact Speculative Sampling

Let (p(t)) represent the target model's probability for a proposed token and (q(t)) represent the draft model's probability for that token.

The acceptance probability is:

[
P(\text{accept }t)=\min\left(1,\frac{p(t)}{q(t)}\right)
]

This rule applies to a token sampled from the draft distribution with positive probability.

If the target assigns a probability at least as high as the draft model, the proposal is accepted.

If the target assigns a lower probability, acceptance is determined probabilistically.

When a token is rejected, the algorithm draws a replacement from a corrected distribution based on the positive difference between the target and draft probabilities.

That correction is essential. Simply sampling a replacement from the original target distribution after rejection would not preserve the required probabilities.

With the correct procedure, the final output follows the same distribution as standard sampling from the target model.

This is what makes the classical method an exact acceleration technique rather than an approximation that sacrifices model quality.

Preserving the Target Distribution

Preserving the output distribution does not mean every independent run will produce identical text.

If sampling is enabled, different valid token sequences can be drawn from the same probability distribution.

Exact speculative decoding preserves the probability of those sequences. It does not require independently executed generations to select the same random tokens.

This distinction becomes important when comparing model outputs during performance testing.

The correct validation is to establish that the implementation preserves the intended generation behavior, not to assume that two independently sampled responses must contain identical words.

Methods that intentionally relax verification rules can change the output distribution. Their performance and quality implications must be evaluated separately from exact speculative decoding.

Acceptance Rate and Decode Speedup

The performance of speculative decoding depends heavily on how many proposed tokens are accepted.

Two measurements help explain the result.

Draft Acceptance Rate measures the proportion of proposed tokens accepted by verification.

Mean Acceptance Length measures how many tokens are accepted during an average verification step. Some serving implementations report emitted tokens per step separately, including tokens generated directly by the target.

These metrics are related, but they answer different questions.

A system proposing two tokens per step can achieve a high acceptance rate while advancing fewer positions than a system proposing five tokens with a somewhat lower acceptance rate.

The number of accepted consecutive tokens also matters because verification stops accepting a drafted continuation at its first rejection.

Draft Length and Verification Cost

Increasing the number of draft tokens gives the target model an opportunity to accept a longer continuation.

However, additional proposals require more draft computation and a larger verification workload.

Suppose a draft mechanism proposes eight tokens, but the target consistently accepts only the first two.

The remaining proposals consume resources without contributing to the accepted continuation.

Reducing the number of proposed tokens can improve efficiency in that situation.

The opposite condition also matters. If the target accepts every token in a short draft, increasing draft length can reduce the number of target verification steps.

A useful configuration balances three quantities: the cost of drafting, the cost of verification, and the number of output tokens produced per verification cycle.

The maximum draft length should therefore be selected through measurement rather than increased simply because the hardware has spare capacity.

Draft Model Quality and Compatibility

Draft quality affects acceptance because the target and draft models must assign sufficiently compatible probabilities to proposed continuations.

A draft model trained on related data and using a compatible architecture can produce proposals that align more closely with the target model.

However, sharing a model family does not guarantee high acceptance.

The relationship also depends on the task, prompt distribution, sampling settings, and the draft model's predictive capability.

For a specialized application, a draft mechanism adapted to the actual workload can improve acceptance.

That benefit must be evaluated against the resources required to operate the draft mechanism.

A high acceptance rate is useful, but the final performance metric is the amount of accepted output produced per unit of total execution time.

Speculative Decoding Methods

The classical approach uses a smaller model to generate proposals, but modern inference systems support additional ways to produce candidate tokens.

These methods share the objective of reducing expensive sequential target executions while differing in how proposals are generated.

Speculative Decoding Methods Compared

The following comparison helps identify which approach deserves evaluation for a particular serving environment.

The methods cannot be ranked by speed without specifying the model, hardware, and workload.

A dedicated draft model introduces additional model execution and memory requirements. N Gram speculation reduces that cost but depends on finding useful patterns in existing tokens.

MTP and EAGLE require compatible model components and serving implementations. Their performance depends on how efficiently those components generate proposals.

The practical choice is the mechanism that provides the greatest reduction in target decoding time after accounting for its own overhead.

Latency, Throughput, and GPU Cost

Speculative decoding is primarily valuable for reducing the time spent generating output tokens.

It does not directly eliminate the work required to process an entirely new input prompt.

This means the distinction between initial response latency and subsequent token generation speed is important.

A workload dominated by long prompt processing can show a limited improvement in total response time even when speculative decoding accelerates the generation stage.

For a workload producing long responses, reducing sequential decode steps can have a larger impact on completion time.

Latency Under Different Batch Sizes

The benefits also change with request concurrency.

At low concurrency, a large target model can leave substantial GPU computing capacity unused during memory intensive decoding.

Speculative verification can use more of that available computation to process several proposed positions together.

At high concurrency, batching already allows the GPU to process multiple sequences during each execution step.

Adding speculative verification increases the amount of work and memory pressure associated with those batches.

The resulting gain can shrink, and an inefficient configuration can reduce aggregate throughput.

This creates a distinction between improving the response time of an individual request and increasing the total capacity of a serving system.

Both require measurement.

Inference Cost and Resource Requirements

Speculative decoding introduces resources beyond ordinary target model inference.

A separate Draft Model occupies memory and requires computation. Verification also needs the serving runtime to manage proposed tokens and their corresponding attention states.

Some methods avoid a separate draft model but still introduce additional execution components.

If speculative decoding lowers request latency while reducing the number of requests a GPU can serve per second, the infrastructure cost per completed request can increase.

That tradeoff can still be worthwhile for applications with strict responsiveness requirements.

The relevant question is whether the latency improvement justifies the additional resource consumption for the intended workload.

Speculative Decoding in Production

The first decision in a production deployment is whether decoding is the performance bottleneck.

If prompt processing, networking, or request queueing dominates total latency, accelerating decode cannot address the majority of the delay.

When decode latency is significant, the next step is choosing a compatible speculative method.

Inference engines such as vLLM and NVIDIA TensorRT LLM support speculative execution mechanisms, although available methods and configurations depend on the engine version and target model.

For a classical Draft Model implementation, compatibility between tokenization, model execution, and verification must be established.

Alternative assisted generation methods can support differing tokenizers through additional alignment mechanisms.

Selecting a Practical Configuration

When evaluating a deployment, I would start with the smallest configuration that provides a meaningful comparison against ordinary decoding.

The baseline should use the same target model, hardware, numerical precision, prompt distribution, and generation settings.

Next, select a compatible draft method and a limited number of proposed tokens.

Measure how many drafts are accepted and how much total time is spent generating and verifying them.

Then adjust the proposal length and compare the results.

This sequence is more informative than enabling a large number of speculative tokens immediately because it shows whether the bottleneck lies in draft generation, verification, or low acceptance.

The same configuration should then be tested under realistic concurrent traffic.

Managing Verified and Rejected Tokens

Production implementations must also keep attention cache state consistent with the accepted sequence.

Draft generation creates temporary state associated with proposed tokens.

When verification rejects part of the continuation, cache information belonging to invalid speculative positions cannot simply be treated as part of the confirmed output.

The serving runtime must retain or reconstruct the state required for the verified continuation.

Incorrect state management can lead to invalid predictions even when the token acceptance logic itself is correct.

This is one reason to prefer mature inference implementations over a simplified custom verification loop for production workloads.

Measuring Speculative Decoding Performance

A reliable benchmark compares the baseline decoder against the speculative configuration using identical workloads.

The most important measurement is not the number of tokens drafted. It is the improvement in completed target model output relative to total execution time.

Decode Speedup can be calculated as:

[
\text{Decode Speedup} =
\frac{\text{Baseline Decode Time}}
{\text{Speculative Decode Time}}
]

Both measurements must cover equivalent output workloads and consistent timing boundaries.

The benchmark should also record Time to First Token, Time per Output Token, total response latency, request throughput, and GPU memory consumption.

Acceptance metrics provide additional diagnostic information.

If acceptance is high but performance does not improve, draft execution or verification overhead deserves investigation.

If drafting is inexpensive but acceptance remains low, the proposal mechanism or draft length needs attention.

If individual response latency improves while aggregate throughput declines, the application must decide which objective carries greater operational value.

Testing With Representative Requests

Short responses and long responses should not be treated as interchangeable benchmark cases.

A request that generates only a few tokens gives speculative decoding limited opportunity to recover its setup and drafting costs.

Longer generations provide more opportunities to amortize those costs across accepted tokens.

Prompt characteristics also matter.

Repetitive continuations can behave differently from open ended text generation, while model sampling settings influence the relationship between draft and target probabilities.

The benchmark should therefore use the application's actual distribution of prompts, output lengths, and sampling configurations.

A result obtained from a single synthetic prompt does not establish production performance.

Final Thoughts

Speculative Decoding accelerates LLM inference by allowing a lightweight mechanism to propose several tokens while the target model verifies them together. Its strongest advantage is reducing expensive sequential decoding steps without changing the intended output distribution when exact verification is used.

The practical benefit depends on draft quality, acceptance length, verification overhead, and GPU workload. A configuration that improves individual response latency does not automatically improve aggregate throughput or infrastructure cost.

For production systems, the right decision is to measure accepted tokens, decode latency, and resource consumption together. Speculative decoding is valuable when it reduces total generation time enough to justify the additional work required to propose and verify tokens.

FAQ

Can I use speculative decoding with an LLM API?

Yes, if the API provider supports speculative decoding internally or exposes compatible configuration options. A client cannot directly control the provider's model execution when those options are unavailable. For self hosted deployments, an inference engine with speculative decoding support provides greater control over method selection and benchmarking.

Do the draft model and target model need the same tokenizer?

The classical approach requires compatible token representations so proposed tokens can be verified correctly. However, techniques such as Universal Assisted Decoding support models with different tokenizers by translating and aligning proposed text between tokenization systems. This introduces additional processing and requires explicit framework support.

Can the draft model run on a different GPU?

Yes, if the serving implementation supports separate device placement. The system must transfer the information required for drafting and verification between devices. Communication latency can reduce the performance benefit, so separate placement should be evaluated against running both components on the same device.

Will speculative decoding work with structured JSON output?

It can, provided the serving engine supports the combination of speculative decoding and structured output constraints. Proposed tokens must remain consistent with the permitted output grammar, and verification must preserve the required decoding behavior. Compatibility should be checked for the exact runtime and constraint mechanism.

Will speculative decoding produce exactly the same text with the same random seed?

Not necessarily across different implementations or execution paths. Exact speculative sampling preserves the target model's output distribution, but identical seeds do not guarantee identical results when random number consumption or numerical execution differs. Deterministic greedy decoding provides a stronger basis for direct token comparisons, subject to numerical consistency.