NF4 self-drafts
Up to2.48x
decode throughput over AR
Acceleration that includes adaptation.
Measured over a 1,000-request stream, including online search and profile updates.
Speed-driven adaptation
In speculative decoding, optimizing for draft–target agreement alone can reduce throughput when a closer match comes at the expense of intrinsic draft speed. SPIRIT optimizes as it decodes, using window-level realized throughput to steer the search for a faster self-draft. Beyond a single request, SPIRIT adapts across workloads by recognizing familiar decoding patterns and reusing a saved draft configuration when a short probe confirms its speed advantage.
Productive decoding windows generate output and throughput feedback. SPIRIT calibrates this feedback to adapt the draft, then recalls useful profiles when workloads return. Full-target verification preserves the target distribution.
The full target scores all seven positions in parallel. Here it replaces ten with noon, the Harbor train’s departure time. Four accepted draft tokens and one correction give five committed tokens.
Skip 4 of 6
Run 1 and 4
Dots show illustrative throughput scores after prefix-length correction and smoothing. Whiskers mark confidence bounds on each ratio’s mean. With four skips fixed, the higher-scoring pattern runs sublayers 1 and 4.
B fits its own history more closely, earns a higher probe score and clears the throughput check. SPIRIT can therefore switch to B on this request.
What SPIRIT delivers
NF4 self-drafts
Up to2.48x
decode throughput over AR
Measured over a 1,000-request stream, including online search and profile updates.
FP16 self-drafts
Up to1.74×
decode throughput over AR
Ahead of SWIFT and D&V in all 9 LLaMA-2 model–task settings evaluated.
How much computation can a self-draft skip before the cost of draft–target disagreement outweighs the time saved? SPIRIT refines the draft as the response grows, guided by the realized throughput of each decoding window.
SPIRIT adapts as workloads shift, using early decoding feedback to decide whether to reuse a tuned draft, refine it for the new request, or search anew. Profile reuse raised mixed-stream throughput by 9.8–10.3% over the no-reuse baseline.
SPIRIT can skip sublayers and use NF4 weights to speed up drafting. The full target model verifies every proposal, so the output follows the same distribution as standard autoregressive decoding.
The problem
Skipping attention or MLP sublayers changes both draft–target consistency and the cost of each draft–verify round. Acceptance observes only consistency. Useful throughput also accounts for verifier-produced tokens and how quickly the chosen configuration executes.
The mechanism
Choose where to skip
A = attention · M = MLP
Dashed sublayers are skipped.
One choice, two effects
Acceptance and verifier bonus × drafted tokens per decode second.
Optimize what is served
Measured through full-target verification.
Measured evidence
Acceptance vs. throughput
Decode throughput · tokens/s
Draft acceptance (%)
Even within the highlighted ≈96–98% acceptance band, throughput spans 21.99–26.39 tokens/s.
The plot shows 400 configurations with acceptance ≥70%.
Skip ratio vs. layer pattern
Decode throughput · tokens/s
Skip ratio, r
Skip ratioThe skip ratio organizes the dominant consistency–speed trade-off.
Layer patternAt each ratio, different attention/MLP patterns still produce different throughput.
Spread across configurationsThe shaded band shows the middle 50% of measured configurations at each ratio.
Measured configuration sweep · LLaMA-2-13B · CNN/DM · FP16 · 21 skip ratios × 24 patterns · the same 24 prompts per configuration · decode-only.
Use realized throughput to localize a promising skip ratio, then refine the layer pattern at that ratio. Evaluate within-request candidates through the same decoding that produces output tokens.
Effective configurations also depend on the workload. When request behavior changes, runtime feedback guides whether to keep, reuse, refine, or rebuild the serving configuration.
The method
SPIRIT adapts a self-draft configuration c = (r, L): how much computation to skip, and which sublayers to skip. It needs neither offline profiling nor a separately trained draft model. Full-target verification preserves the target model’s output distribution.
The serving objective
Committed tokens / elapsed decode time
Within a request
Window w
Gw tokens in Tw seconds
At the window boundary
Prefix growth increases KV-cache work even with the same configuration. Calibrate for this cost, then smooth window noise.
Boundary updates need no additional model forward pass.
Observe, update, and decode again until the request ends.
Comparable window feedback
An online latency model estimates the cost of a growing prefix. It rebases measured throughput to a common reference length; exponential smoothing then reduces short-window noise.
Committed tokens per decode second.
Correct for the estimated length effect.
Score configurations within the request.
The length model uses latency per committed token and speculative-step coverage. Request-head profile matching uses raw throughput S and acceptance α.
Stage I · Skip ratio
Compare confidence bounds on window scores. The leader has the highest mean Q; a rival’s upper bound still reaches the leader’s lower bound.
Algorithm schematic; positions and intervals are illustrative.
Stage II · Layer pattern
Keep the skip count fixed. Mix broad coverage with proposals informed by configurations already scored at this ratio.
Random 96%
Explore layer positions.
GP-UCB 4%
Use prior scores.
Both proposals keep the skip count fixed; the illustrated masks each skip four of eight sublayers.
Across requests
A profile pairs a skip ratio and layer pattern with its request-head history: the mean and standard deviation of raw throughput S and draft acceptance α. Matching uses runtime evidence; no task labels are required.
Stored configuration
Ratio r★ + layer pattern L★
Request-head signature
Stored mean ± SD · Current head
Continue or compare
Compatible heads keep serving.
Supported deviations trigger probes of the active and stored profiles.
Every trial uses the same request head. Trial outputs are not committed.
Choose how to continue
Each z is the head measurement’s deviation from that profile’s historical mean, divided by its historical standard deviation. Lower d indicates a closer fit.
Request-head matching uses raw throughput and acceptance. The calibrated score Qw guides within-request search.
Schematic control flow. Within-request evaluation commits tokens; cross-request trials compare profiles separately. The control-flow diagram (PDF) gives the complete decision sequence.
The evidence
Decode throughput relative to autoregressive decoding (AR), with full-target verification. The headline values are maxima from their respective settings.
Models & tasks
Summarization, reasoning, and text generation across the LLaMA-2 family.
Matched comparisons
| Model | Task | SWIFT | D&V | SPIRIT |
|---|---|---|---|---|
| 13B-Chat | CNN/DM | 1.19 | 1.33 | 1.37 |
| GSM8K | 1.21 | 1.20 | 1.27 | |
| TinyStories | 1.10 | 1.16 | 1.20 | |
| 13B | CNN/DM | 1.46 | 1.54 | 1.67 |
| GSM8K | 1.40 | 1.39 | 1.49 | |
| TinyStories | 1.36 | 1.46 | 1.68 | |
| 70B | CNN/DM | 1.29 | 1.41 | 1.50 |
| GSM8K | 1.39 | 1.42 | 1.57 | |
| TinyStories | 1.45 | 1.49 | 1.74 |
Speedup / AR · FP16 self-drafts · paper Table 1. Each setting uses the matching autoregressive baseline.
Broader tasks · SpecBench
1.462× versus 1.247× over AR on the same backend, including exploration and profile updates.
Speedup / AR
Batched serving
13B · speedup over matching batched AR · full-model verification at every batch size.
Each batch size uses its own autoregressive baseline.
Dense GQA · sparse MoE
Transfer across architectures.
The controller adapts to different draft paths while retaining full BF16 verification.
| Model | BF16 draft | NF4 draft |
|---|---|---|
| Qwen2.5-7BDense GQA | 1.304× | 1.358× |
| Qwen3-8B-BaseDense GQA | 1.273× | 1.343× |
| Mixtral-8×7BSparse MoE | 1.240× | 1.296× |
Aggregate over CNN/DM, GSM8K, and TinyStories. Point estimates shown; paper Table 6 reports 95% intervals.
Mixtral uses top-1 routing for drafts and full top-2 routing for verification.
Code generation · HumanEval
Faster decoding, with matching greedy accuracy.
CodeLLaMA gains speed while preserving the autoregressive pass@1 result in both evaluated settings.
| Model | Speedup / AR | Pass@1 |
|---|---|---|
| CodeLLaMA-13B | 1.40× | 34.1% |
| CodeLLaMA-34B | 1.47× | 48.1% |
Greedy decoding · all 164 HumanEval problems. Pass@1 matches AR for both models.
Paper Table 2 also reports the separate sampling evaluation.
Sustained NF4 acceleration
Two complementary studies separate sustained serving performance from the effect of draft precision.
1,000-request stream
Cumulative throughput remains above AR throughout adaptation in all eight runs. Later requests amortize the initial optimization. The best full-stream run reaches 2.48x.
Cumulative decode speedup / AR · full stream
Cumulative decode speedup / AR · first 50
Search and reuse phases in the first 50 requests.
Requests served
SetupLLaMA-2-13B · TinyStories · fused NF4 self-draft / full FP16 verifier · one RTX A6000 per run.
Across runsEight distinct controller seeds, empty initial controllers, and the same ordered 1,000-request stream. The pooled curve uses each method’s token and time totals across runs.
Decode timingIncludes online search and profile updates; excludes prefill, model loading, and one-time draft setup.
Paired draft-precision study
Relative to FP16 self-drafts, with a request-paired bootstrap 95% interval of +21.4–26.3%.
Aggregate speedup / AR · 95% intervals
LLaMA-2-13B · CNN/DM, GSM8K, TinyStories · the same FP16 verifier.
Each draft precision is optimized separately.
Why adaptation pays off
The studies below separate configuration quality, controller overhead, and the benefit of profile reuse.
Configuration discovery
SPIRIT’s final and best-found speedups both reach 1.61×. SWIFT finds 1.52× but finishes at 1.41×; D&V finishes at 1.22×.
Replay speedup / AR
Online configuration-discovery time (s)
Selection matters.Agreement-based scores can misrank configurations by decode throughput.
Paper Fig. 3a · 13B · CNN/DM · FP16 · the same 20-request replay probe for all choices.
Controller update cost
Controller updates account for 8.2% of measured adaptation time in SPIRIT, versus 76.6% in SWIFT.
Mean total time · normalized ms
Paper Fig. 3b · 13B · CNN/DM · FP16 · step-normalized window timing · 13 requests, one seed.
Stacks include the <1% residual execution time.
Mechanism checks
Acceptance-only scoring and a fixed skip ratio each reduce mean speedup.
Paper Table 4 · stationary FP16 tasks.
| Variant | 13B | 70B |
|---|---|---|
| Full SPIRIT | 1.61× | 1.60× |
| Acceptance only | 1.32× | 1.21× |
| Fixed r = 0.5 | 1.53× | 1.46× |
The fixed-ratio variant keeps layer-pattern refinement enabled.
Profile reuse
Returning tasks benefit from configurations found earlier, instead of starting each search again.
Stories → CNN/DM → GSM8K
Stories → CNN/DM → GSM8K
Full-stream speedup / AR
600-request FP16 streams · task shifts every 100 requests.
The gain compares full SPIRIT with SPIRIT without profile reuse.
@inproceedings{liu2026spirit,
title = {{SPIRIT}: Speed-Driven Online Adaptation
for Self-Speculative Decoding},
author = {Liu, Chenghao and Zhan, Chengbo and
Lu, Hui and Rao, Jia},
booktitle = {Advances in Neural Information Processing Systems},
year = {2026},
url = {https://chenghaoliu519.github.io/spirit/}
}