SPIRIT: Speed-Driven Online Adaptation for Self-Speculative Decoding
Training-free self-speculative decoding that adapts online to optimize window-level realized throughput rather than token agreement alone.
Up to 2.48x throughput over autoregressive decoding. NF4 self-draft · 1,000-request stream
Throughput drives adaptation.
- Self-draft
- Full verifier
- Adapt
using window-level realized throughput.
Research details
The problem
Agreement alone misses the execution cost.
Skipping attention or MLP sublayers changes both draft–target consistency and the cost of each draft–verify round. Acceptance observes only consistency. Useful throughput also accounts for verifier-produced tokens and how quickly the chosen configuration executes.
The mechanism
Throughput combines yield and execution rate.
Choose where to skip
A = attention · M = MLP
Dashed sublayers are skipped.
One choice, two effects
Acceptance and verifier bonus × drafted tokens per decode second.
Optimize what is served
Measured through full-target verification.
Measured evidence
Two views of the throughput landscape.
Acceptance vs. throughput
The most agreeable draft is not the fastest.
Decode throughput · tokens/s
Draft acceptance (%)
- Highest acceptance
- 99.41% · 24.36 tok/s
- Highest throughput
- 98.02% · 26.39 tok/s
Even within the highlighted ≈96–98% acceptance band, throughput spans 21.99–26.39 tokens/s.
The plot shows 400 configurations with acceptance ≥70%.
Skip ratio vs. layer pattern
Ratio gives structure. Placement leaves room.
Decode throughput · tokens/s
Skip ratio, r
Skip ratioThe skip ratio organizes the dominant consistency–speed trade-off.
Layer patternAt each ratio, different attention/MLP patterns still produce different throughput.
Spread across configurationsThe shaded band shows the middle 50% of measured configurations at each ratio.
Measured configuration sweep · LLaMA-2-13B · CNN/DM · FP16 · 21 skip ratios × 24 patterns · the same 24 prompts per configuration · decode-only.
Use realized throughput to localize a promising skip ratio, then refine the layer pattern at that ratio. Evaluate within-request candidates through the same decoding that produces output tokens.
Effective configurations also depend on the workload. When request behavior changes, runtime feedback guides whether to keep, reuse, refine, or rebuild the serving configuration.
The method
Search while decoding. Reuse when workloads recur.
SPIRIT adapts a self-draft configuration c = (r, L): how much computation to skip, and which sublayers to skip. It needs neither offline profiling nor a separately trained draft model. Full-target verification preserves the target model’s output distribution.
The serving objective
Committed tokens / elapsed decode time
- αw
- Accepted / proposed draft tokens
- βw
- Extra verifier tokens per proposed draft token
- νw
- Proposed draft tokens / total decode time
Within a request
Measure the draft path while it serves tokens.
Window w
Serve & measure
Gw tokens in Tw seconds
At the window boundary
Update the configuration
Prefix growth increases KV-cache work even with the same configuration. Calibrate for this cost, then smooth window noise.
Boundary updates need no additional model forward pass.
Observe, update, and decode again until the request ends.
Search detailsCalibration · ratio · pattern
Comparable window feedback
Separate configuration quality from prefix growth.
An online latency model estimates the cost of a growing prefix. It rebases measured throughput to a common reference length; exponential smoothing then reduces short-window noise.
- MeasureSw = Gw / Tw
Committed tokens per decode second.
- CalibrateS̃w = Sw · flen(xw)
Correct for the estimated length effect.
- SmoothQw = EMA[S̃w]
Score configurations within the request.
The length model uses latency per committed token and speculative-step coverage. Request-head profile matching uses raw throughput S and acceptance α.
Stage I · Skip ratio
How much should the draft skip?
Compare confidence bounds on window scores. The leader has the highest mean Q; a rival’s upper bound still reaches the leader’s lower bound.
- Zoom into the unresolved bracket; probe the midpoint of its wider side.
- Decode, update confidence, and compare the bounds again.
- Confirm the leader and validate the edges before locking r★.
Algorithm schematic; positions and intervals are illustrative.
Stage II · Layer pattern
Which sublayers should it skip?
Keep the skip count fixed. Mix broad coverage with proposals informed by configurations already scored at this ratio.
Random 96%
Explore layer positions.
GP-UCB 4%
Use prior scores.
- Record the candidate pattern and observed score.
- Retain the best observed pattern if it improves.
- Stop when stable or the request’s search budget is spent.
Both proposals keep the skip count fixed; the illustrated masks each skip four of eight sublayers.
Across requests
Recognize a returning workload.
A profile pairs a skip ratio and layer pattern with its request-head history: the mean and standard deviation of raw throughput S and draft acceptance α. Matching uses runtime evidence; no task labels are required.
Stored configuration
Ratio r★ + layer pattern L★
Request-head signature
Stored mean ± SD · Current head
Continue or compare
Compatible heads keep serving.
Supported deviations trigger probes of the active and stored profiles.
Every trial uses the same request head. Trial outputs are not committed.
Choose how to continue
- Stay
- Keep the active profile when it fits or trials are inconclusive.
- Switch
- Reuse an eligible stored configuration.
- Switch + refine
- Reuse its ratio; refine the layer pattern.
- Restart
- Search again when drift persists and no profile fits.
Profile reuse detailsHistory matching · switch eligibility
Compare each profile with its own history.
Each z is the head measurement’s deviation from that profile’s historical mean, divided by its historical standard deviation. Lower d indicates a closer fit.
A switch must pass all three checks.
- A closer fit to the candidate’s own history
- Higher relative probe merit
- Safe throughput
Request-head matching uses raw throughput and acceptance. The calibrated score Qw guides within-request search.
Schematic control flow. Within-request evaluation commits tokens; cross-request trials compare profiles separately. The control-flow diagram (PDF) gives the complete decision sequence.
The evidence
Online adaptation delivers sustained speedups.
Decode throughput relative to autoregressive decoding (AR), with full-target verification. The headline values are maxima from their respective settings.
- FP16 self-draft
- Up to1.74×
- NF4 self-draft
- Up to2.48x
Models & tasks
Leads in all nine FP16 model–task settings.
Summarization, reasoning, and text generation across the LLaMA-2 family.
Matched comparisons
Three models. Three tasks.
| Task | SWIFT | D&V | SPIRIT |
|---|---|---|---|
| CNN/DM | 1.19 | 1.33 | 1.37 |
| GSM8K | 1.21 | 1.20 | 1.27 |
| TinyStories | 1.10 | 1.16 | 1.20 |
| Task | SWIFT | D&V | SPIRIT |
|---|---|---|---|
| CNN/DM | 1.46 | 1.54 | 1.67 |
| GSM8K | 1.40 | 1.39 | 1.49 |
| TinyStories | 1.36 | 1.46 | 1.68 |
| Task | SWIFT | D&V | SPIRIT |
|---|---|---|---|
| CNN/DM | 1.29 | 1.41 | 1.50 |
| GSM8K | 1.39 | 1.42 | 1.57 |
| TinyStories | 1.45 | 1.49 | 1.74 |
Speedup / AR · FP16 self-drafts · paper Table 1. Each table uses the matching autoregressive baseline.
Broader tasks · SpecBench
+17.2% full-stream throughput over SWIFT.
1.462× versus 1.247× over AR on the same backend, including exploration and profile updates.
Speedup / AR
Batched serving
The gains persist with batching.
13B · speedup over matching batched AR · full-model verification at every batch size.
- Batch 1
- 1.44×
- Batch 2
- 1.42×
- Batch 4
- 1.23×
Each batch size uses its own autoregressive baseline.
Broader validation
Beyond the LLaMA-2 family.
Dense GQA · sparse MoE
Transfer across architectures.
The controller adapts to different draft paths while retaining full BF16 verification.
| Model | BF16 draft | NF4 draft |
|---|---|---|
| Qwen2.5-7BDense GQA | 1.304× | 1.358× |
| Qwen3-8B-BaseDense GQA | 1.273× | 1.343× |
| Mixtral-8×7BSparse MoE | 1.240× | 1.296× |
Aggregate over CNN/DM, GSM8K, and TinyStories. Point estimates shown; paper Table 6 reports 95% intervals.
Mixtral uses top-1 routing for drafts and full top-2 routing for verification.
Code generation · HumanEval
Faster decoding, with matching greedy accuracy.
CodeLLaMA gains speed while preserving the autoregressive pass@1 result in both evaluated settings.
| Model | Speedup / AR | Pass@1 |
|---|---|---|
| CodeLLaMA-13B | 1.40× | 34.1% |
| CodeLLaMA-34B | 1.47× | 48.1% |
Greedy decoding · all 164 HumanEval problems. Pass@1 matches AR for both models.
Paper Table 2 also reports the separate sampling evaluation.
Sustained NF4 acceleration
Gains persist as optimization costs are amortized.
Two complementary studies separate sustained serving performance from the effect of draft precision.
1,000-request stream
2.46× pooled decode speedup.
Cumulative throughput remains above AR throughout adaptation in all eight runs. Later requests amortize the initial optimization. The best full-stream run reaches 2.48x.
Cumulative decode speedup / AR · full stream
Cumulative decode speedup / AR · first 50
Search and reuse phases in the first 50 requests.
Requests served
SetupLLaMA-2-13B · TinyStories · fused NF4 self-draft / full FP16 verifier · one RTX A6000 per run.
Across runsEight distinct controller seeds, empty initial controllers, and the same ordered 1,000-request stream. The pooled curve uses each method’s token and time totals across runs.
Decode timingIncludes online search and profile updates; excludes prefill, model loading, and one-time draft setup.
Paired draft-precision study
+23.6% throughput with NF4 self-drafts.
Relative to FP16 self-drafts, with a request-paired bootstrap 95% interval of +21.4–26.3%.
Aggregate speedup / AR · 95% intervals
LLaMA-2-13B · CNN/DM, GSM8K, TinyStories · the same FP16 verifier.
Each draft precision is optimized separately.
Why adaptation pays off
Search quality and update cost both matter.
The studies below separate configuration quality, controller overhead, and the benefit of profile reuse.
Configuration discovery
Choose configurations for speed.
SPIRIT’s final and best-found speedups both reach 1.61×. SWIFT finds 1.52× but finishes at 1.41×; D&V finishes at 1.22×.
Replay speedup / AR
Online configuration-discovery time (s)
Selection matters.Agreement-based scores can misrank configurations by decode throughput.
Paper Fig. 3a · 13B · CNN/DM · FP16 · the same 20-request replay probe for all choices.
Controller update cost
Productive decoding doubles as evaluation.
Controller updates account for 8.2% of measured adaptation time in SPIRIT, versus 76.6% in SWIFT.
Mean total time · normalized ms
Paper Fig. 3b · 13B · CNN/DM · FP16 · step-normalized window timing · 13 requests, one seed.
Stacks include the <1% residual execution time.
Mechanism checks
Both the objective and the ratio matter.
Acceptance-only scoring and a fixed skip ratio each reduce mean speedup.
Paper Table 4 · stationary FP16 tasks.
| Variant | 13B | 70B |
|---|---|---|
| Full SPIRIT | 1.61× | 1.60× |
| Acceptance only | 1.32× | 1.21× |
| Fixed r = 0.5 | 1.53× | 1.46× |
The fixed-ratio variant keeps layer-pattern refinement enabled.
Profile reuse
+9.8–10.3% when workloads recur.
Returning tasks benefit from configurations found earlier, instead of starting each search again.
Stories → CNN/DM → GSM8K
Stories → CNN/DM → GSM8K
Full-stream speedup / AR
600-request FP16 streams · task shifts every 100 requests.
The gain compares full SPIRIT with SPIRIT without profile reuse.