Machine learning & computer systems

Chenghao Liu

Making language-model inference more efficient.

I’m a Ph.D. student in Computer Science at The University of Texas at Arlington, advised by Prof. Jia Rao. I develop adaptive inference methods and study how runtime and memory-system design shape their performance.

Selected work

NeurIPS 2026 Accepted · First author

SPIRIT: Speed-Driven Online Adaptation for Self-Speculative Decoding

Training-free self-speculative decoding that adapts online to optimize window-level realized throughput rather than token agreement alone.

Up to 2.48x throughput over autoregressive decoding. NF4 self-draft · 1,000-request stream

Throughput drives adaptation.

  1. Self-draft
  2. Full verifier
  3. Adapt
Throughput feedback
Tune the skip ratio r and layer pattern L
using window-level realized throughput.
Research details

The problem

Agreement alone misses the execution cost.

Skipping attention or MLP sublayers changes both draft–target consistency and the cost of each draft–verify round. Acceptance observes only consistency. Useful throughput also accounts for verifier-produced tokens and how quickly the chosen configuration executes.

The mechanism

Throughput combines yield and execution rate.

Choose where to skip

A = attention · M = MLP
Dashed sublayers are skipped.

One choice, two effects

α + βUseful token yield
νExecution rate

Acceptance and verifier bonus × drafted tokens per decode second.

Optimize what is served

Committed output tokensElapsed decode time

Measured through full-target verification.

Measured evidence

Two views of the throughput landscape.

Acceptance vs. throughput

The most agreeable draft is not the fastest.

Decode throughput · tokens/s

Acceptance does not uniquely determine throughput400 of 504 measured configurations with at least 70% acceptance. The highest acceptance is 99.41% at 24.36 tokens per second; the highest throughput is 26.39 tokens per second at 98.02% acceptance. In the highlighted approximately 96–98% acceptance band, throughput spans 21.99–26.39 tokens per second.20222426708090100Acceptance 97.12%; 21.987 tokens/s; skip ratio 0.303Acceptance 98.06%; 23.019 tokens/s; skip ratio 0.303Acceptance 96.53%; 22.362 tokens/s; skip ratio 0.303Acceptance 98.29%; 23.520 tokens/s; skip ratio 0.303Acceptance 98.20%; 23.263 tokens/s; skip ratio 0.303Acceptance 96.59%; 24.199 tokens/s; skip ratio 0.303Acceptance 98.82%; 24.175 tokens/s; skip ratio 0.303Acceptance 97.47%; 24.066 tokens/s; skip ratio 0.303Acceptance 98.82%; 23.765 tokens/s; skip ratio 0.303Acceptance 96.92%; 22.613 tokens/s; skip ratio 0.303Acceptance 97.26%; 23.699 tokens/s; skip ratio 0.303Acceptance 95.95%; 23.123 tokens/s; skip ratio 0.303Acceptance 99.13%; 24.615 tokens/s; skip ratio 0.303Acceptance 98.31%; 23.571 tokens/s; skip ratio 0.303Acceptance 96.71%; 22.863 tokens/s; skip ratio 0.303Acceptance 97.46%; 23.923 tokens/s; skip ratio 0.303Acceptance 94.06%; 22.126 tokens/s; skip ratio 0.303Acceptance 97.47%; 23.500 tokens/s; skip ratio 0.303Acceptance 96.48%; 23.758 tokens/s; skip ratio 0.303Acceptance 93.90%; 21.462 tokens/s; skip ratio 0.303Acceptance 97.93%; 23.385 tokens/s; skip ratio 0.303Acceptance 93.51%; 22.406 tokens/s; skip ratio 0.303Acceptance 96.38%; 23.303 tokens/s; skip ratio 0.303Acceptance 91.90%; 21.962 tokens/s; skip ratio 0.303Acceptance 98.43%; 23.237 tokens/s; skip ratio 0.316Acceptance 96.25%; 22.615 tokens/s; skip ratio 0.316Acceptance 93.81%; 21.967 tokens/s; skip ratio 0.316Acceptance 96.58%; 22.300 tokens/s; skip ratio 0.316Acceptance 97.53%; 22.498 tokens/s; skip ratio 0.316Acceptance 98.69%; 23.632 tokens/s; skip ratio 0.316Acceptance 94.23%; 23.113 tokens/s; skip ratio 0.316Acceptance 97.22%; 23.567 tokens/s; skip ratio 0.316Acceptance 97.14%; 24.521 tokens/s; skip ratio 0.316Acceptance 94.72%; 21.990 tokens/s; skip ratio 0.316Acceptance 97.36%; 23.029 tokens/s; skip ratio 0.316Acceptance 97.13%; 23.818 tokens/s; skip ratio 0.316Acceptance 88.81%; 21.613 tokens/s; skip ratio 0.316Acceptance 98.57%; 23.756 tokens/s; skip ratio 0.316Acceptance 94.76%; 22.603 tokens/s; skip ratio 0.316Acceptance 98.30%; 24.116 tokens/s; skip ratio 0.316Acceptance 92.03%; 22.090 tokens/s; skip ratio 0.316Acceptance 98.33%; 24.119 tokens/s; skip ratio 0.316Acceptance 97.58%; 22.949 tokens/s; skip ratio 0.316Acceptance 88.91%; 21.170 tokens/s; skip ratio 0.316Acceptance 90.09%; 21.900 tokens/s; skip ratio 0.316Acceptance 95.07%; 22.423 tokens/s; skip ratio 0.316Acceptance 92.82%; 22.301 tokens/s; skip ratio 0.316Acceptance 95.81%; 23.981 tokens/s; skip ratio 0.316Acceptance 96.89%; 22.738 tokens/s; skip ratio 0.329Acceptance 92.57%; 21.089 tokens/s; skip ratio 0.329Acceptance 95.78%; 22.500 tokens/s; skip ratio 0.329Acceptance 97.62%; 22.443 tokens/s; skip ratio 0.329Acceptance 95.81%; 22.677 tokens/s; skip ratio 0.329Acceptance 95.79%; 22.601 tokens/s; skip ratio 0.329Acceptance 96.94%; 24.056 tokens/s; skip ratio 0.329Acceptance 98.77%; 24.000 tokens/s; skip ratio 0.329Acceptance 98.31%; 23.910 tokens/s; skip ratio 0.329Acceptance 95.74%; 23.685 tokens/s; skip ratio 0.329Acceptance 95.78%; 24.489 tokens/s; skip ratio 0.329Acceptance 96.61%; 22.780 tokens/s; skip ratio 0.329Acceptance 97.53%; 23.284 tokens/s; skip ratio 0.329Acceptance 94.06%; 22.876 tokens/s; skip ratio 0.329Acceptance 96.05%; 24.383 tokens/s; skip ratio 0.329Acceptance 90.40%; 21.363 tokens/s; skip ratio 0.329Acceptance 96.48%; 22.841 tokens/s; skip ratio 0.329Acceptance 96.12%; 22.336 tokens/s; skip ratio 0.329Acceptance 96.56%; 24.074 tokens/s; skip ratio 0.329Acceptance 94.17%; 22.904 tokens/s; skip ratio 0.329Acceptance 87.14%; 20.644 tokens/s; skip ratio 0.329Acceptance 97.85%; 23.371 tokens/s; skip ratio 0.329Acceptance 94.26%; 24.572 tokens/s; skip ratio 0.329Acceptance 94.55%; 23.996 tokens/s; skip ratio 0.329Acceptance 99.41%; 24.359 tokens/s; skip ratio 0.342Acceptance 96.33%; 23.156 tokens/s; skip ratio 0.342Acceptance 95.92%; 23.206 tokens/s; skip ratio 0.342Acceptance 98.44%; 25.365 tokens/s; skip ratio 0.342Acceptance 98.57%; 24.247 tokens/s; skip ratio 0.342Acceptance 97.39%; 22.820 tokens/s; skip ratio 0.342Acceptance 95.60%; 24.482 tokens/s; skip ratio 0.342Acceptance 91.47%; 21.463 tokens/s; skip ratio 0.342Acceptance 97.47%; 23.759 tokens/s; skip ratio 0.342Acceptance 97.99%; 24.657 tokens/s; skip ratio 0.342Acceptance 93.81%; 23.005 tokens/s; skip ratio 0.342Acceptance 96.95%; 23.364 tokens/s; skip ratio 0.342Acceptance 93.73%; 22.781 tokens/s; skip ratio 0.342Acceptance 96.63%; 23.142 tokens/s; skip ratio 0.342Acceptance 98.35%; 23.594 tokens/s; skip ratio 0.342Acceptance 96.72%; 23.763 tokens/s; skip ratio 0.342Acceptance 93.91%; 24.718 tokens/s; skip ratio 0.342Acceptance 99.23%; 24.318 tokens/s; skip ratio 0.342Acceptance 94.63%; 23.182 tokens/s; skip ratio 0.342Acceptance 96.19%; 24.074 tokens/s; skip ratio 0.342Acceptance 93.19%; 23.572 tokens/s; skip ratio 0.342Acceptance 85.71%; 21.244 tokens/s; skip ratio 0.342Acceptance 93.48%; 23.699 tokens/s; skip ratio 0.342Acceptance 94.07%; 21.977 tokens/s; skip ratio 0.342Acceptance 96.26%; 22.607 tokens/s; skip ratio 0.355Acceptance 93.55%; 21.572 tokens/s; skip ratio 0.355Acceptance 91.81%; 21.665 tokens/s; skip ratio 0.355Acceptance 98.03%; 23.063 tokens/s; skip ratio 0.355Acceptance 96.87%; 22.826 tokens/s; skip ratio 0.355Acceptance 95.27%; 22.358 tokens/s; skip ratio 0.355Acceptance 95.11%; 23.863 tokens/s; skip ratio 0.355Acceptance 95.76%; 23.040 tokens/s; skip ratio 0.355Acceptance 96.02%; 22.897 tokens/s; skip ratio 0.355Acceptance 95.85%; 24.018 tokens/s; skip ratio 0.355Acceptance 99.09%; 24.470 tokens/s; skip ratio 0.355Acceptance 95.77%; 23.423 tokens/s; skip ratio 0.355Acceptance 94.94%; 23.901 tokens/s; skip ratio 0.355Acceptance 96.34%; 22.996 tokens/s; skip ratio 0.355Acceptance 94.57%; 22.290 tokens/s; skip ratio 0.355Acceptance 95.92%; 23.471 tokens/s; skip ratio 0.355Acceptance 92.61%; 22.702 tokens/s; skip ratio 0.355Acceptance 97.36%; 22.901 tokens/s; skip ratio 0.355Acceptance 91.41%; 23.287 tokens/s; skip ratio 0.355Acceptance 91.32%; 21.289 tokens/s; skip ratio 0.355Acceptance 93.31%; 22.977 tokens/s; skip ratio 0.355Acceptance 92.92%; 23.333 tokens/s; skip ratio 0.355Acceptance 94.64%; 23.315 tokens/s; skip ratio 0.355Acceptance 96.32%; 22.682 tokens/s; skip ratio 0.355Acceptance 93.80%; 21.881 tokens/s; skip ratio 0.368Acceptance 94.46%; 22.202 tokens/s; skip ratio 0.368Acceptance 97.19%; 24.433 tokens/s; skip ratio 0.368Acceptance 88.25%; 21.114 tokens/s; skip ratio 0.368Acceptance 95.61%; 22.395 tokens/s; skip ratio 0.368Acceptance 97.15%; 22.967 tokens/s; skip ratio 0.368Acceptance 97.96%; 24.041 tokens/s; skip ratio 0.368Acceptance 97.53%; 23.991 tokens/s; skip ratio 0.368Acceptance 92.90%; 24.319 tokens/s; skip ratio 0.368Acceptance 98.17%; 24.677 tokens/s; skip ratio 0.368Acceptance 90.81%; 22.964 tokens/s; skip ratio 0.368Acceptance 93.44%; 23.289 tokens/s; skip ratio 0.368Acceptance 89.79%; 22.796 tokens/s; skip ratio 0.368Acceptance 95.82%; 22.435 tokens/s; skip ratio 0.368Acceptance 92.87%; 22.817 tokens/s; skip ratio 0.368Acceptance 95.08%; 23.099 tokens/s; skip ratio 0.368Acceptance 96.36%; 23.827 tokens/s; skip ratio 0.368Acceptance 95.94%; 23.057 tokens/s; skip ratio 0.368Acceptance 88.81%; 22.554 tokens/s; skip ratio 0.368Acceptance 90.57%; 22.339 tokens/s; skip ratio 0.368Acceptance 92.93%; 23.538 tokens/s; skip ratio 0.368Acceptance 90.69%; 21.601 tokens/s; skip ratio 0.368Acceptance 87.22%; 21.701 tokens/s; skip ratio 0.368Acceptance 88.57%; 23.203 tokens/s; skip ratio 0.368Acceptance 95.79%; 22.514 tokens/s; skip ratio 0.395Acceptance 92.25%; 22.537 tokens/s; skip ratio 0.395Acceptance 92.94%; 22.008 tokens/s; skip ratio 0.395Acceptance 93.10%; 22.111 tokens/s; skip ratio 0.395Acceptance 89.62%; 21.040 tokens/s; skip ratio 0.395Acceptance 94.95%; 22.397 tokens/s; skip ratio 0.395Acceptance 92.40%; 23.095 tokens/s; skip ratio 0.395Acceptance 90.18%; 21.970 tokens/s; skip ratio 0.395Acceptance 94.73%; 24.306 tokens/s; skip ratio 0.395Acceptance 94.61%; 23.674 tokens/s; skip ratio 0.395Acceptance 92.72%; 24.513 tokens/s; skip ratio 0.395Acceptance 95.03%; 23.284 tokens/s; skip ratio 0.395Acceptance 90.89%; 22.260 tokens/s; skip ratio 0.395Acceptance 95.84%; 24.989 tokens/s; skip ratio 0.395Acceptance 91.09%; 21.902 tokens/s; skip ratio 0.395Acceptance 93.76%; 23.316 tokens/s; skip ratio 0.395Acceptance 94.06%; 24.386 tokens/s; skip ratio 0.395Acceptance 89.00%; 21.570 tokens/s; skip ratio 0.395Acceptance 89.97%; 22.422 tokens/s; skip ratio 0.395Acceptance 90.15%; 23.463 tokens/s; skip ratio 0.395Acceptance 93.24%; 23.246 tokens/s; skip ratio 0.395Acceptance 96.16%; 23.377 tokens/s; skip ratio 0.395Acceptance 81.20%; 21.137 tokens/s; skip ratio 0.395Acceptance 84.84%; 21.128 tokens/s; skip ratio 0.395Acceptance 96.23%; 23.107 tokens/s; skip ratio 0.408Acceptance 87.22%; 21.262 tokens/s; skip ratio 0.408Acceptance 96.96%; 22.805 tokens/s; skip ratio 0.408Acceptance 94.77%; 22.658 tokens/s; skip ratio 0.408Acceptance 95.86%; 23.150 tokens/s; skip ratio 0.408Acceptance 88.62%; 22.358 tokens/s; skip ratio 0.408Acceptance 94.51%; 22.993 tokens/s; skip ratio 0.408Acceptance 85.33%; 21.174 tokens/s; skip ratio 0.408Acceptance 94.66%; 24.120 tokens/s; skip ratio 0.408Acceptance 90.89%; 23.682 tokens/s; skip ratio 0.408Acceptance 92.00%; 22.393 tokens/s; skip ratio 0.408Acceptance 89.28%; 21.369 tokens/s; skip ratio 0.408Acceptance 91.04%; 22.950 tokens/s; skip ratio 0.408Acceptance 83.16%; 20.913 tokens/s; skip ratio 0.408Acceptance 91.60%; 22.250 tokens/s; skip ratio 0.408Acceptance 91.15%; 23.459 tokens/s; skip ratio 0.408Acceptance 90.53%; 21.930 tokens/s; skip ratio 0.408Acceptance 90.60%; 22.345 tokens/s; skip ratio 0.408Acceptance 88.25%; 21.940 tokens/s; skip ratio 0.408Acceptance 91.36%; 23.191 tokens/s; skip ratio 0.408Acceptance 83.46%; 21.286 tokens/s; skip ratio 0.408Acceptance 89.16%; 23.210 tokens/s; skip ratio 0.408Acceptance 86.53%; 21.726 tokens/s; skip ratio 0.408Acceptance 85.16%; 21.838 tokens/s; skip ratio 0.408Acceptance 91.44%; 21.728 tokens/s; skip ratio 0.421Acceptance 85.63%; 20.834 tokens/s; skip ratio 0.421Acceptance 93.74%; 23.777 tokens/s; skip ratio 0.421Acceptance 93.40%; 22.185 tokens/s; skip ratio 0.421Acceptance 93.44%; 22.543 tokens/s; skip ratio 0.421Acceptance 86.13%; 21.174 tokens/s; skip ratio 0.421Acceptance 96.38%; 22.823 tokens/s; skip ratio 0.421Acceptance 98.02%; 26.395 tokens/s; skip ratio 0.421Acceptance 89.34%; 23.307 tokens/s; skip ratio 0.421Acceptance 95.15%; 23.730 tokens/s; skip ratio 0.421Acceptance 93.90%; 23.580 tokens/s; skip ratio 0.421Acceptance 93.22%; 22.561 tokens/s; skip ratio 0.421Acceptance 95.00%; 23.478 tokens/s; skip ratio 0.421Acceptance 90.84%; 21.961 tokens/s; skip ratio 0.421Acceptance 91.98%; 23.044 tokens/s; skip ratio 0.421Acceptance 96.45%; 24.748 tokens/s; skip ratio 0.421Acceptance 95.18%; 23.391 tokens/s; skip ratio 0.421Acceptance 94.21%; 24.052 tokens/s; skip ratio 0.421Acceptance 83.31%; 21.903 tokens/s; skip ratio 0.421Acceptance 92.45%; 23.623 tokens/s; skip ratio 0.421Acceptance 88.49%; 21.501 tokens/s; skip ratio 0.421Acceptance 90.44%; 22.483 tokens/s; skip ratio 0.421Acceptance 86.70%; 22.075 tokens/s; skip ratio 0.421Acceptance 89.38%; 23.779 tokens/s; skip ratio 0.421Acceptance 85.06%; 21.481 tokens/s; skip ratio 0.434Acceptance 91.85%; 22.547 tokens/s; skip ratio 0.434Acceptance 87.86%; 21.748 tokens/s; skip ratio 0.434Acceptance 89.72%; 21.908 tokens/s; skip ratio 0.434Acceptance 91.88%; 22.498 tokens/s; skip ratio 0.434Acceptance 89.84%; 21.726 tokens/s; skip ratio 0.434Acceptance 91.28%; 24.075 tokens/s; skip ratio 0.434Acceptance 91.28%; 22.876 tokens/s; skip ratio 0.434Acceptance 90.94%; 22.728 tokens/s; skip ratio 0.434Acceptance 95.18%; 24.849 tokens/s; skip ratio 0.434Acceptance 92.11%; 22.883 tokens/s; skip ratio 0.434Acceptance 96.42%; 23.448 tokens/s; skip ratio 0.434Acceptance 89.95%; 22.891 tokens/s; skip ratio 0.434Acceptance 94.53%; 24.524 tokens/s; skip ratio 0.434Acceptance 90.25%; 23.349 tokens/s; skip ratio 0.434Acceptance 94.30%; 23.220 tokens/s; skip ratio 0.434Acceptance 86.28%; 21.153 tokens/s; skip ratio 0.434Acceptance 93.07%; 23.123 tokens/s; skip ratio 0.434Acceptance 80.80%; 20.857 tokens/s; skip ratio 0.434Acceptance 83.43%; 21.259 tokens/s; skip ratio 0.434Acceptance 92.57%; 23.051 tokens/s; skip ratio 0.434Acceptance 82.68%; 21.743 tokens/s; skip ratio 0.434Acceptance 87.70%; 22.503 tokens/s; skip ratio 0.434Acceptance 74.04%; 20.213 tokens/s; skip ratio 0.434Acceptance 89.01%; 21.655 tokens/s; skip ratio 0.447Acceptance 77.01%; 20.055 tokens/s; skip ratio 0.447Acceptance 88.80%; 22.113 tokens/s; skip ratio 0.447Acceptance 90.89%; 22.342 tokens/s; skip ratio 0.447Acceptance 92.10%; 22.054 tokens/s; skip ratio 0.447Acceptance 88.22%; 22.501 tokens/s; skip ratio 0.447Acceptance 86.85%; 23.116 tokens/s; skip ratio 0.447Acceptance 93.89%; 23.544 tokens/s; skip ratio 0.447Acceptance 85.51%; 21.957 tokens/s; skip ratio 0.447Acceptance 91.35%; 22.860 tokens/s; skip ratio 0.447Acceptance 93.50%; 23.225 tokens/s; skip ratio 0.447Acceptance 93.98%; 24.600 tokens/s; skip ratio 0.447Acceptance 87.90%; 22.050 tokens/s; skip ratio 0.447Acceptance 90.52%; 22.338 tokens/s; skip ratio 0.447Acceptance 79.86%; 21.423 tokens/s; skip ratio 0.447Acceptance 85.54%; 21.333 tokens/s; skip ratio 0.447Acceptance 94.68%; 23.840 tokens/s; skip ratio 0.447Acceptance 78.82%; 21.159 tokens/s; skip ratio 0.447Acceptance 90.62%; 22.859 tokens/s; skip ratio 0.447Acceptance 84.47%; 23.106 tokens/s; skip ratio 0.447Acceptance 88.64%; 24.006 tokens/s; skip ratio 0.447Acceptance 88.82%; 22.002 tokens/s; skip ratio 0.447Acceptance 93.32%; 25.277 tokens/s; skip ratio 0.447Acceptance 90.62%; 22.173 tokens/s; skip ratio 0.461Acceptance 85.68%; 21.422 tokens/s; skip ratio 0.461Acceptance 94.31%; 23.394 tokens/s; skip ratio 0.461Acceptance 87.16%; 21.615 tokens/s; skip ratio 0.461Acceptance 80.39%; 20.486 tokens/s; skip ratio 0.461Acceptance 77.32%; 20.009 tokens/s; skip ratio 0.461Acceptance 88.95%; 22.605 tokens/s; skip ratio 0.461Acceptance 89.93%; 22.550 tokens/s; skip ratio 0.461Acceptance 93.64%; 24.647 tokens/s; skip ratio 0.461Acceptance 89.00%; 23.357 tokens/s; skip ratio 0.461Acceptance 90.85%; 23.916 tokens/s; skip ratio 0.461Acceptance 93.20%; 23.922 tokens/s; skip ratio 0.461Acceptance 82.46%; 21.839 tokens/s; skip ratio 0.461Acceptance 81.79%; 21.542 tokens/s; skip ratio 0.461Acceptance 83.32%; 21.765 tokens/s; skip ratio 0.461Acceptance 93.01%; 23.488 tokens/s; skip ratio 0.461Acceptance 82.66%; 21.391 tokens/s; skip ratio 0.461Acceptance 80.15%; 22.165 tokens/s; skip ratio 0.461Acceptance 86.17%; 22.944 tokens/s; skip ratio 0.461Acceptance 88.52%; 22.227 tokens/s; skip ratio 0.461Acceptance 75.74%; 20.792 tokens/s; skip ratio 0.461Acceptance 76.76%; 20.805 tokens/s; skip ratio 0.461Acceptance 78.03%; 21.277 tokens/s; skip ratio 0.461Acceptance 73.39%; 20.069 tokens/s; skip ratio 0.474Acceptance 88.52%; 22.302 tokens/s; skip ratio 0.474Acceptance 81.91%; 20.848 tokens/s; skip ratio 0.474Acceptance 87.05%; 21.826 tokens/s; skip ratio 0.474Acceptance 76.56%; 20.429 tokens/s; skip ratio 0.474Acceptance 86.74%; 22.559 tokens/s; skip ratio 0.474Acceptance 86.04%; 22.184 tokens/s; skip ratio 0.474Acceptance 94.43%; 24.392 tokens/s; skip ratio 0.474Acceptance 90.99%; 23.297 tokens/s; skip ratio 0.474Acceptance 83.64%; 21.650 tokens/s; skip ratio 0.474Acceptance 86.22%; 23.662 tokens/s; skip ratio 0.474Acceptance 85.82%; 22.480 tokens/s; skip ratio 0.474Acceptance 92.29%; 23.093 tokens/s; skip ratio 0.474Acceptance 84.29%; 21.813 tokens/s; skip ratio 0.474Acceptance 89.43%; 23.274 tokens/s; skip ratio 0.474Acceptance 81.56%; 21.568 tokens/s; skip ratio 0.474Acceptance 71.86%; 19.889 tokens/s; skip ratio 0.474Acceptance 86.84%; 21.994 tokens/s; skip ratio 0.474Acceptance 73.41%; 21.844 tokens/s; skip ratio 0.474Acceptance 75.87%; 20.814 tokens/s; skip ratio 0.474Acceptance 70.42%; 20.127 tokens/s; skip ratio 0.474Acceptance 82.77%; 21.646 tokens/s; skip ratio 0.474Acceptance 81.33%; 21.268 tokens/s; skip ratio 0.500Acceptance 83.02%; 21.464 tokens/s; skip ratio 0.500Acceptance 83.49%; 21.456 tokens/s; skip ratio 0.500Acceptance 79.06%; 20.964 tokens/s; skip ratio 0.500Acceptance 76.88%; 20.694 tokens/s; skip ratio 0.500Acceptance 86.69%; 22.175 tokens/s; skip ratio 0.500Acceptance 81.84%; 21.498 tokens/s; skip ratio 0.500Acceptance 86.68%; 22.981 tokens/s; skip ratio 0.500Acceptance 81.39%; 21.620 tokens/s; skip ratio 0.500Acceptance 79.27%; 21.496 tokens/s; skip ratio 0.500Acceptance 86.85%; 22.808 tokens/s; skip ratio 0.500Acceptance 90.59%; 24.022 tokens/s; skip ratio 0.500Acceptance 90.62%; 23.775 tokens/s; skip ratio 0.500Acceptance 82.31%; 21.523 tokens/s; skip ratio 0.500Acceptance 93.75%; 23.758 tokens/s; skip ratio 0.500Acceptance 85.56%; 22.283 tokens/s; skip ratio 0.500Acceptance 78.12%; 21.416 tokens/s; skip ratio 0.500Acceptance 84.16%; 22.075 tokens/s; skip ratio 0.500Acceptance 75.12%; 20.786 tokens/s; skip ratio 0.500Acceptance 78.62%; 21.155 tokens/s; skip ratio 0.513Acceptance 73.69%; 20.515 tokens/s; skip ratio 0.513Acceptance 84.11%; 21.783 tokens/s; skip ratio 0.513Acceptance 82.04%; 21.571 tokens/s; skip ratio 0.513Acceptance 86.79%; 22.897 tokens/s; skip ratio 0.513Acceptance 88.26%; 23.155 tokens/s; skip ratio 0.513Acceptance 87.85%; 23.393 tokens/s; skip ratio 0.513Acceptance 92.01%; 23.249 tokens/s; skip ratio 0.513Acceptance 83.20%; 21.889 tokens/s; skip ratio 0.513Acceptance 87.43%; 22.616 tokens/s; skip ratio 0.513Acceptance 93.05%; 24.305 tokens/s; skip ratio 0.513Acceptance 77.88%; 21.455 tokens/s; skip ratio 0.513Acceptance 76.15%; 21.431 tokens/s; skip ratio 0.513Acceptance 74.81%; 20.917 tokens/s; skip ratio 0.513Acceptance 80.41%; 21.132 tokens/s; skip ratio 0.513Acceptance 70.85%; 20.473 tokens/s; skip ratio 0.513Acceptance 74.93%; 20.523 tokens/s; skip ratio 0.526Acceptance 85.73%; 22.060 tokens/s; skip ratio 0.526Acceptance 92.14%; 23.118 tokens/s; skip ratio 0.526Acceptance 78.91%; 20.921 tokens/s; skip ratio 0.526Acceptance 83.63%; 22.133 tokens/s; skip ratio 0.526Acceptance 87.04%; 22.927 tokens/s; skip ratio 0.526Acceptance 84.04%; 22.925 tokens/s; skip ratio 0.526Acceptance 84.83%; 22.544 tokens/s; skip ratio 0.526Acceptance 91.65%; 24.869 tokens/s; skip ratio 0.526Acceptance 85.21%; 22.300 tokens/s; skip ratio 0.526Acceptance 71.01%; 20.611 tokens/s; skip ratio 0.526Acceptance 76.26%; 21.450 tokens/s; skip ratio 0.526Acceptance 79.50%; 21.640 tokens/s; skip ratio 0.526Acceptance 75.92%; 20.948 tokens/s; skip ratio 0.526Acceptance 72.74%; 21.017 tokens/s; skip ratio 0.526Acceptance 72.92%; 20.871 tokens/s; skip ratio 0.526Acceptance 70.31%; 20.392 tokens/s; skip ratio 0.526Acceptance 73.37%; 20.792 tokens/s; skip ratio 0.539Acceptance 77.71%; 21.101 tokens/s; skip ratio 0.539Acceptance 70.44%; 20.140 tokens/s; skip ratio 0.539Acceptance 75.98%; 20.756 tokens/s; skip ratio 0.539Acceptance 71.65%; 20.263 tokens/s; skip ratio 0.539Acceptance 81.51%; 22.080 tokens/s; skip ratio 0.539Acceptance 83.27%; 22.733 tokens/s; skip ratio 0.539Acceptance 82.83%; 22.542 tokens/s; skip ratio 0.539Acceptance 79.11%; 21.677 tokens/s; skip ratio 0.539Acceptance 81.08%; 22.612 tokens/s; skip ratio 0.539Acceptance 81.20%; 21.924 tokens/s; skip ratio 0.539Acceptance 74.83%; 21.086 tokens/s; skip ratio 0.539Acceptance 78.23%; 21.239 tokens/s; skip ratio 0.539Acceptance 82.14%; 22.524 tokens/s; skip ratio 0.539Acceptance 77.66%; 21.539 tokens/s; skip ratio 0.539Acceptance 77.63%; 21.201 tokens/s; skip ratio 0.553Acceptance 79.37%; 21.402 tokens/s; skip ratio 0.553Acceptance 72.92%; 21.129 tokens/s; skip ratio 0.553Acceptance 77.90%; 21.888 tokens/s; skip ratio 0.553Acceptance 83.67%; 23.574 tokens/s; skip ratio 0.553Acceptance 70.74%; 21.079 tokens/s; skip ratio 0.553Acceptance 87.54%; 24.091 tokens/s; skip ratio 0.553Acceptance 84.51%; 22.415 tokens/s; skip ratio 0.553Acceptance 78.47%; 21.517 tokens/s; skip ratio 0.553Acceptance 84.87%; 22.592 tokens/s; skip ratio 0.553Acceptance 72.84%; 20.777 tokens/s; skip ratio 0.566Acceptance 82.45%; 22.879 tokens/s; skip ratio 0.566Acceptance 79.83%; 22.448 tokens/s; skip ratio 0.566Acceptance 77.80%; 22.001 tokens/s; skip ratio 0.566Acceptance 85.47%; 23.764 tokens/s; skip ratio 0.566Acceptance 73.65%; 21.288 tokens/s; skip ratio 0.566Acceptance 70.11%; 20.379 tokens/s; skip ratio 0.566Acceptance 75.63%; 21.643 tokens/s; skip ratio 0.566Acceptance 77.18%; 21.606 tokens/s; skip ratio 0.579Acceptance 82.90%; 22.560 tokens/s; skip ratio 0.579Acceptance 74.69%; 22.263 tokens/s; skip ratio 0.579Acceptance 76.11%; 22.132 tokens/s; skip ratio 0.579Acceptance 76.12%; 23.353 tokens/s; skip ratio 0.605Acceptance 70.35%; 21.855 tokens/s; skip ratio 0.605Acceptance 70.76%; 21.844 tokens/s; skip ratio 0.605Highest acceptance: 99.41%, 24.36 tokens/sHighest throughput: 98.02%, 26.39 tokens/s24.3626.39

Draft acceptance (%)

Highest acceptanceHighest throughput
Highest acceptance
99.41% · 24.36 tok/s
Highest throughput
98.02% · 26.39 tok/s

Even within the highlighted ≈96–98% acceptance band, throughput spans 21.99–26.39 tokens/s.

The plot shows 400 configurations with acceptance ≥70%.

Skip ratio vs. layer pattern

Ratio gives structure. Placement leaves room.

Decode throughput · tokens/s

Skip ratio structures the throughput landscapeAll 504 configurations: 21 skip ratios and 24 layer patterns per ratio. The blue line is the best measured throughput at each ratio, the gray line is the median, and the shaded band is the middle 50% of configurations, not a confidence interval.1520250.30.40.50.6

Skip ratio, r

Best per ratioMedianMiddle 50%

Skip ratioThe skip ratio organizes the dominant consistency–speed trade-off.

Layer patternAt each ratio, different attention/MLP patterns still produce different throughput.

Spread across configurationsThe shaded band shows the middle 50% of measured configurations at each ratio.

Measured configuration sweep · LLaMA-2-13B · CNN/DM · FP16 · 21 skip ratios × 24 patterns · the same 24 prompts per configuration · decode-only.

Design implication

Use realized throughput to localize a promising skip ratio, then refine the layer pattern at that ratio. Evaluate within-request candidates through the same decoding that produces output tokens.

Effective configurations also depend on the workload. When request behavior changes, runtime feedback guides whether to keep, reuse, refine, or rebuild the serving configuration.

The method

Search while decoding. Reuse when workloads recur.

SPIRIT adapts a self-draft configuration c = (r, L): how much computation to skip, and which sublayers to skip. It needs neither offline profiling nor a separately trained draft model. Full-target verification preserves the target model’s output distribution.

The serving objective

Sw = GwTw = (αw + βw) νw

Committed tokens / elapsed decode time

αw
Accepted / proposed draft tokens
βw
Extra verifier tokens per proposed draft token
νw
Proposed draft tokens / total decode time

Within a request

Measure the draft path while it serves tokens.

Window w

Serve & measure
Self-draft with cw
Full target verifies
Commit accepted + verifier tokens

Gw tokens in Tw seconds

At the window boundary

Update the configuration
Qw = EMA[Sw · flen(xw)]

Prefix growth increases KV-cache work even with the same configuration. Calibrate for this cost, then smooth window noise.

Choose the next (r, L)Ratio localization → pattern refinement

Boundary updates need no additional model forward pass.

Window w + 1

Continue decoding
Self-draft with cw+1

Run the selected configuration in the next productive window.

D → V ▪▪D → V ▪▪▪

Full-target verification continues throughout.

Search details

Comparable window feedback

Separate configuration quality from prefix growth.

An online latency model estimates the cost of a growing prefix. It rebases measured throughput to a common reference length; exponential smoothing then reduces short-window noise.

  1. Measure
    Sw = Gw / Tw

    Committed tokens per decode second.

  2. Calibrate
    S̃w = Sw · flen(xw)

    Correct for the estimated length effect.

  3. Smooth
    Qw = EMA[S̃w]

    Score configurations within the request.

The length model uses latency per committed token and speculative-step coverage. Request-head profile matching uses raw throughput S and acceptance α.

Stage I · Skip ratio

How much should the draft skip?

Compare confidence bounds on window scores. The leader has the highest mean Q; a rival’s upper bound still reaches the leader’s lower bound.

LeaderRivalRuled out
Confidence-guided ratio search, schematicThe leader has the highest mean score. Ratios whose upper bound reaches the leader’s lower bound remain rivals. Probe the midpoint of the wider unresolved interval, then update bounds. This is an algorithm illustration, not measured data.rLeader’s lower boundProbe the wider gap’s midpoint
  1. Zoom into the unresolved bracket; probe the midpoint of its wider side.
  2. Decode, update confidence, and compare the bounds again.
  3. Confirm the leader and validate the edges before locking r★.

Algorithm schematic; positions and intervals are illustrative.

Stage II · Layer pattern

Which sublayers should it skip?

Keep the skip count fixed. Mix broad coverage with proposals informed by configurations already scored at this ratio.

Random 96%

Explore layer positions.

GP-UCB 4%

Use prior scores.

Run in the next productive windowObserve Qw+1
  1. Record the candidate pattern and observed score.
  2. Retain the best observed pattern if it improves.
  3. Stop when stable or the request’s search budget is spent.

Both proposals keep the skip count fixed; the illustrated masks each skip four of eight sublayers.

Across requests

Recognize a returning workload.

A profile pairs a skip ratio and layer pattern with its request-head history: the mean and standard deviation of raw throughput S and draft acceptance α. Matching uses runtime evidence; no task labels are required.

Stored configuration

Ratio r★ + layer pattern L★

Continue or compare

Compatible heads keep serving.
Supported deviations trigger probes of the active and stored profiles.

Every trial uses the same request head. Trial outputs are not committed.

Choose how to continue

Stay
Keep the active profile when it fits or trials are inconclusive.
Switch
Reuse an eligible stored configuration.
Switch + refine
Reuse its ratio; refine the layer pattern.
Restart
Search again when drift persists and no profile fits.
Profile reuse details
Compare each profile with its own history.
d(p) = |zS| + ω|zα|

Each z is the head measurement’s deviation from that profile’s historical mean, divided by its historical standard deviation. Lower d indicates a closer fit.

A switch must pass all three checks.
  • A closer fit to the candidate’s own history
  • Higher relative probe merit
  • Safe throughput

Request-head matching uses raw throughput and acceptance. The calibrated score Qw guides within-request search.

Schematic control flow. Within-request evaluation commits tokens; cross-request trials compare profiles separately. The control-flow diagram (PDF) gives the complete decision sequence.

The evidence

Online adaptation delivers sustained speedups.

Decode throughput relative to autoregressive decoding (AR), with full-target verification. The headline values are maxima from their respective settings.

FP16 self-draft
Up to1.74×
NF4 self-draft
Up to2.48x

Models & tasks

Leads in all nine FP16 model–task settings.

Summarization, reasoning, and text generation across the LLaMA-2 family.

Matched comparisons

Three models. Three tasks.
LLaMA-2-13B-Chat
TaskSWIFTD&VSPIRIT
CNN/DM1.191.331.37
GSM8K1.211.201.27
TinyStories1.101.161.20
LLaMA-2-13B
TaskSWIFTD&VSPIRIT
CNN/DM1.461.541.67
GSM8K1.401.391.49
TinyStories1.361.461.68
LLaMA-2-70B
TaskSWIFTD&VSPIRIT
CNN/DM1.291.411.50
GSM8K1.391.421.57
TinyStories1.451.491.74

Speedup / AR · FP16 self-drafts · paper Table 1. Each table uses the matching autoregressive baseline.

Broader tasks · SpecBench

+17.2% full-stream throughput over SWIFT.

1.462× versus 1.247× over AR on the same backend, including exploration and profile updates.

After adaptation · six task comparisons
SWIFTSPIRIT
Post-adaptation SpecBench resultsSpeedup over AR, three-seed means with standard deviations. Translation: SWIFT 1.291 ± 0.017, SPIRIT 1.444 ± 0.037; Summarization: SWIFT 1.394 ± 0.018, SPIRIT 1.559 ± 0.013; QA: SWIFT 1.296 ± 0.072, SPIRIT 1.475 ± 0.011; Math: SWIFT 1.219 ± 0.066, SPIRIT 1.562 ± 0.006; RAG: SWIFT 1.292 ± 0.021, SPIRIT 1.458 ± 0.058; Multi-turn: SWIFT 1.278 ± 0.013, SPIRIT 1.480 ± 0.0521.01.21.41.6TranslationTranslation, SWIFT: 1.291 ± 0.017Translation, SPIRIT: 1.444 ± 0.037SummarizationSummarization, SWIFT: 1.394 ± 0.018Summarization, SPIRIT: 1.559 ± 0.013QAQA, SWIFT: 1.296 ± 0.072QA, SPIRIT: 1.475 ± 0.011MathMath, SWIFT: 1.219 ± 0.066Math, SPIRIT: 1.562 ± 0.006RAGRAG, SWIFT: 1.292 ± 0.021RAG, SPIRIT: 1.458 ± 0.058Multi-turnMulti-turn, SWIFT: 1.278 ± 0.013Multi-turn, SPIRIT: 1.480 ± 0.052

Speedup / AR

LLaMA-2-13B · post-adaptation performance · three-seed mean ± SD.

Batched serving

The gains persist with batching.

13B · speedup over matching batched AR · full-model verification at every batch size.

Batch 1
1.44×
Batch 2
1.42×
Batch 4
1.23×

Each batch size uses its own autoregressive baseline.

Broader validation

Beyond the LLaMA-2 family.

Dense GQA · sparse MoE

Transfer across architectures.

The controller adapts to different draft paths while retaining full BF16 verification.

Decode speedup / matched AR
ModelBF16 draftNF4 draft
Qwen2.5-7BDense GQA1.304×1.358×
Qwen3-8B-BaseDense GQA1.273×1.343×
Mixtral-8×7BSparse MoE1.240×1.296×

Aggregate over CNN/DM, GSM8K, and TinyStories. Point estimates shown; paper Table 6 reports 95% intervals.

Mixtral uses top-1 routing for drafts and full top-2 routing for verification.

Code generation · HumanEval

Faster decoding, with matching greedy accuracy.

CodeLLaMA gains speed while preserving the autoregressive pass@1 result in both evaluated settings.

Greedy decoding on HumanEval
ModelSpeedup / ARPass@1
CodeLLaMA-13B1.40×34.1%
CodeLLaMA-34B1.47×48.1%

Greedy decoding · all 164 HumanEval problems. Pass@1 matches AR for both models.

Paper Table 2 also reports the separate sampling evaluation.

Sustained NF4 acceleration

Gains persist as optimization costs are amortized.

Two complementary studies separate sustained serving performance from the effect of draft precision.

1,000-request stream

2.46× pooled decode speedup.

Cumulative throughput remains above AR throughout adaptation in all eight runs. Later requests amortize the initial optimization. The best full-stream run reaches 2.48x.

Pooled throughputEight runs

Cumulative decode speedup / AR · full stream

Cumulative decode speedup, 1,000 requestsEight runs with distinct controller seeds, each starting from an empty controller on the same ordered 1,000-request TinyStories stream. LLaMA-2-13B uses fused NF4 self-drafts with full FP16 verification. Gray lines show individual runs; the blue curve uses each method’s token and time totals pooled across runs. Cumulative throughput stays above AR throughout adaptation in all eight runs. Full-stream speedup is 2.46× pooled and 2.48× for the best run. 1.01.52.02.5125050075010002.46× pooled

Cumulative decode speedup / AR · first 50

Cumulative decode speedup, first 50 requestsEight runs with distinct controller seeds, each starting from an empty controller on the same ordered 1,000-request TinyStories stream. LLaMA-2-13B uses fused NF4 self-drafts with full FP16 verification. Gray lines show individual runs; the blue curve uses each method’s token and time totals pooled across runs. Cumulative throughput stays above AR throughout adaptation in all eight runs. Full-stream speedup is 2.46× pooled and 2.48× for the best run. Search and reuse phases in the first 50 requests.1.01.52.02.512550SearchReuse1.42×

Search and reuse phases in the first 50 requests.

Requests served

SetupLLaMA-2-13B · TinyStories · fused NF4 self-draft / full FP16 verifier · one RTX A6000 per run.

Across runsEight distinct controller seeds, empty initial controllers, and the same ordered 1,000-request stream. The pooled curve uses each method’s token and time totals across runs.

Decode timingIncludes online search and profile updates; excludes prefill, model loading, and one-time draft setup.

Paired draft-precision study

+23.6% throughput with NF4 self-drafts.

Relative to FP16 self-drafts, with a request-paired bootstrap 95% interval of +21.4–26.3%.

Aggregate speedup / AR · 95% intervals

Separate three-task draft-precision comparisonLLaMA-2-13B, same full FP16 verifier, independently optimized draft configurations. Aggregate speedup over AR: FP16 1.603×, 95% interval 1.584–1.628; NF4 1.981×, interval 1.964–2.017. NF4 versus FP16 throughput ratio 1.236, request-paired bootstrap 95% interval 1.214–1.263. These are separate from the 1,000-request runs.0.01.02.01.603×FP161.981×NF4

LLaMA-2-13B · CNN/DM, GSM8K, TinyStories · the same FP16 verifier.

Each draft precision is optimized separately.

Why adaptation pays off

Search quality and update cost both matter.

The studies below separate configuration quality, controller overhead, and the benefit of profile reuse.

Configuration discovery

Choose configurations for speed.

SPIRIT’s final and best-found speedups both reach 1.61×. SWIFT finds 1.52× but finishes at 1.41×; D&V finishes at 1.22×.

SPIRITSWIFTD&V

Replay speedup / AR

Published configuration-discovery tracePaper Figure 3a. Shared 20-request replay probes, LLaMA-2-13B, CNN/DM, FP16. Solid lines show the selected configuration, dashed lines the best replay speedup found so far. D&V: final 1.217×, best 1.294×; SWIFT: final 1.409×, best 1.521×; SPIRIT: final 1.610×, best 1.610×1.01.21.41.60100200300D&V: 115.24 seconds, current 1.294×, best 1.294×D&V: 194.04 seconds, current 1.287×, best 1.294×D&V: 303.02 seconds, current 1.217×, best 1.294×SWIFT: 2.20 seconds, current 1.320×, best 1.320×SWIFT: 2.46 seconds, current 1.312×, best 1.320×SWIFT: 3.23 seconds, current 1.354×, best 1.354×SWIFT: 3.52 seconds, current 1.279×, best 1.354×SWIFT: 4.17 seconds, current 1.351×, best 1.354×SWIFT: 41.57 seconds, current 1.521×, best 1.521×SWIFT: 94.38 seconds, current 1.425×, best 1.521×SWIFT: 116.70 seconds, current 1.400×, best 1.521×SWIFT: 137.54 seconds, current 1.489×, best 1.521×SWIFT: 197.75 seconds, current 1.512×, best 1.521×SWIFT: 256.11 seconds, current 1.409×, best 1.521×SPIRIT: 0.47 seconds, current 1.323×, best 1.323×SPIRIT: 1.06 seconds, current 1.301×, best 1.323×SPIRIT: 6.76 seconds, current 1.241×, best 1.323×SPIRIT: 9.61 seconds, current 1.284×, best 1.323×SPIRIT: 10.14 seconds, current 1.234×, best 1.323×SPIRIT: 33.03 seconds, current 1.295×, best 1.323×SPIRIT: 33.56 seconds, current 1.116×, best 1.323×SPIRIT: 60.49 seconds, current 1.367×, best 1.367×SPIRIT: 86.75 seconds, current 1.263×, best 1.367×SPIRIT: 88.77 seconds, current 1.253×, best 1.367×SPIRIT: 94.28 seconds, current 1.486×, best 1.486×SPIRIT: 206.37 seconds, current 1.578×, best 1.578×SPIRIT: 294.91 seconds, current 1.610×, best 1.610×1.61×

Online configuration-discovery time (s)

Current choiceBest found so far

Selection matters.Agreement-based scores can misrank configurations by decode throughput.

Paper Fig. 3a · 13B · CNN/DM · FP16 · the same 20-request replay probe for all choices.

Controller update cost

Productive decoding doubles as evaluation.

Controller updates account for 8.2% of measured adaptation time in SPIRIT, versus 76.6% in SWIFT.

DraftVerifyUpdate

Mean total time · normalized ms

Normalized adaptation-window costSPIRIT: 105.8 normalized milliseconds; 8.2% update share; SWIFT: 704.7 normalized milliseconds; 76.6% update share. Paper Figure 3b: step-normalized window timing on LLaMA-2-13B, CNN/DM, FP16; 13 requests, one seed. SPIRIT window means are divided by six speculative steps; SWIFT uses window means.0200400600800SPIRIT draft: 46.356 normalized msSPIRIT verify: 49.761 normalized msSPIRIT other: 0.955 normalized msSPIRIT update: 8.697 normalized ms105.8SPIRITSWIFT draft: 103.918 normalized msSWIFT verify: 59.261 normalized msSWIFT other: 1.718 normalized msSWIFT update: 539.783 normalized ms704.7SWIFT

Paper Fig. 3b · 13B · CNN/DM · FP16 · step-normalized window timing · 13 requests, one seed.

Stacks include the <1% residual execution time.

Mechanism checks

Both the objective and the ratio matter.

Acceptance-only scoring and a fixed skip ratio each reduce mean speedup.

Paper Table 4 · stationary FP16 tasks.

Mean speedup / AR · three tasks per model
Variant13B70B
Full SPIRIT1.61×1.60×
Acceptance only1.32×1.21×
Fixed r = 0.51.53×1.46×

The fixed-ratio variant keeps layer-pattern refinement enabled.

Profile reuse

+9.8–10.3% when workloads recur.

Returning tasks benefit from configurations found earlier, instead of starting each search again.

First pass

Stories → CNN/DM → GSM8K

Workloads return

Stories → CNN/DM → GSM8K

Without reuseFull SPIRIT

Full-stream speedup / AR

Profile reuse under recurring workload shifts600-request FP16 streams with a task change every 100 requests. Speedup over AR. 13B: without reuse 1.43×, with reuse 1.57×; 70B: without reuse 1.45×, with reuse 1.60×0.01.01.51.43×1.57×13B1.45×1.60×70B

600-request FP16 streams · task shifts every 100 requests.

The gain compares full SPIRIT with SPIRIT without profile reuse.

Three connected questions guide my work: how much computation is needed, how it should execute, and how its data should move.

Efficient language-model inference

Reducing inference cost through speculative decoding, diffusion language models, and adaptive computation.

Systems-aware optimization

Adapting execution to changing workloads using measured throughput and latency.

Memory systems for AI

Managing KV caches and data movement across GPU and pooled-memory systems.

Education

  1. 2024 — Present

    The University of Texas at Arlington

    Ph.D. in Computer Science

    Advisor: Prof. Jia Rao

  2. 2020 — 2023

    University of Southern California

    M.S. in Computer Science

  3. 2016 — 2020

    Wuhan University of Technology

    B.Eng. in Computer Science

Experience

Jul — Dec 2021

Advanced Institute of Information Technology, Peking University

NLP Algorithm Engineer Intern

Developed transformer-based models for clinical entity and relation extraction, and designed a personalized reviewer–paper matching system.

Let’s talk research.

liuchenghao97@gmail.com

Mochi

Chenghao’s research guide

A little guide. A closer look.

Hi, I’m Mochi. Ask me about Chenghao’s background, research, or the ideas behind SPIRIT.