---
title: "FlashInfer — paper by Ye et al. (2025) · MLSys 2025"
description: "FlashInfer replaces them with one code-generation-based engine built on three ideas: Unified… 92 references, cited by 173 papers here."
authors: ["Zihao Ye", "Lequn Chen", "Ruihang Lai", "Wuwei Lin", "Yineng Zhang", "Stephanie Wang", "Tianqi Chen", "Baris Kasikci", "Vinod Grover", "Arvind Krishnamurthy"]
year: 2025
venue: "MLSys 2025"
source: "https://arxiv.org/abs/2501.01005"
arxiv: "2501.01005"
licence: "CC BY-SA 4.0 (arXiv OAI)"
licence_url: "http://creativecommons.org/licenses/by-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ye-2025-flashinfer-efficient-customizable-attention-engine/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ye-2025-flashinfer-efficient-customizable-attention-engine/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/) / FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving

# FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving

**Authors:** [Zihao Ye](https://md.research.elamur.ai/free/en/authors/zihao-ye/), [Lequn Chen](https://md.research.elamur.ai/free/en/authors/lequn-chen/), Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, [Tianqi Chen](https://md.research.elamur.ai/free/en/authors/tianqi-chen/), [Baris Kasikci](https://md.research.elamur.ai/free/en/authors/baris-kasikci/), Vinod Grover and [Arvind Krishnamurthy](https://md.research.elamur.ai/free/en/authors/arvind-krishnamurthy/) · **Year:** 2025 · **Venue:** MLSys 2025 · **Licence:** [CC BY-SA 4.0 (arXiv OAI)](http://creativecommons.org/licenses/by-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2501.01005) · [PDF](https://arxiv.org/pdf/2501.01005) · [arXiv 2501.01005](https://arxiv.org/abs/2501.01005) · [OpenAlex](https://openalex.org/W4406032994)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [173](https://md.research.elamur.ai/free/en/papers/ye-2025-flashinfer-efficient-customizable-attention-engine/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
FlashInfer is an attention engine for LLM inference serving. It targets two problems: serving workloads keep changing, and attention variants and GPU architectures differ widely. Today each serving system ships its own narrow attention kernels, which is costly to maintain. FlashInfer replaces them with one code-generation-based engine built on three ideas:

- **Unified storage format:** KV-cache layouts such as paged storage and radix trees are all expressed as block-sparse matrices. Block sizes can be very small, down to the vector level, and formats can be composed, for example to handle shared prefixes.
- **Customizable kernels:** a template lets users define their own attention variant, which JIT compilation turns into optimized block-sparse kernels.
- **Load-balanced scheduling:** tile sizes are fixed at compile time, while work is distributed at runtime according to actual sequence lengths. This keeps the setup compatible with CUDAGraph.

FlashInfer is already used inside SGLang, vLLM and MLC-Engine. Against compiler backends on a serving benchmark, it cuts inter-token latency by 29–69%. It also gives gains for long-context inference and for parallel generation. This matters because one well-tuned engine can serve many systems and new attention variants without hand-writing kernels each time.

## Key points
- **End-to-end gains:** 29–69% lower inter-token latency than compiler backends on an LLM serving benchmark. Long-context inference is 28–30% faster, and parallel generation sees a 13–17% speedup.
- **vs. FlexAttention:** on the AttentionGym benchmark (NVIDIA H100), FlashInfer is faster in all four tested variants, and the gap grows with sequence length. For example, with ALiBi at length 16,384 it reaches about 578 TFLOPS/s versus about 435 for FlexAttention. The authors credit Hopper features (warp specialization, TMA) and register-level control in CUTLASS.
- **Fine-grained sparsity (Quest decoding, block size 16):** the paper reports up to 20x speedup over FlexAttention at long sequences. Latency stays in the tens of microseconds, while PyTorch SDPA and FlexAttention take hundreds to over a thousand.
- **Scheduler ablation (Llama 3.1-8B-Instruct, SGLang, H100):** with long inputs sampled from U(4096, 16,384), inter-token latency drops from 13.89 ms without the scheduler to 8.63 ms with it.
- **vLLM integration:** about 13% lower inter-token latency with an fp8 KV-cache. With bf16 there are slight regressions, caused by Python-side host overhead.

## Interesting ideas
- **One sparse abstraction for many layouts:** describing paged, tree-shaped and masked caches all as block-sparse matrices means a single kernel family covers all of them. Any system juggling several memory layouts could reduce code duplication the same way.
- **Partial results as mergeable states:** each attention result is kept as an (output, log-sum-exp) pair, combined with an associative merge operator, much like summation in matrix multiply. This lets work be split, reordered or offloaded freely, and the trick suits any softmax-style reduction that must be computed in chunks.
- **Gather into contiguous memory for small blocks:** scattered rows are first collected into contiguous shared memory, and then dense tensor-core instructions are applied. This makes very small sparse blocks efficient, which helps wherever sparsity is fine-grained but dense hardware units are available.
- **Split decisions by when information is known:** tile sizes are chosen at compile time, while load balancing happens at runtime from real sequence lengths. This reconciles dynamic inputs with static execution graphs like CUDAGraph and could apply to other workloads with variable-length batches.
- **Ablations on synthetic length distributions:** besides realistic traces, inputs are drawn from controlled uniform length ranges. This shows where a scheduler actually helps (long, uneven inputs) and is a cheap way to stress-test any batching system.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 598 works · 191,226 citations
- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 402 works · 212,763 citations
- [Nvidia](https://md.research.elamur.ai/free/en/companies/nvidia/) — Company, Santa Clara, United States · 37 works · 12,656 citations
- [Nvidia (United Kingdom)](https://md.research.elamur.ai/free/en/companies/nvidia-united-kingdom/) — Company, Reading, United Kingdom · 27 works · 11,141 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover and Arvind Krishnamurthy. 2025. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving. MLSys 2025.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ye2025flashinfer,
  title = {{FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving}},
  author = {Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover and Arvind Krishnamurthy},
  booktitle = {MLSys 2025},
  year = {2025},
  eprint = {2501.01005},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.01005}
}
```

- [← previous Large Language Diffusion Models](https://md.research.elamur.ai/free/en/papers/nie-2025-large-language-diffusion-models/)
- [next → Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2025-towards-reasoning-era-survey-long/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/)

## References

92 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., Tumanov, A., and Ramjee, R. [Taming throughput-latency tradeoff in llm inference with sarathi-serve](https://md.research.elamur.ai/free/en/papers/agrawal-2024-taming-throughput-latency-tradeoff-llm/). Proceedings of 18th USENIX Symposium on Operating Systems Design and Implementation, 2024, Santa Clara, 2024.
2. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. [GQA: training generalized multi-query transformer models from multi-head checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/). In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Proce…
3. Ansel, J., Yang, E., He, H., Gimelshein, N., Jain, A., Voznesensky, M., Bao, B., Bell, P., Berard, D., Burovski, E., Chauhan, G., Chourdia, A., Constable, W., Desmaison, A., DeVito, Z., Ellison, E., Feng, W., Gong, J., Gschwind, M., Hirsh, B., Huang, S., Kalambarkar, K., Kirsch, L., Lazos, M., Lezca… — [PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation](https://md.research.elamur.ai/free/en/papers/ansel-2024-pytorch-2-faster-machine-learning/)
4. Beltagy, I., Peters, M. E., and Cohan, A. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). CoRR, abs/2004.05150, 2020. URL https://arxiv.org/abs/2004. 05150. [↗](https://arxiv.org/abs/2004)
5. Buluç, A., Fineman, J. T., Frigo, M., Gilbert, J. R., and Leiserson, C. E. Parallel sparse matrix-vector and matrix-transpose-vector multiplication using compressed sparse blocks. In auf der Heide, F. M. and Bender, M. A. (eds.), SPAA 2009: Proceedings of the 21st Annual ACM Symposium on Parallelism… [find](https://scholar.google.com/scholar?q=Bulu%C3%A7%2C+A%2E%2C+Fineman%2C+J%2E+T%2E%2C+Frigo%2C+M%2E%2C+Gilbert%2C+J%2E+R%2E%2C+and+Leiserson%2C+C%2E+E%2E+Parallel+sparse+matrix%2Dvector+and+matrix%2Dtranspose%2Dvector+multiplication+using+compressed+sparse+blocks%2E+In+auf+der+Heide%2C+F%2E)
6. ACM, 2009. doi: 10. 1145/1583991.1584053. URL https://doi.org/ 10.1145/1583991.1584053. — [doi:10.1145/1583991.1584053](https://doi.org/10.1145/1583991.1584053) [find](https://scholar.google.com/scholar?q=ACM%2C+2009%2E+doi%3A+10%2E+1145%2F1583991%2E1584053%2E+URL+https%3A%2F%2Fdoi%2Eorg%2F+10%2E1145%2F1583991%2E1584053%2E+%E2%80%94+%5Bdoi%3A10%2E1145%2F1583991%2E1584053%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E1145%2F1583991%2E1584053%29)
7. Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., and Dao, T. [Medusa: Simple LLM inference acceleration framework with multiple decoding heads](https://md.research.elamur.ai/free/en/papers/cai-2024-medusa-simple-llm-inference-acceleration/). CoRR, abs/2401.10774, 2024. doi: 10.48550/ARXIV. 2401.10774. URL https://doi.org/10.48550/ arXiv.2401.10774.
8. Chen, Z., Qu, Z., Liu, L., Ding, Y., and Xie, Y. Efficient tensor core-based GPU kernels for structured sparsity under reduced precision. In de Supinski, B. R., Hall, M. W., and Gamblin, T. (eds.), International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2021, St… [find](https://scholar.google.com/scholar?q=Chen%2C+Z%2E%2C+Qu%2C+Z%2E%2C+Liu%2C+L%2E%2C+Ding%2C+Y%2E%2C+and+Xie%2C+Y%2E+Efficient+tensor+core%2Dbased+GPU+kernels+for+structured+sparsity+under+reduced+precision%2E+In+de+Supinski%2C+B%2E+R%2E%2C+Hall%2C+M%2E+W%2E%2C+and+Gamblin%2C+T%2E+%28eds%2E%29%2C+In)
9. ACM, 2021. doi: 10.1145/3458817.3476182. URL https://doi.org/10.1145/3458817.3476182. — [doi:10.1145/3458817.3476182](https://doi.org/10.1145/3458817.3476182) [↗](https://doi.org/10.1145/3458817.3476182)
10. Chen, Z., May, A., Svirschevski, R., Huang, Y., Ryabinin, M., Jia, Z., and Chen, B. Sequoia: Scalable, robust, and hardware-aware speculative decoding. CoRR, abs/2402.12374, 2024. doi: 10.48550/ARXIV. 2402.12374. URL https://doi.org/10.48550/ arXiv.2402.12374. [find](https://scholar.google.com/scholar?q=Chen%2C+Z%2E%2C+May%2C+A%2E%2C+Svirschevski%2C+R%2E%2C+Huang%2C+Y%2E%2C+Ryabinin%2C+M%2E%2C+Jia%2C+Z%2E%2C+and+Chen%2C+B%2E+Sequoia%3A+Scalable%2C+robust%2C+and+hardware%2Daware+speculative+decoding%2E+CoRR%2C+abs%2F2402%2E12374%2C+2024%2E+doi%3A+10%2E48550%2FARXIV%2E)
11. Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL [https://lmsys.org/blog/2023-03-30-vicuna/](https://lmsys.org/blog/2023-03-30-vicuna/). [↗](https://lmsys.org/blog/2023-03-30-vicuna/)
12. Colfax. Cutlass tutorial: Mastering the nvidia tensor memory accelerator (tma), 2024. URL [https://research.colfax-intl.com/tutorial-hopper-tma/](https://research.colfax-intl.com/tutorial-hopper-tma/). [↗](https://research.colfax-intl.com/tutorial-hopper-tma/)
13. Dao, T. [Flashattention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). CoRR, abs/2307.08691, 2023. doi: 10.48550/ARXIV.2307.08691. URL https://doi.org/10.48550/arXiv.2307.08691. [↗](https://doi.org/10.48550/arXiv.2307.08691)
14. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Infor… [find](https://scholar.google.com/scholar?q=Dao%2C+T%2E%2C+Fu%2C+D%2E+Y%2E%2C+Ermon%2C+S%2E%2C+Rudra%2C+A%2E%2C+and+R%C3%A9%2C+C%2E+Flashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2E+In+Koyejo%2C+S%2E%2C+Mohamed%2C+S%2E%2C+Agarwal%2C+A%2E%2C+Belgrave%2C+D%2E%2C+Cho%2C+K%2E%2C+and+Oh)
15. Dao, T., Haziza, D., Massa, F., and Sizov, G. Flash-decoding for long-context inference, 2023. [find](https://scholar.google.com/scholar?q=Dao%2C+T%2E%2C+Haziza%2C+D%2E%2C+Massa%2C+F%2E%2C+and+Sizov%2C+G%2E+Flash%2Ddecoding+for+long%2Dcontext+inference%2C+2023%2E)
16. DeepSeek-AI, Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Deng, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Xu, H., Yang, H., Zhang, H., Ding, H., Xin, H., Gao, H., Li, H., Qu, H., Cai, J. L., Liang, J., Guo… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Liu%2C+A%2E%2C+Feng%2C+B%2E%2C+Wang%2C+B%2E%2C+Wang%2C+B%2E%2C+Liu%2C+B%2E%2C+Zhao%2C+C%2E%2C+Deng%2C+C%2E%2C+Ruan%2C+C%2E%2C+Dai%2C+D%2E%2C+Guo%2C+D%2E%2C+Yang%2C+D%2E%2C+Chen%2C+D%2E%2C+Ji%2C+D%2E%2C+Li%2C+E%2E%2C+Lin%2C+F%2E%2C+Luo%2C+F%2E%2C+Hao%2C+G%2E%2C+Chen%2C+G%2E%2C+Li%2C+G%2E%2C+Zhang%2C+H%2E%2C)
17. Gupta, M. Mixed-input matrix multiplication performance optimizations. [https://research.google/blog/mixed-input-matrix-multiplication-performance-optimizations/](https://research.google/blog/mixed-input-matrix-multiplication-performance-optimizations/), January 2024. Google Research Blog, Accessed: 2024-01-26. [↗](https://research.google/blog/mixed-input-matrix-multiplication-performance-optimizations/)
18. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Rozière, B., Biron, B., Tang… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
19. Im, E., Yelick, K. A., and Vuduc, R. W. Sparsity: Optimization framework for sparse matrix kernels. Int. J. High Perform. Comput. Appl., 18(1):135–158, 2004. doi: 10.1177/1094342004041296. URL https://doi.org/10.1177/1094342004041296. — [doi:10.1177/1094342004041296](https://doi.org/10.1177/1094342004041296) [↗](https://doi.org/10.1177/1094342004041296)
20. Gray, A. Getting Started with CUDA Graphs \| NVIDIA Technical Blog. [https://developer.nvidia.com/blog/cuda-graphs/](https://developer.nvidia.com/blog/cuda-graphs/), 2019. [Accessed 19-10-2024]. [↗](https://developer.nvidia.com/blog/cuda-graphs/)
21. Gray, S., Radford, A., and Kingma, D. P. Gpu kernels for block-sparse weights. arXiv preprint arXiv:1711.09224, 3(2):2, 2017. [find](https://scholar.google.com/scholar?q=Gray%2C+S%2E%2C+Radford%2C+A%2E%2C+and+Kingma%2C+D%2E+P%2E+Gpu+kernels+for+block%2Dsparse+weights%2E+arXiv+preprint+arXiv%3A1711%2E09224%2C+3%282%29%3A2%2C+2017%2E)
22. He, H., Guessous, D., Liang, Y., and Dong, J. Flexattention: The flexibility of pytorch with the performance of flashattention, Aug 2024. URL [https://pytorch.org/](https://pytorch.org/) blog/flexattention/. [↗](https://pytorch.org/)
23. Hong, K., Dai, G., Xu, J., Mao, Q., Li, X., Liu, J., chen, k., Dong, Y., and Wang, Y. Flashdecoding++: Faster large language model inference with asynchronization, flat gemm optimization, and heuristics. In Gibbons, P., Pekhimenko, G., and Sa, C. D. (eds.), Proceedings of Machine Learning and System… [find](https://scholar.google.com/scholar?q=Hong%2C+K%2E%2C+Dai%2C+G%2E%2C+Xu%2C+J%2E%2C+Mao%2C+Q%2E%2C+Li%2C+X%2E%2C+Liu%2C+J%2E%2C+chen%2C+k%2E%2C+Dong%2C+Y%2E%2C+and+Wang%2C+Y%2E+Flashdecoding%2B%2B%3A+Faster+large+language+model+inference+with+asynchronization%2C+flat+gemm+optimization%2C+and+heuristi)
24. Juravsky, J., Brown, B. C. A., Ehrlich, R. S., Fu, D. Y., Ré, C., and Mirhoseini, A. Hydragen: High-throughput LLM inference with shared prefixes. CoRR, abs/2402.05099, 2024. doi: 10.48550/ARXIV.2402.05099. URL https://doi.org/10.48550/arXiv.2402.05099. [↗](https://doi.org/10.48550/arXiv.2402.05099)
25. Kamath, A. K., Prabhu, R., Mohan, J., Peter, S., Ramjee, R., and Panwar, A. Pod-attention: Unlocking full prefill-decode overlap for faster llm inference. CoRR, abs/2410.18038, 2024. URL [https://arxiv.org/](https://arxiv.org/) abs/2410.18038. [↗](https://arxiv.org/)
26. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Flinn, J., Seltzer, M. I., Druschel, P., Kaufmann, A., and Mace, J. (eds.), Proceedings of the 29th Symposium on…
27. ACM, 2023. doi: 10. 1145/3600006.3613165. URL https://doi.org/ 10.1145/3600006.3613165. — [doi:10.1145/3600006.3613165](https://doi.org/10.1145/3600006.3613165) [find](https://scholar.google.com/scholar?q=ACM%2C+2023%2E+doi%3A+10%2E+1145%2F3600006%2E3613165%2E+URL+https%3A%2F%2Fdoi%2Eorg%2F+10%2E1145%2F3600006%2E3613165%2E+%E2%80%94+%5Bdoi%3A10%2E1145%2F3600006%2E3613165%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E1145%2F3600006%2E3613165%29)
28. Lai, R., Shao, J., Feng, S., Lyubomirsky, S. S., Hou, B., Lin, W., Ye, Z., Jin, H., Jin, Y., Liu, J., Jin, L., Cai, Y., Jiang, Z., Wu, Y., Park, S., Srivastava, P., Roesch, J. G., Mowry, T. C., and Chen, T. Relax: Composable abstractions for end-to-end dynamic machine learning. CoRR, abs/2311.02103,… [find](https://scholar.google.com/scholar?q=Lai%2C+R%2E%2C+Shao%2C+J%2E%2C+Feng%2C+S%2E%2C+Lyubomirsky%2C+S%2E+S%2E%2C+Hou%2C+B%2E%2C+Lin%2C+W%2E%2C+Ye%2C+Z%2E%2C+Jin%2C+H%2E%2C+Jin%2C+Y%2E%2C+Liu%2C+J%2E%2C+Jin%2C+L%2E%2C+Cai%2C+Y%2E%2C+Jiang%2C+Z%2E%2C+Wu%2C+Y%2E%2C+Park%2C+S%2E%2C+Srivastava%2C+P%2E%2C+Roesch%2C+J%2E+G%2E%2C+Mowry%2C+T%2E+C%2E%2C+and+Ch)
29. Li, S., Osawa, K., and Hoefler, T. Efficient quantized sparse matrix operations on tensor cores. In Wolf, F., Shende, S., Culhane, C., Alam, S. R., and Jagode, H. (eds.), SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, Dallas, TX, USA, November 13-18,… [find](https://scholar.google.com/scholar?q=Li%2C+S%2E%2C+Osawa%2C+K%2E%2C+and+Hoefler%2C+T%2E+Efficient+quantized+sparse+matrix+operations+on+tensor+cores%2E+In+Wolf%2C+F%2E%2C+Shende%2C+S%2E%2C+Culhane%2C+C%2E%2C+Alam%2C+S%2E+R%2E%2C+and+Jagode%2C+H%2E+%28eds%2E%29%2C+SC22%3A+International+Conferenc)
30. IEEE, 2022. doi: 10.1109/SC41404.2022.00042. URL https://doi.org/10.1109/SC41404.2022.00042. [↗](https://doi.org/10.1109/SC41404.2022.00042)
31. Lin, C., Han, Z., Zhang, C., Yang, Y., Yang, F., Chen, C., and Qiu, L. Parrot: Efficient serving of LLM-based applications with semantic variable. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pp. 929–945, Santa Clara, CA, July 2024. USENIX Association. ISBN 978-… [find](https://scholar.google.com/scholar?q=Lin%2C+C%2E%2C+Han%2C+Z%2E%2C+Zhang%2C+C%2E%2C+Yang%2C+Y%2E%2C+Yang%2C+F%2E%2C+Chen%2C+C%2E%2C+and+Qiu%2C+L%2E+Parrot%3A+Efficient+serving+of+LLM%2Dbased+applications+with+semantic+variable%2E+In+18th+USENIX+Symposium+on+Operating+Systems+Design+)
32. Liu, H. and Abbeel, P. Blockwise parallel transformers for large context models. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, N… [find](https://scholar.google.com/scholar?q=Liu%2C+H%2E+and+Abbeel%2C+P%2E+Blockwise+parallel+transformers+for+large+context+models%2E+In+Oh%2C+A%2E%2C+Naumann%2C+T%2E%2C+Globerson%2C+A%2E%2C+Saenko%2C+K%2E%2C+Hardt%2C+M%2E%2C+and+Levine%2C+S%2E+%28eds%2E%29%2C+Advances+in+Neural+Information+Pro)
33. Liu, H., Zaharia, M., and Abbeel, P. [Ring attention with blockwise transformers for near-infinite context](https://md.research.elamur.ai/free/en/papers/liu-2023-ring-attention-blockwise-transformers-near/). CoRR, abs/2310.01889, 2023. doi: 10.48550/ARXIV. 2310.01889. URL https://doi.org/10.48550/ arXiv.2310.01889.
34. Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Zhang, Z., Wong, R. Y. Y., Zhu, A., Yang, L., Shi, X., Shi, C., Chen, Z., Arfeen, D., Abhyankar, R., and Jia, Z. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Gupta, R., Abu-Ghazal… [find](https://scholar.google.com/scholar?q=Miao%2C+X%2E%2C+Oliaro%2C+G%2E%2C+Zhang%2C+Z%2E%2C+Cheng%2C+X%2E%2C+Wang%2C+Z%2E%2C+Zhang%2C+Z%2E%2C+Wong%2C+R%2E+Y%2E+Y%2E%2C+Zhu%2C+A%2E%2C+Yang%2C+L%2E%2C+Shi%2C+X%2E%2C+Shi%2C+C%2E%2C+Chen%2C+Z%2E%2C+Arfeen%2C+D%2E%2C+Abhyankar%2C+R%2E%2C+and+Jia%2C+Z%2E+Specinfer%3A+Accelerating+large+lan)
35. ACM, 2024. [doi: 10.1145/3620666.3651335](https://md.research.elamur.ai/free/en/papers/miao-2023-specinfer-accelerating-generative-large-language/). URL https://doi.org/10.1145/3620666.3651335. [↗](https://doi.org/10.1145/3620666.3651335)
36. Micikevicius, P., Stosic, D., Burgess, N., Cornea, M., Dubey, P., Grisenthwaite, R., Ha, S., Heinecke, A., Judd, P., Kamalu, J., Mellempudi, N., Oberman, S. F., Shoeybi, M., Siu, M. Y., and Wu, H. FP8 formats for deep learning. CoRR, abs/2209.05433, 2022. doi: 10.48550/ARXIV. 2209.05433. URL https:/… [find](https://scholar.google.com/scholar?q=Micikevicius%2C+P%2E%2C+Stosic%2C+D%2E%2C+Burgess%2C+N%2E%2C+Cornea%2C+M%2E%2C+Dubey%2C+P%2E%2C+Grisenthwaite%2C+R%2E%2C+Ha%2C+S%2E%2C+Heinecke%2C+A%2E%2C+Judd%2C+P%2E%2C+Kamalu%2C+J%2E%2C+Mellempudi%2C+N%2E%2C+Oberman%2C+S%2E+F%2E%2C+Shoeybi%2C+M%2E%2C+Siu%2C+M%2E+Y%2E%2C+and+Wu%2C+H%2E+FP8)
37. Milakov, M. and Gimelshein, N. [Online normalizer calculation for softmax](https://md.research.elamur.ai/free/en/papers/milakov-2018-online-normalizer-calculation-softmax/). CoRR, abs/1805.02867, 2018. URL http://arxiv.org/abs/1805.02867. [↗](http://arxiv.org/abs/1805.02867)
38. Mirchandaney, R., Saltz, J. H., Smith, R. M., Nicol, D. M., and Crowley, K. Principles of runtime support for parallel processors. In Lenfant, J. (ed.), Proceedings of the 2nd international conference on Supercomputing, ICS 1988, Saint Malo, France, July 4-8, 1988, pp. 140–152. [find](https://scholar.google.com/scholar?q=Mirchandaney%2C+R%2E%2C+Saltz%2C+J%2E+H%2E%2C+Smith%2C+R%2E+M%2E%2C+Nicol%2C+D%2E+M%2E%2C+and+Crowley%2C+K%2E+Principles+of+runtime+support+for+parallel+processors%2E+In+Lenfant%2C+J%2E+%28ed%2E%29%2C+Proceedings+of+the+2nd+international+conference)
39. ACM, 1988. doi: 10.1145/55364.55378. URL https://doi.org/10.1145/55364.55378. — [doi:10.1145/55364.55378](https://doi.org/10.1145/55364.55378) [↗](https://doi.org/10.1145/55364.55378)
40. MLC Community. Optimizing and characterizing high-throughput low-latency LLM inference in MLCEngine, Oct 2024. URL [https://blog.mlc.ai/2024/10/10/optimizing-and-characterizing-high-throughput-low-latency-llm-inference](https://blog.mlc.ai/2024/10/10/optimizing-and-characterizing-high-throughput-low-latency-llm-inference). [Online; accessed April 23, 2025]. [↗](https://blog.mlc.ai/2024/10/10/optimizing-and-characterizing-high-throughput-low-latency-llm-inference)
41. Mostafa, H. Sequential aggregation and rematerialization: Distributed full-batch training of graph neural networks on large graphs. In Marculescu, D., Chi, Y., and Wu, C. (eds.), Proceedings of Machine Learning and Systems 2022, MLSys 2022, Santa Clara, CA, USA, August 29 - September 1, 2022. mlsys.… [find](https://scholar.google.com/scholar?q=Mostafa%2C+H%2E+Sequential+aggregation+and+rematerialization%3A+Distributed+full%2Dbatch+training+of+graph+neural+networks+on+large+graphs%2E+In+Marculescu%2C+D%2E%2C+Chi%2C+Y%2E%2C+and+Wu%2C+C%2E+%28eds%2E%29%2C+Proceedings+of+Machin)
42. Narang, S., Undersander, E., and Diamos, G. F. Block-sparse recurrent neural networks. CoRR, abs/1711.02782, 2017. URL http://arxiv.org/abs/1711. 02782. — [arXiv:1711.02782](https://arxiv.org/abs/1711.02782) [↗](http://arxiv.org/abs/1711)
43. Nguyen, V., Carilli, M., Eryilmaz, S. B., Singh, V., Lin, M., Gimelshein, N., Desmaison, A., and Yang, E. Accelerating PyTorch with CUDA Graphs. [https://pytorch.org/blog/accelerating-pytorch-with-cuda-graphs/](https://pytorch.org/blog/accelerating-pytorch-with-cuda-graphs/), 2021. [Accessed 19-10-2024]. [↗](https://pytorch.org/blog/accelerating-pytorch-with-cuda-graphs/)
44. NVIDIA. FasterTransformer. [https://github.com/](https://github.com/) NVIDIA/FasterTransformer, 2021. [↗](https://github.com/)
45. NVIDIA. Nvidia hopper architecture in-depth, 2022. URL [https://developer.nvidia.com/blog/](https://developer.nvidia.com/blog/) nvidia-hopper-architecture-in-depth/. [↗](https://developer.nvidia.com/blog/)
46. NVIDIA. NVIDIA TensorRT-LLM, 2023a. URL [https://docs.nvidia.com/tensorrt-llm/index.html](https://docs.nvidia.com/tensorrt-llm/index.html). [Online; accessed April 23, 2025]. [↗](https://docs.nvidia.com/tensorrt-llm/index.html)
47. NVIDIA. Matrix multiplication background user’s guide, 2023b. URL [https://docs.nvidia.com/](https://docs.nvidia.com/) deeplearning/performance/pdf/Matrix-Multiplication-Background-User-Guide.pdf. [↗](https://docs.nvidia.com/)
48. NVIDIA. Spatial partitioning (also known as warp specialization), 2024a. URL [https://docs.nvidia.com/cuda/cuda-c-programming-guide/](https://docs.nvidia.com/cuda/cuda-c-programming-guide/) index.html\#spatial-partitioning-also-known-as-warp-specialization. [↗](https://docs.nvidia.com/cuda/cuda-c-programming-guide/)
49. NVIDIA. New xqa-kernel provides 2.4x more llama-70b throughput within the same latency budget, 2024b. URL [https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/XQA-kernel.md](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/XQA-kernel.md). [↗](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/XQA-kernel.md)
50. Osama, M., Merrill, D., Cecka, C., Garland, M., and Owens, J. D. Stream-k: Work-centric parallel decomposition for dense matrix-matrix multiplication on the GPU. In Dehnavi, M. M., Kulkarni, M., and Krishnamoorthy, S. (eds.), Proceedings of the 28th ACM SIGPLAN Annual Symposium on Principles and Pra… [find](https://scholar.google.com/scholar?q=Osama%2C+M%2E%2C+Merrill%2C+D%2E%2C+Cecka%2C+C%2E%2C+Garland%2C+M%2E%2C+and+Owens%2C+J%2E+D%2E+Stream%2Dk%3A+Work%2Dcentric+parallel+decomposition+for+dense+matrix%2Dmatrix+multiplication+on+the+GPU%2E+In+Dehnavi%2C+M%2E+M%2E%2C+Kulkarni%2C+M%2E%2C+and+K)
51. ACM, 2023. doi: 10.1145/3572848.3577479. URL https://doi.org/10.1145/3572848.3577479. — [doi:10.1145/3572848.3577479](https://doi.org/10.1145/3572848.3577479) [↗](https://doi.org/10.1145/3572848.3577479)
52. Ozen, G. Nvdsl: Simplifying tensor cores with python-driven mlir metaprogramming. In Efficient Systems for Foundation Models (ES-FoMo) Workshop at ICML 2024, 2024. [find](https://scholar.google.com/scholar?q=Ozen%2C+G%2E+Nvdsl%3A+Simplifying+tensor+cores+with+python%2Ddriven+mlir+metaprogramming%2E+In+Efficient+Systems+for+Foundation+Models+%28ES%2DFoMo%29+Workshop+at+ICML+2024%2C+2024%2E)
53. Ponnusamy, R., Saltz, J. H., and Choudhary, A. N. Runtime compilation techniques for data partitioning and communication schedule reuse. In Borchers, B. and Crawford, D. (eds.), Proceedings Supercomputing ’93, Portland, Oregon, USA, November 15-19, 1993, pp. 361– 370. [find](https://scholar.google.com/scholar?q=Ponnusamy%2C+R%2E%2C+Saltz%2C+J%2E+H%2E%2C+and+Choudhary%2C+A%2E+N%2E+Runtime+compilation+techniques+for+data+partitioning+and+communication+schedule+reuse%2E+In+Borchers%2C+B%2E+and+Crawford%2C+D%2E+%28eds%2E%29%2C+Proceedings+Supercompu)
54. ACM, 1993. doi: 10.1145/169627.169752. URL https://doi.org/10.1145/169627.169752. — [doi:10.1145/169627.169752](https://doi.org/10.1145/169627.169752) [↗](https://doi.org/10.1145/169627.169752)
55. Prabhu, R., Nayak, A., Mohan, J., Ramjee, R., and Panwar, A. vattention: Dynamic memory management for serving llms without pagedattention, 2024. [find](https://scholar.google.com/scholar?q=Prabhu%2C+R%2E%2C+Nayak%2C+A%2E%2C+Mohan%2C+J%2E%2C+Ramjee%2C+R%2E%2C+and+Panwar%2C+A%2E+vattention%3A+Dynamic+memory+management+for+serving+llms+without+pagedattention%2C+2024%2E)
56. Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. [find](https://scholar.google.com/scholar?q=Press%2C+O%2E%2C+Smith%2C+N%2E+A%2E%2C+and+Lewis%2C+M%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2E+In+The+Tenth+International+Conference+on+Learning+Representations%2C+ICLR+)
57. OpenReview.net, 2022. URL [https://openreview.net/forum?id=R8sQPpGCv0](https://openreview.net/forum?id=R8sQPpGCv0). [↗](https://openreview.net/forum?id=R8sQPpGCv0)
58. PyTorch-Labs. attention-gym. [https://github.com/](https://github.com/) pytorch-labs/attention-gym, 2024. [↗](https://github.com/)
59. Rahman, M. K., Sujon, M. H., and Azad, A. Fusedmm: A unified sddmm-spmm kernel for graph embedding and graph neural networks. In 35th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2021, Portland, OR, USA, May 17-21, 2021, pp. 256–266. [find](https://scholar.google.com/scholar?q=Rahman%2C+M%2E+K%2E%2C+Sujon%2C+M%2E+H%2E%2C+and+Azad%2C+A%2E+Fusedmm%3A+A+unified+sddmm%2Dspmm+kernel+for+graph+embedding+and+graph+neural+networks%2E+In+35th+IEEE+International+Parallel+and+Distributed+Processing+Symposium%2C+)
60. IEEE, 2021. doi: 10.1109/IPDPS49936. 2021.00034. URL https://doi.org/10.1109/ IPDPS49936.2021.00034. — [doi:10.1109/ipdps49936](https://doi.org/10.1109/ipdps49936) [find](https://scholar.google.com/scholar?q=IEEE%2C+2021%2E+doi%3A+10%2E1109%2FIPDPS49936%2E+2021%2E00034%2E+URL+https%3A%2F%2Fdoi%2Eorg%2F10%2E1109%2F+IPDPS49936%2E2021%2E00034%2E+%E2%80%94+%5Bdoi%3A10%2E1109%2Fipdps49936%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E1109%2Fipdps49936%29)
61. Ramapuram, J., Danieli, F., Dhekane, E. G., Weers, F., Busbridge, D., Ablin, P., Likhomanenko, T., Digani, J., Gu, Z., Shidani, A., and Webb, R. Theory, analysis, and best practices for sigmoid self-attention. CoRR, abs/2409.04431, 2024. doi: 10.48550/ARXIV. 2409.04431. URL https://doi.org/10.48550/… [find](https://scholar.google.com/scholar?q=Ramapuram%2C+J%2E%2C+Danieli%2C+F%2E%2C+Dhekane%2C+E%2E+G%2E%2C+Weers%2C+F%2E%2C+Busbridge%2C+D%2E%2C+Ablin%2C+P%2E%2C+Likhomanenko%2C+T%2E%2C+Digani%2C+J%2E%2C+Gu%2C+Z%2E%2C+Shidani%2C+A%2E%2C+and+Webb%2C+R%2E+Theory%2C+analysis%2C+and+best+practices+for+sigmoid+self%2Da)
62. Rivière, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C. L., Jerome, S., Tsitsulin, A., Vieillard, N., Stanczyk, P., Girgin, S., Momchev, N., Hoffman,… [find](https://scholar.google.com/scholar?q=Rivi%C3%A8re%2C+M%2E%2C+Pathak%2C+S%2E%2C+Sessa%2C+P%2E+G%2E%2C+Hardin%2C+C%2E%2C+Bhupatiraju%2C+S%2E%2C+Hussenot%2C+L%2E%2C+Mesnard%2C+T%2E%2C+Shahriari%2C+B%2E%2C+Ram%C3%A9%2C+A%2E%2C+Ferret%2C+J%2E%2C+Liu%2C+P%2E%2C+Tafti%2C+P%2E%2C+Friesen%2C+A%2E%2C+Casbon%2C+M%2E%2C+Ramos%2C+S%2E%2C+Kumar%2C+R%2E%2C+L)
63. Saltz, J. H. and Mirchandaney, R. The preprocessed doacross loop. In Proceedings of the International Conference on Parallel Processing, ICPP ’91, Austin, Texas, USA, August 1991. Volume II: Software, pp. 174–179. [find](https://scholar.google.com/scholar?q=Saltz%2C+J%2E+H%2E+and+Mirchandaney%2C+R%2E+The+preprocessed+doacross+loop%2E+In+Proceedings+of+the+International+Conference+on+Parallel+Processing%2C+ICPP+%E2%80%9991%2C+Austin%2C+Texas%2C+USA%2C+August+1991%2E+Volume+II%3A+Software%2C)
64. Sanovar, R., Bharadwaj, S., Amant, R. S., Rühle, V., and Rajmohan, S. Lean attention: Hardware-aware scalable attention mechanism for the decode-phase of transformers. CoRR, abs/2405.10480, 2024. doi: 10.48550/ARXIV. 2405.10480. URL https://doi.org/10.48550/ arXiv.2405.10480. [find](https://scholar.google.com/scholar?q=Sanovar%2C+R%2E%2C+Bharadwaj%2C+S%2E%2C+Amant%2C+R%2E+S%2E%2C+R%C3%BChle%2C+V%2E%2C+and+Rajmohan%2C+S%2E+Lean+attention%3A+Hardware%2Daware+scalable+attention+mechanism+for+the+decode%2Dphase+of+transformers%2E+CoRR%2C+abs%2F2405%2E10480%2C+2024%2E+doi%3A)
65. Setaluri, R., Aanjaneya, M., Bauer, S., and Sifakis, E. Spgrid: a sparse paged grid structure applied to adaptive smoke simulation. ACM Trans. Graph., 33(6):205:1–205:12, 2014. doi: 10.1145/2661229. 2661269. URL https://doi.org/10.1145/ 2661229.2661269. — [doi:10.1145/2661229](https://doi.org/10.1145/2661229) [find](https://scholar.google.com/scholar?q=Setaluri%2C+R%2E%2C+Aanjaneya%2C+M%2E%2C+Bauer%2C+S%2E%2C+and+Sifakis%2C+E%2E+Spgrid%3A+a+sparse+paged+grid+structure+applied+to+adaptive+smoke+simulation%2E+ACM+Trans%2E+Graph%2E%2C+33%286%29%3A205%3A1%E2%80%93205%3A12%2C+2014%2E+doi%3A+10%2E1145%2F2661229%2E+2)
66. Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., and Dao, T. [Flashattention-3: Fast and accurate attention with asynchrony and low-precision](https://md.research.elamur.ai/free/en/papers/shah-2024-flashattention-3-fast-accurate-attention/). CoRR, abs/2407.08608, 2024. doi: 10.48550/ARXIV. 2407.08608. URL https://doi.org/10.48550/ arXiv.2407.08608.
67. Shazeer, N. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150. [↗](http://arxiv.org/abs/1911.02150)
68. Spector, B., Singhal, A., Arora, S., and Re, C. ThunderKittens: A Simple Embedded DSL for AI kernels, May 2024. URL [https://hazyresearch.stanford.edu/](https://hazyresearch.stanford.edu/) blog/2024-05-12-quick-tk. [↗](https://hazyresearch.stanford.edu/)
69. Su, J., Ahmed, M. H. M., Lu, Y., Pan, S., Bo, W., and Liu, Y. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Neurocomputing, 568:127063, 2024. doi: 10. 1016/J.NEUCOM.2023.127063. URL https://doi.org/10.1016/j.neucom.2023.127063. [↗](https://doi.org/10.1016/j.neucom.2023.127063)
70. Tang, J., Zhao, Y., Zhu, K., Xiao, G., Kasikci, B., and Han, S. [QUEST: query-aware sparsity for efficient long-context LLM inference](https://md.research.elamur.ai/free/en/papers/tang-2024-quest-query-aware-sparsity-efficient/). In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024.
71. OpenReview.net, 2024. URL [https://openreview.net/forum?id=KzACYw0MTV](https://openreview.net/forum?id=KzACYw0MTV). [↗](https://openreview.net/forum?id=KzACYw0MTV)
72. Tensorflow Developers. Ragged tensors \| tensorflow core. [https://www.tensorflow.org/guide/](https://www.tensorflow.org/guide/) ragged_tensor, 2018. [↗](https://www.tensorflow.org/guide/)
73. Thakkar, V., Ramani, P., Cecka, C., Shivam, A., Lu, H., Yan, E., Kosaian, J., Hoemmen, M., Wu, H., Kerr, A., Nicely, M., Merrill, D., Blasig, D., Qiao, F., Majcher, P., Springer, P., Hohnerbach, M., Wang, J., and Gupta, M. CUTLASS, January 2023. URL [https://github.com/NVIDIA/cutlass](https://github.com/NVIDIA/cutlass). [↗](https://github.com/NVIDIA/cutlass)
74. Tillet, P., Kung, H. T., and Cox, D. [Triton: an intermediate language and compiler for tiled neural network computations](https://md.research.elamur.ai/free/en/papers/tillet-2019-triton-intermediate-language-compiler-tiled/). In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, pp. 10–19, New York, NY, USA, 2019. Association for Computi…
75. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural Information Processing Sys…
76. Wang, Y., Feng, B., Wang, Z., Huang, G., and Ding, Y. TC-GNN: bridging sparse GNN computation and dense tensor cores on gpus. In Lawall, J. and Williams, D. (eds.), 2023 USENIX Annual Technical Conference, USENIX ATC 2023, Boston, MA, USA, July 10-12, 2023, pp. 149–164. [find](https://scholar.google.com/scholar?q=Wang%2C+Y%2E%2C+Feng%2C+B%2E%2C+Wang%2C+Z%2E%2C+Huang%2C+G%2E%2C+and+Ding%2C+Y%2E+TC%2DGNN%3A+bridging+sparse+GNN+computation+and+dense+tensor+cores+on+gpus%2E+In+Lawall%2C+J%2E+and+Williams%2C+D%2E+%28eds%2E%29%2C+2023+USENIX+Annual+Technical+Confer)
77. USENIX Association, 2023. URL [https://www.usenix.org/conference/](https://www.usenix.org/conference/) atc23/presentation/wang-yuke. [↗](https://www.usenix.org/conference/)
78. Wu, M., Cheng, X., Padon, O., and Jia, Z. A multi-level superoptimizer for tensor programs. CoRR, abs/2405.05751, 2024. doi: 10.48550/ARXIV.2405.05751. URL https://doi.org/10.48550/arXiv.2405.05751. xAI. Open Release of Grok-1. https://x.ai/blog/ grok-os, 2023. [Accessed 24-06-2024]. [↗](https://doi.org/10.48550/arXiv.2405.05751)
79. Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. arXiv, 2023. [find](https://scholar.google.com/scholar?q=Xiao%2C+G%2E%2C+Tian%2C+Y%2E%2C+Chen%2C+B%2E%2C+Han%2C+S%2E%2C+and+Lewis%2C+M%2E+Efficient+streaming+language+models+with+attention+sinks%2E+arXiv%2C+2023%2E)
80. Yang, S., Wang, B., Shen, Y., Panda, R., and Kim, Y. Gated linear attention transformers with hardware-efficient training, 2024. URL https://arxiv.org/abs/2312. 06635. [↗](https://arxiv.org/abs/2312)
81. Ye, L., Tao, Z., Huang, Y., and Li, Y. Chunkattention: Efficient self-attention with prefix-aware KV cache and two-phase partition. In Ku, L., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 20… [find](https://scholar.google.com/scholar?q=Ye%2C+L%2E%2C+Tao%2C+Z%2E%2C+Huang%2C+Y%2E%2C+and+Li%2C+Y%2E+Chunkattention%3A+Efficient+self%2Dattention+with+prefix%2Daware+KV+cache+and+two%2Dphase+partition%2E+In+Ku%2C+L%2E%2C+Martins%2C+A%2E%2C+and+Srikumar%2C+V%2E+%28eds%2E%29%2C+Proceedings+of+the+)
82. Ye, Z., Lai, R., Shao, J., Chen, T., and Ceze, L. Sparsetir: Composable abstractions for sparse compilation in deep learning. In Aamodt, T. M., Jerger, N. D. E., and Swift, M. M. (eds.), Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Opera… [find](https://scholar.google.com/scholar?q=Ye%2C+Z%2E%2C+Lai%2C+R%2E%2C+Shao%2C+J%2E%2C+Chen%2C+T%2E%2C+and+Ceze%2C+L%2E+Sparsetir%3A+Composable+abstractions+for+sparse+compilation+in+deep+learning%2E+In+Aamodt%2C+T%2E+M%2E%2C+Jerger%2C+N%2E+D%2E+E%2E%2C+and+Swift%2C+M%2E+M%2E+%28eds%2E%29%2C+Proceedings+o)
83. ACM, 2023. doi: 10. 1145/3582016.3582047. URL https://doi.org/ 10.1145/3582016.3582047. — [doi:10.1145/3582016.3582047](https://doi.org/10.1145/3582016.3582047) [find](https://scholar.google.com/scholar?q=ACM%2C+2023%2E+doi%3A+10%2E+1145%2F3582016%2E3582047%2E+URL+https%3A%2F%2Fdoi%2Eorg%2F+10%2E1145%2F3582016%2E3582047%2E+%E2%80%94+%5Bdoi%3A10%2E1145%2F3582016%2E3582047%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E1145%2F3582016%2E3582047%29)
84. Yu, G., Jeong, J. S., Kim, G., Kim, S., and Chun, B. [Orca: A distributed serving system for transformer-based generative models](https://md.research.elamur.ai/free/en/papers/yu-2022-orca-distributed-serving-system-transformer/). In Aguilera, M. K. and Weatherspoon, H. (eds.), 16th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2022, Carlsbad, CA, USA, July 11-13, 2022, pp. 5…
85. USENIX Association, 2022. URL [https://www.usenix.org/](https://www.usenix.org/) conference/osdi22/presentation/yu. [↗](https://www.usenix.org/)
86. Zhai, Y., Jiang, C., Wang, L., Jia, X., Zhang, S., Chen, Z., Liu, X., and Zhu, Y. Bytetransformer: A high-performance transformer boosted for variable-length inputs. In IEEE International Parallel and Distributed Processing Symposium, IPDPS 2023, St. Petersburg, FL, USA, May 15-19, 2023, pp. 344–355… [find](https://scholar.google.com/scholar?q=Zhai%2C+Y%2E%2C+Jiang%2C+C%2E%2C+Wang%2C+L%2E%2C+Jia%2C+X%2E%2C+Zhang%2C+S%2E%2C+Chen%2C+Z%2E%2C+Liu%2C+X%2E%2C+and+Zhu%2C+Y%2E+Bytetransformer%3A+A+high%2Dperformance+transformer+boosted+for+variable%2Dlength+inputs%2E+In+IEEE+International+Parallel+and)
87. IEEE, 2023. doi: 10.1109/IPDPS54959.2023.00042. URL https://doi.org/10.1109/IPDPS54959.2023.00042. — [doi:10.1109/ipdps54959.2023.00042](https://doi.org/10.1109/ipdps54959.2023.00042) [↗](https://doi.org/10.1109/IPDPS54959.2023.00042)
88. Zhang, H., Yu, Z., Dai, G., Huang, G., Ding, Y., Xie, Y., and Wang, Y. Understanding GNN computational graph: A coordinated computation, io, and memory perspective. In Marculescu, D., Chi, Y., and Wu, C. (eds.), Proceedings of Machine Learning and Systems 2022, MLSys 2022, Santa Clara, CA, USA, Augu… [find](https://scholar.google.com/scholar?q=Zhang%2C+H%2E%2C+Yu%2C+Z%2E%2C+Dai%2C+G%2E%2C+Huang%2C+G%2E%2C+Ding%2C+Y%2E%2C+Xie%2C+Y%2E%2C+and+Wang%2C+Y%2E+Understanding+GNN+computational+graph%3A+A+coordinated+computation%2C+io%2C+and+memory+perspective%2E+In+Marculescu%2C+D%2E%2C+Chi%2C+Y%2E%2C+and+Wu%2C)
89. Zheng, L., Chiang, W., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in…
90. Zheng, L., Yin, L., Xie, Z., Huang, J., Sun, C., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C. W., and Sheng, Y. Efficiently programming large language models using sglang. CoRR, abs/2312.[07104, 2023b](https://md.research.elamur.ai/free/en/papers/zheng-2023-sglang-efficient-execution-structured-language/). doi: 10.48550/ARXIV.2312.07104. URL https://doi.org/10.48550/arXiv.… [↗](https://doi.org/10.48550/arXiv.…)
91. Zhu, K., Zhao, Y., Zhao, L., Zuo, G., Gu, Y., Xie, D., Gao, Y., Xu, Q., Tang, T., Ye, Z., Kamahori, K., Lin, C., Wang, S., Krishnamurthy, A., and Kasikci, B. [Nanoflow: Towards optimal large language model serving throughput](https://md.research.elamur.ai/free/en/papers/zhu-2024-nanoflow-optimal-large-language-model/). CoRR, abs/2408.12757, 2024a. doi: 10.48550/ARXIV.2408.12757. URL https://do…
92. Zhu, L., Wang, X., Zhang, W., and Lau, R. W. H. Relayattention for efficient large language model serving with long system prompts. In Ku, L., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 20… [find](https://scholar.google.com/scholar?q=Zhu%2C+L%2E%2C+Wang%2C+X%2E%2C+Zhang%2C+W%2E%2C+and+Lau%2C+R%2E+W%2E+H%2E+Relayattention+for+efficient+large+language+model+serving+with+long+system+prompts%2E+In+Ku%2C+L%2E%2C+Martins%2C+A%2E%2C+and+Srikumar%2C+V%2E+%28eds%2E%29%2C+Proceedings+of+the+)

*Updated 2 November 2026*