---
title: "LLM Inference Unveiled: Survey and Roofline Model Insights"
description: "The authors build their review around the Roofline model. 254 references, cited by 78 papers here."
authors: ["Zhihang Yuan", "Yuzhang Shang", "Yang Zhou", "Zhen Dong", "Zhe Zhou", "Chenhao Xue", "Bingzhe Wu", "Zhikai Li", "Qingyi Gu", "Yong Jae Lee"]
year: 2024
venue: "arXiv 2024"
source: "https://arxiv.org/abs/2402.16363"
arxiv: "2402.16363"
licence: "arXiv — non-exclusive distribution license (arXiv OAI)"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yuan-2024-llm-inference-unveiled-survey-roofline/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yuan-2024-llm-inference-unveiled-survey-roofline/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / LLM Inference Unveiled: Survey and Roofline Model Insights

# LLM Inference Unveiled: Survey and Roofline Model Insights

**Authors:** [Zhihang Yuan](https://md.research.elamur.ai/free/en/authors/zhihang-yuan/), [Yuzhang Shang](https://md.research.elamur.ai/free/en/authors/yuzhang-shang/), [Yang Zhou](https://md.research.elamur.ai/free/en/authors/yang-zhou/), [Zhen Dong](https://md.research.elamur.ai/free/en/authors/zhen-dong/), Zhe Zhou, Chenhao Xue, [Bingzhe Wu](https://md.research.elamur.ai/free/en/authors/bingzhe-wu/), Zhikai Li, Qingyi Gu and [Yong Jae Lee](https://md.research.elamur.ai/free/en/authors/yong-jae-lee/) · **Year:** 2024 · **Venue:** arXiv 2024 · **Licence:** [arXiv — non-exclusive distribution license (arXiv OAI)](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.16363) · [PDF](https://arxiv.org/pdf/2402.16363) · [arXiv 2402.16363](https://arxiv.org/abs/2402.16363) · [OpenAlex](https://openalex.org/W4392222465)

**Citations:** 13 citations (OpenAlex, 2026-10-14) · [78](https://md.research.elamur.ai/free/en/papers/yuan-2024-llm-inference-unveiled-survey-roofline/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This survey targets a practical gap in research on efficient LLM inference. Many acceleration techniques exist, but there has been no unified way to reason about why they help on real hardware. The authors build their review around the Roofline model. It relates a workload's demands to a device's compute capacity and memory bandwidth, which shows whether a step is limited by computation or by data movement.

They implement this analysis in an open-source tool, LLM-Viewer. It takes a model's layer configuration, inference settings and hardware specs, then reports the bottleneck, the memory footprint and the attainable performance. Using this lens, they classify the literature into four groups:

- parameter-level compression
- fast decoding algorithms
- compiler/system optimizations
- hardware-level designs

For each group they explain the effect on memory traffic versus arithmetic. The work also explains why LLM decoding tends to be memory-bound and how to pick suitable hardware. This matters for teams with limited compute, who need to know which optimizations will actually pay off before investing in them.

## Key points
- **LLM-Viewer inputs and outputs.** The tool takes an LLM (e.g., LLaMA, Vicuna) and a hardware target (e.g., Nvidia A100, Intel 13900K). It also takes these settings:
  - batch size
  - sequence length
  - quantization bitwidth
  - FlashAttention on/off
  - inference stage
  - decoding method

  It outputs peak performance, the bottleneck, the memory footprint and batch-size/performance curves.
- **Two inference stages.** The prefill stage processes the whole prompt and builds a per-layer KV cache. The decode stage generates one token at a time, reusing and extending that cache.
- **Taxonomy of techniques.** The survey groups methods into four areas:
  - Model compression: quantization (QAT, PTQ, Q-PEFT), pruning, knowledge distillation, factorization.
  - Fast decoding: MoE, contextual sparsity, early exiting, speculative decoding, parallel decoding.
  - System level: operator fusion, memory management, workload offloading, parallel serving.
  - Hardware level: spatial architectures, PIM/NMC, new data formats, new processing elements.
- **Discussion topics.** The closing discussion covers safety and reliability, large multimodal models and long sequences. Long-context methods reviewed include StreamingLLM, H2O, Transformer-XL, Compressive Transformer, ALiBi, NTK-aware RoPE, YaRN and Giraffe.
- **Ongoing maintenance.** The paper (arXiv 2402.16363) is announced as a maintained, updated project.

## Interesting ideas
- **Organize a survey around one analytical lens.** Each method is judged by how it changes memory access versus computation, rather than only listed by topic. The same approach could structure literature reviews in other areas, such as data pipelines or treebank tooling.
- **Model prefill and decode separately.** The two stages have very different bottlenecks, so analysing them separately avoids misleading averages. This helps when estimating serving costs for chatbots or batch annotation jobs.
- **Use a config-driven "what-if" calculator.** Describing the model, settings and hardware declaratively, then getting bottleneck and memory estimates, lets you compare deployment options before buying or renting GPUs. This is valuable for small teams on tight budgets.
- **Keep the first tokens when truncating context.** StreamingLLM's observation is that the opening tokens act as anchors for softmax attention. A sink-plus-sliding-window mask can therefore extend usable context without fine-tuning, a cheap trick for long-document processing.
- **Rescale position encodings by frequency.** Giraffe protects the well-trained high-frequency components with a power-law scaling scheme. More generally, how well each part of a learned representation was trained should guide how aggressively it is extrapolated.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 598 works · 191,226 citations
- [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 209 works · 244,782 citations
- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 315 works · 43,267 citations
- [Tencent](https://md.research.elamur.ai/free/en/companies/tencent/) — Company, Shenzhen, China · 163 works · 21,032 citations
- [Institute of Automation](https://md.research.elamur.ai/free/en/labs/institute-of-automation/) — Research institute, Beijing, China · 87 works · 14,883 citations
- [University of Wisconsin–Madison](https://md.research.elamur.ai/free/en/universities/university-of-wisconsin-madison/) — University, Madison, United States · 54 works · 49,089 citations
- [Illinois Institute of Technology](https://md.research.elamur.ai/free/en/universities/illinois-institute-of-technology/) — University, Chicago, United States · 18 works · 2,097 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🔌 Hardware and Architecture](https://md.research.elamur.ai/free/en/shelves/computer-science/hardware-and-architecture/) · 167 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Zhe Zhou, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu and Yong Jae Lee. 2024. LLM Inference Unveiled: Survey and Roofline Model Insights. arXiv 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{yuan2024llm,
  title = {{LLM Inference Unveiled: Survey and Roofline Model Insights}},
  author = {Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Zhe Zhou, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu and Yong Jae Lee},
  year = {2024},
  eprint = {2402.16363},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.16363}
}
```

- [← previous LLaMA-Omni: Seamless Speech Interaction with Large Language Models](https://md.research.elamur.ai/free/en/papers/fang-2024-llama-omni-seamless-speech-interaction/)
- [next → PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference](https://md.research.elamur.ai/free/en/papers/yang-2024-pyramidinfer-pyramid-kv-cache-compression/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

254 references, 77 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abts, D., Kimmell, G., Ling, A., Kim, J., Boyd, M., Bitar, A., Parmar, S., Ahmed, I., DiCecco, R., Han, D., et al. (2022). A software-defined tensor streaming multiprocessor for large-scale machine learning. In Proceedings of the 49th Annual International Symposium on Computer Architecture, pages 56… [find](https://scholar.google.com/scholar?q=Abts%2C+D%2E%2C+Kimmell%2C+G%2E%2C+Ling%2C+A%2E%2C+Kim%2C+J%2E%2C+Boyd%2C+M%2E%2C+Bitar%2C+A%2E%2C+Parmar%2C+S%2E%2C+Ahmed%2C+I%2E%2C+DiCecco%2C+R%2E%2C+Han%2C+D%2E%2C+et+al%2E+%282022%29%2E+A+software%2Ddefined+tensor+streaming+multiprocessor+for+large%2Dscale+machine+le)
2. Agarwal, R., Vieillard, N., Stanczyk, P., Ramos, S., Geist, M., and Bachem, O. (2023). [Gkd: Generalized knowledge distillation for auto-regressive sequence models](https://md.research.elamur.ai/free/en/papers/agarwal-2023-policy-distillation-language-models-learning/). arXiv preprint arXiv:2306.13649. 12
3. Agrawal, A., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., and Ramjee, R. (2023). [Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills](https://md.research.elamur.ai/free/en/papers/agrawal-2023-sarathi-efficient-llm-inference-piggybacking/). arXiv preprint arXiv:2308.16369. 22
4. Ainslie, J., Ontanon, S., Alberti, C., Cvicek, V., Fisher, Z., Pham, P., Ravula, A., Sanghai, S., Wang, Q., and Yang, L. (2020). [Etc: Encoding long and structured inputs in transformers](https://md.research.elamur.ai/free/en/papers/ainslie-2020-etc-encoding-long-structured-inputs/). arXiv preprint arXiv:2004.08483. 27
5. Alizadeh, K., Mirzadeh, I., Belenko, D., Khatamifard, K., Cho, M., Del Mundo, C. C., Rastegari, M., and Farajtabar, M. (2023). [Llm in a flash: Efficient large language model inference with limited memory](https://md.research.elamur.ai/free/en/papers/alizadeh-2024-llm-flash-efficient-large-language/). arXiv preprint arXiv:2312.11514. 21
6. Aminabadi, R. Y., Rajbhandari, S., Awan, A. A., Li, C., Li, D., Zheng, E., Ruwase, O., Smith, S., Zhang, M., Rasley, J., et al. (2022). Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Ne… [find](https://scholar.google.com/scholar?q=Aminabadi%2C+R%2E+Y%2E%2C+Rajbhandari%2C+S%2E%2C+Awan%2C+A%2E+A%2E%2C+Li%2C+C%2E%2C+Li%2C+D%2E%2C+Zheng%2C+E%2E%2C+Ruwase%2C+O%2E%2C+Smith%2C+S%2E%2C+Zhang%2C+M%2E%2C+Rasley%2C+J%2E%2C+et+al%2E+%282022%29%2E+Deepspeed%2Dinference%3A+enabling+efficient+inference+of+transformer)
7. Bae, S., Ko, J., Song, H., and Yun, S.-Y. (2023). Fast and robust early-exiting framework for autoregressive language models with synchronized parallel decoding. 13, 14, 15 [find](https://scholar.google.com/scholar?q=Bae%2C+S%2E%2C+Ko%2C+J%2E%2C+Song%2C+H%2E%2C+and+Yun%2C+S%2E%2DY%2E+%282023%29%2E+Fast+and+robust+early%2Dexiting+framework+for+autoregressive+language+models+with+synchronized+parallel+decoding%2E+13%2C+14%2C+15)
8. Bai, H., Shi, P., Lin, J., Xie, Y., Tan, L., Xiong, K., Gao, W., and Li, M. (2021). Segatron: Segment-aware transformer for language modeling and understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12526–12534. 27 [find](https://scholar.google.com/scholar?q=Bai%2C+H%2E%2C+Shi%2C+P%2E%2C+Lin%2C+J%2E%2C+Xie%2C+Y%2E%2C+Tan%2C+L%2E%2C+Xiong%2C+K%2E%2C+Gao%2C+W%2E%2C+and+Li%2C+M%2E+%282021%29%2E+Segatron%3A+Segment%2Daware+transformer+for+language+modeling+and+understanding%2E+In+Proceedings+of+the+AAAI+Conference+o)
9. Baier-Reinio, A. and Sterck, H. D. (2020). N-ode transformer: A depth-adaptive variant of the transformer using neural ordinary differential equations. 13 [find](https://scholar.google.com/scholar?q=Baier%2DReinio%2C+A%2E+and+Sterck%2C+H%2E+D%2E+%282020%29%2E+N%2Dode+transformer%3A+A+depth%2Dadaptive+variant+of+the+transformer+using+neural+ordinary+differential+equations%2E+13)
10. Behdin, K., Acharya, A., Gupta, A., Keerthi, S., and Mazumder, R. (2023). Quantease: Optimization-based quantization for language models–an efficient and intuitive algorithm. arXiv preprint arXiv:2309.01885. 8 [find](https://scholar.google.com/scholar?q=Behdin%2C+K%2E%2C+Acharya%2C+A%2E%2C+Gupta%2C+A%2E%2C+Keerthi%2C+S%2E%2C+and+Mazumder%2C+R%2E+%282023%29%2E+Quantease%3A+Optimization%2Dbased+quantization+for+language+models%E2%80%93an+efficient+and+intuitive+algorithm%2E+arXiv+preprint+arXiv%3A2309)
11. Beltagy, I., Peters, M. E., and Cohan, A. (2020). [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). arXiv preprint arXiv:2004.05150. 27
12. Bertsch, A., Alon, U., Neubig, G., and Gormley, M. (2024). [Unlimiformer: Long-range transformers with unlimited length input](https://md.research.elamur.ai/free/en/papers/bertsch-2023-unlimiformer-long-range-transformers-unlimited/). Advances in Neural Information Processing Systems, 36. 27
13. Borgeaud, S. et al. (2021). Improving language models by retrieving from trillions of tokens. arxiv e-prints, art. arXiv preprint arXiv:2112.04426. 27 [find](https://scholar.google.com/scholar?q=Borgeaud%2C+S%2E+et+al%2E+%282021%29%2E+Improving+language+models+by+retrieving+from+trillions+of+tokens%2E+arxiv+e%2Dprints%2C+art%2E+arXiv+preprint+arXiv%3A2112%2E04426%2E+27)
14. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901. 1, 16
15. Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., and Dao, T. (2024). [Medusa: Simple llm inference acceleration framework with multiple decoding heads](https://md.research.elamur.ai/free/en/papers/cai-2024-medusa-simple-llm-inference-acceleration/). arXiv preprint arXiv:2401.10774. 18, 19
16. Chae, H., Song, Y., Ong, K. T.-i., Kwon, T., Kim, M., Yu, Y., Lee, D., Kang, D., and Yeo, J. (2023). Dialogue chain-of-thought distillation for commonsense-aware conversational agents. arXiv preprint arXiv:2310.09343. 12 [find](https://scholar.google.com/scholar?q=Chae%2C+H%2E%2C+Song%2C+Y%2E%2C+Ong%2C+K%2E+T%2E%2Di%2E%2C+Kwon%2C+T%2E%2C+Kim%2C+M%2E%2C+Yu%2C+Y%2E%2C+Lee%2C+D%2E%2C+Kang%2C+D%2E%2C+and+Yeo%2C+J%2E+%282023%29%2E+Dialogue+chain%2Dof%2Dthought+distillation+for+commonsense%2Daware+conversational+agents%2E+arXiv+preprint+)
17. Chee, J., Cai, Y., Kuleshov, V., and De Sa, C. (2023). Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems. 8 [find](https://scholar.google.com/scholar?q=Chee%2C+J%2E%2C+Cai%2C+Y%2E%2C+Kuleshov%2C+V%2E%2C+and+De+Sa%2C+C%2E+%282023%29%2E+Quip%3A+2%2Dbit+quantization+of+large+language+models+with+guarantees%2E+Advances+in+Neural+Information+Processing+Systems%2E+8)
18. Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. (2023a). Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. 17 [find](https://scholar.google.com/scholar?q=Chen%2C+C%2E%2C+Borgeaud%2C+S%2E%2C+Irving%2C+G%2E%2C+Lespiau%2C+J%2E%2DB%2E%2C+Sifre%2C+L%2E%2C+and+Jumper%2C+J%2E+%282023a%29%2E+Accelerating+large+language+model+decoding+with+speculative+sampling%2E+arXiv+preprint+arXiv%3A2302%2E01318%2E+17)
19. Chen, P. (2021). Permuteformer: Efficient relative position encoding for long sequences. 28 [find](https://scholar.google.com/scholar?q=Chen%2C+P%2E+%282021%29%2E+Permuteformer%3A+Efficient+relative+position+encoding+for+long+sequences%2E+28)
20. Chen, T., Moreau, T., Jiang, Z., Zheng, L., Yan, E., Shen, H., Cowan, M., Wang, L., Hu, Y., Ceze, L., et al. (2018). [Tvm: An automated end-to-end optimizing compiler for deep learning](https://md.research.elamur.ai/free/en/papers/chen-2018-tvm-automated-end-end-optimizing/). In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 578–594. 20
21. Chen, Y., Pan, X., Li, Y., Ding, B., and Zhou, J. (2023b). Ee-llm: Large-scale training and inference of early-exit large language models with 3d parallelism. 14, 15 [find](https://scholar.google.com/scholar?q=Chen%2C+Y%2E%2C+Pan%2C+X%2E%2C+Li%2C+Y%2E%2C+Ding%2C+B%2E%2C+and+Zhou%2C+J%2E+%282023b%29%2E+Ee%2Dllm%3A+Large%2Dscale+training+and+inference+of+early%2Dexit+large+language+models+with+3d+parallelism%2E+14%2C+15)
22. Chi, T.-C., Fan, T.-H., Ramadge, P. J., and Rudnicky, A. (2022). Kerple: Kernelized relative positional embedding for length extrapolation. Advances in Neural Information Processing Systems, 35:8386–8399. 28 [find](https://scholar.google.com/scholar?q=Chi%2C+T%2E%2DC%2E%2C+Fan%2C+T%2E%2DH%2E%2C+Ramadge%2C+P%2E+J%2E%2C+and+Rudnicky%2C+A%2E+%282022%29%2E+Kerple%3A+Kernelized+relative+positional+embedding+for+length+extrapolation%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A8386%E2%80%938)
23. Choi, J., Park, J., Kyung, K., Kim, N. S., and Ahn, J. H. (2023). Unleashing the potential of pim: Accelerating large batched inference of transformer-based generative models. IEEE Computer Architecture Letters. 23 [find](https://scholar.google.com/scholar?q=Choi%2C+J%2E%2C+Park%2C+J%2E%2C+Kyung%2C+K%2E%2C+Kim%2C+N%2E+S%2E%2C+and+Ahn%2C+J%2E+H%2E+%282023%29%2E+Unleashing+the+potential+of+pim%3A+Accelerating+large+batched+inference+of+transformer%2Dbased+generative+models%2E+IEEE+Computer+Architectu)
24. Choi, J., Wang, Z., Venkataramani, S., Chuang, P. I.-J., Srinivasan, V., and Gopalakrishnan, K. (2018). Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv:1805.06085. 8 — [arXiv:1805.06085](https://arxiv.org/abs/1805.06085) [find](https://scholar.google.com/scholar?q=Choi%2C+J%2E%2C+Wang%2C+Z%2E%2C+Venkataramani%2C+S%2E%2C+Chuang%2C+P%2E+I%2E%2DJ%2E%2C+Srinivasan%2C+V%2E%2C+and+Gopalakrishnan%2C+K%2E+%282018%29%2E+Pact%3A+Parameterized+clipping+activation+for+quantized+neural+networks%2E+arXiv+preprint+arXiv%3A1805)
25. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Aust… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
26. Chu, X., Qiao, L., Lin, X., Xu, S., Yang, Y., Hu, Y., Wei, F., Zhang, X., Zhang, B., Wei, X., et al. (2023). Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886. 26 [find](https://scholar.google.com/scholar?q=Chu%2C+X%2E%2C+Qiao%2C+L%2E%2C+Lin%2C+X%2E%2C+Xu%2C+S%2E%2C+Yang%2C+Y%2E%2C+Hu%2C+Y%2E%2C+Wei%2C+F%2E%2C+Zhang%2C+X%2E%2C+Zhang%2C+B%2E%2C+Wei%2C+X%2E%2C+et+al%2E+%282023%29%2E+Mobilevlm%3A+A+fast%2C+reproducible+and+strong+vision+language+assistant+for+mobile+devices%2E+ar)
27. Chu, X., Qiao, L., Zhang, X., Xu, S., Wei, F., Yang, Y., Sun, X., Hu, Y., Lin, X., Zhang, B., et al. (2024). Mobilevlm v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766. 26 — [arXiv:2402.03766](https://arxiv.org/abs/2402.03766) [find](https://scholar.google.com/scholar?q=Chu%2C+X%2E%2C+Qiao%2C+L%2E%2C+Zhang%2C+X%2E%2C+Xu%2C+S%2E%2C+Wei%2C+F%2E%2C+Yang%2C+Y%2E%2C+Sun%2C+X%2E%2C+Hu%2C+Y%2E%2C+Lin%2C+X%2E%2C+Zhang%2C+B%2E%2C+et+al%2E+%282024%29%2E+Mobilevlm+v2%3A+Faster+and+stronger+baseline+for+vision+language+model%2E+arXiv+preprint+arXiv%3A)
28. Corro, L. D., Giorno, A. D., Agarwal, S., Yu, B., Awadallah, A., and Mukherjee, S. (2023). Skipdecode: Autoregressive skip decoding with batching and caching for efficient llm inference. 15 [find](https://scholar.google.com/scholar?q=Corro%2C+L%2E+D%2E%2C+Giorno%2C+A%2E+D%2E%2C+Agarwal%2C+S%2E%2C+Yu%2C+B%2E%2C+Awadallah%2C+A%2E%2C+and+Mukherjee%2C+S%2E+%282023%29%2E+Skipdecode%3A+Autoregressive+skip+decoding+with+batching+and+caching+for+efficient+llm+inference%2E+15)
29. Courbariaux, M., Bengio, Y., and David, J.-P. (2015). [Binaryconnect: Training deep neural networks with binary weights during propagations](https://md.research.elamur.ai/free/en/papers/courbariaux-2015-binaryconnect-training-deep-neural-networks/). Advances in neural information processing systems, 28. 8
30. Dai, Z., Lai, G., Yang, Y., and Le, Q. (2020). Funnel-transformer: Filtering out sequential redundancy for efficient language processing. Advances in neural information processing systems, 33:4271–4282. 27 [find](https://scholar.google.com/scholar?q=Dai%2C+Z%2E%2C+Lai%2C+G%2E%2C+Yang%2C+Y%2E%2C+and+Le%2C+Q%2E+%282020%29%2E+Funnel%2Dtransformer%3A+Filtering+out+sequential+redundancy+for+efficient+language+processing%2E+Advances+in+neural+information+processing+systems%2C+33%3A4271%E2%80%93428)
31. Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., and Salakhutdinov, R. (2019). Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860. 27 [find](https://scholar.google.com/scholar?q=Dai%2C+Z%2E%2C+Yang%2C+Z%2E%2C+Yang%2C+Y%2E%2C+Carbonell%2C+J%2E%2C+Le%2C+Q%2E+V%2E%2C+and+Salakhutdinov%2C+R%2E+%282019%29%2E+Transformer%2Dxl%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+arXiv+preprint+arXiv%3A1901%2E02860%2E+27)
32. Dao, T. (2023). [Flashattention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). 20
33. Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. (2022). Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359. 20 [find](https://scholar.google.com/scholar?q=Dao%2C+T%2E%2C+Fu%2C+D%2E%2C+Ermon%2C+S%2E%2C+Rudra%2C+A%2E%2C+and+Re%CC%81%2C+C%2E+%282022%29%2E+Flashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A16344%E2%80%931635)
34. Dao, T., Haziza, D., Massa, F., and Sizov, G. (2023). Flashdecoding for long-context inference. 20 [find](https://scholar.google.com/scholar?q=Dao%2C+T%2E%2C+Haziza%2C+D%2E%2C+Massa%2C+F%2E%2C+and+Sizov%2C+G%2E+%282023%29%2E+Flashdecoding+for+long%2Dcontext+inference%2E+20)
35. Darvish Rouhani, B., Zhao, R., Elango, V., Shafipour, R., Hall, M., Mesmakhosroshahi, M., More, A., Melnick, L., Golub, M., Varatkar, G., et al. (2023). With shared microexponents, a little shifting goes a long way. In Proceedings of the 50th Annual International Symposium on Computer Architecture,… [find](https://scholar.google.com/scholar?q=Darvish+Rouhani%2C+B%2E%2C+Zhao%2C+R%2E%2C+Elango%2C+V%2E%2C+Shafipour%2C+R%2E%2C+Hall%2C+M%2E%2C+Mesmakhosroshahi%2C+M%2E%2C+More%2C+A%2E%2C+Melnick%2C+L%2E%2C+Golub%2C+M%2E%2C+Varatkar%2C+G%2E%2C+et+al%2E+%282023%29%2E+With+shared+microexponents%2C+a+little+shifting+g)
36. Deng, G., Liu, Y., Li, Y., Wang, K., Zhang, Y., Li, Z., Wang, H., Zhang, T., and Liu, Y. (2023). Jailbreaker: Automated jailbreak across multiple large language model chatbots. arXiv preprint arXiv:2307.08715. 26 — [arXiv:2307.08715](https://arxiv.org/abs/2307.08715) [find](https://scholar.google.com/scholar?q=Deng%2C+G%2E%2C+Liu%2C+Y%2E%2C+Li%2C+Y%2E%2C+Wang%2C+K%2E%2C+Zhang%2C+Y%2E%2C+Li%2C+Z%2E%2C+Wang%2C+H%2E%2C+Zhang%2C+T%2E%2C+and+Liu%2C+Y%2E+%282023%29%2E+Jailbreaker%3A+Automated+jailbreak+across+multiple+large+language+model+chatbots%2E+arXiv+preprint+arXiv%3A23)
37. Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. (2022). Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339. 8, 26 [find](https://scholar.google.com/scholar?q=Dettmers%2C+T%2E%2C+Lewis%2C+M%2E%2C+Belkada%2C+Y%2E%2C+and+Zettlemoyer%2C+L%2E+%282022%29%2E+Llm%2E+int8+%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2E+arXiv+preprint+arXiv%3A2208%2E07339%2E+8%2C+26)
38. Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2023a). [Qlora: Efficient finetuning of quantized llms](https://md.research.elamur.ai/free/en/papers/dettmers-2023-qlora-efficient-finetuning-quantized-llms/). arXiv preprint arXiv:2305.14314. 10
39. Dettmers, T., Svirschevski, R., Egiazarian, V., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T., and Alistarh, D. (2023b). Spqr: A sparse-quantized representation for near-lossless llm weight compression. arXiv preprint arXiv:2306.03078. 8 [find](https://scholar.google.com/scholar?q=Dettmers%2C+T%2E%2C+Svirschevski%2C+R%2E%2C+Egiazarian%2C+V%2E%2C+Kuznedelev%2C+D%2E%2C+Frantar%2C+E%2E%2C+Ashkboos%2C+S%2E%2C+Borzunov%2C+A%2E%2C+Hoefler%2C+T%2E%2C+and+Alistarh%2C+D%2E+%282023b%29%2E+Spqr%3A+A+sparse%2Dquantized+representation+for+near%2Dlossles)
40. Dettmers, T. and Zettlemoyer, L. (2023). [The case for 4-bit precision: k-bit inference scaling laws](https://md.research.elamur.ai/free/en/papers/dettmers-2022-case-4-bit-precision-k/). In International Conference on Machine Learning, pages 7750–7774. PMLR. 8
41. Devaux, F. (2019a). The true processing in memory accelerator. In 2019 IEEE Hot Chips 31 Symposium (HCS), pages 1–24. IEEE Computer Society. 23 [find](https://scholar.google.com/scholar?q=Devaux%2C+F%2E+%282019a%29%2E+The+true+processing+in+memory+accelerator%2E+In+2019+IEEE+Hot+Chips+31+Symposium+%28HCS%29%2C+pages+1%E2%80%9324%2E+IEEE+Computer+Society%2E+23)
42. Devaux, F. (2019b). The true processing in memory accelerator. In 2019 IEEE Hot Chips 31 Symposium (HCS), pages 1–24. IEEE Computer Society. 23 [find](https://scholar.google.com/scholar?q=Devaux%2C+F%2E+%282019b%29%2E+The+true+processing+in+memory+accelerator%2E+In+2019+IEEE+Hot+Chips+31+Symposium+%28HCS%29%2C+pages+1%E2%80%9324%2E+IEEE+Computer+Society%2E+23)
43. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). 16
44. Devvrit, Kudugunta, S., Kusupati, A., Dettmers, T., Chen, K., Dhillon, I., Tsvetkov, Y., Hajishirzi, H., Kakade, S., Farhadi, A., and Jain, P. (2023). Matformer: Nested transformer for elastic inference. 15 [find](https://scholar.google.com/scholar?q=Devvrit%2C+Kudugunta%2C+S%2E%2C+Kusupati%2C+A%2E%2C+Dettmers%2C+T%2E%2C+Chen%2C+K%2E%2C+Dhillon%2C+I%2E%2C+Tsvetkov%2C+Y%2E%2C+Hajishirzi%2C+H%2E%2C+Kakade%2C+S%2E%2C+Farhadi%2C+A%2E%2C+and+Jain%2C+P%2E+%282023%29%2E+Matformer%3A+Nested+transformer+for+elastic+inferen)
45. Din, A. Y., Karidi, T., Choshen, L., and Geva, M. (2023). Jump to conclusions: Short-cutting transformers with linear transformations. 14 [find](https://scholar.google.com/scholar?q=Din%2C+A%2E+Y%2E%2C+Karidi%2C+T%2E%2C+Choshen%2C+L%2E%2C+and+Geva%2C+M%2E+%282023%29%2E+Jump+to+conclusions%3A+Short%2Dcutting+transformers+with+linear+transformations%2E+14)
46. Ding, T., Chen, T., Zhu, H., Jiang, J., Zhong, Y., Zhou, J., Wang, G., Zhu, Z., Zharkov, I., and Liang, L. (2023). The efficiency spectrum of large language models: An algorithmic survey. arXiv preprint arXiv:2312.00678. 3, 14 — [arXiv:2312.00678](https://arxiv.org/abs/2312.00678) [find](https://scholar.google.com/scholar?q=Ding%2C+T%2E%2C+Chen%2C+T%2E%2C+Zhu%2C+H%2E%2C+Jiang%2C+J%2E%2C+Zhong%2C+Y%2E%2C+Zhou%2C+J%2E%2C+Wang%2C+G%2E%2C+Zhu%2C+Z%2E%2C+Zharkov%2C+I%2E%2C+and+Liang%2C+L%2E+%282023%29%2E+The+efficiency+spectrum+of+large+language+models%3A+An+algorithmic+survey%2E+arXiv+prepri)
47. Dong, Z., Yao, Z., Gholami, A., Mahoney, M. W., and Keutzer, K. (2019). Hawq: Hessian aware quantization of neural networks with mixed-precision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 293–302. 8 [find](https://scholar.google.com/scholar?q=Dong%2C+Z%2E%2C+Yao%2C+Z%2E%2C+Gholami%2C+A%2E%2C+Mahoney%2C+M%2E+W%2E%2C+and+Keutzer%2C+K%2E+%282019%29%2E+Hawq%3A+Hessian+aware+quantization+of+neural+networks+with+mixed%2Dprecision%2E+In+Proceedings+of+the+IEEE%2FCVF+International+Conferenc)
48. Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., et al. (2022). [Glam: Efficient scaling of language models with mixture-of-experts](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/). In International Conference on Machine Learning, pages 5547– 5569. PMLR. 16
49. Elbayad, M., Gu, J., Grave, E., and Auli, M. (2020). Depth-adaptive transformer. 13, 14, 15 [find](https://scholar.google.com/scholar?q=Elbayad%2C+M%2E%2C+Gu%2C+J%2E%2C+Grave%2C+E%2E%2C+and+Auli%2C+M%2E+%282020%29%2E+Depth%2Dadaptive+transformer%2E+13%2C+14%2C+15)
50. Fan, A., Lavril, T., Grave, E., Joulin, A., and Sukhbaatar, S. (2020). Addressing some limitations of transformers with feedback memory. arXiv preprint arXiv:2002.09402. 27 — [arXiv:2002.09402](https://arxiv.org/abs/2002.09402) [find](https://scholar.google.com/scholar?q=Fan%2C+A%2E%2C+Lavril%2C+T%2E%2C+Grave%2C+E%2E%2C+Joulin%2C+A%2E%2C+and+Sukhbaatar%2C+S%2E+%282020%29%2E+Addressing+some+limitations+of+transformers+with+feedback+memory%2E+arXiv+preprint+arXiv%3A2002%2E09402%2E+27+%E2%80%94+%5BarXiv%3A2002%2E09402%5D%28https%3A)
51. Fedus, W., Dean, J., and Zoph, B. (2022). A review of sparse expert models in deep learning. arXiv preprint arXiv:2209.01667. 16 [find](https://scholar.google.com/scholar?q=Fedus%2C+W%2E%2C+Dean%2C+J%2E%2C+and+Zoph%2C+B%2E+%282022%29%2E+A+review+of+sparse+expert+models+in+deep+learning%2E+arXiv+preprint+arXiv%3A2209%2E01667%2E+16)
52. Fedus, W., Zoph, B., and Shazeer, N. (2021). Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.(2021). arXiv preprint cs.LG/2101.03961. 15 [find](https://scholar.google.com/scholar?q=Fedus%2C+W%2E%2C+Zoph%2C+B%2E%2C+and+Shazeer%2C+N%2E+%282021%29%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E%282021%29%2E+arXiv+preprint+cs%2ELG%2F2101%2E03961%2E+15)
53. Frantar, E. and Alistarh, D. (2023). Sparsegpt: Massive language models can be accurately pruned in one-shot. ICML. 11 [find](https://scholar.google.com/scholar?q=Frantar%2C+E%2E+and+Alistarh%2C+D%2E+%282023%29%2E+Sparsegpt%3A+Massive+language+models+can+be+accurately+pruned+in+one%2Dshot%2E+ICML%2E+11)
54. Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. (2022). Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323. 8 [find](https://scholar.google.com/scholar?q=Frantar%2C+E%2E%2C+Ashkboos%2C+S%2E%2C+Hoefler%2C+T%2E%2C+and+Alistarh%2C+D%2E+%282022%29%2E+Gptq%3A+Accurate+post%2Dtraining+quantization+for+generative+pre%2Dtrained+transformers%2E+arXiv+preprint+arXiv%3A2210%2E17323%2E+8)
55. Fu, Y., Bailis, P., Stoica, I., and Zhang, H. (2023a). Breaking the sequential dependency of llm inference using lookahead decoding. 19 [find](https://scholar.google.com/scholar?q=Fu%2C+Y%2E%2C+Bailis%2C+P%2E%2C+Stoica%2C+I%2E%2C+and+Zhang%2C+H%2E+%282023a%29%2E+Breaking+the+sequential+dependency+of+llm+inference+using+lookahead+decoding%2E+19)
56. Fu, Y., Peng, H., Ou, L., Sabharwal, A., and Khot, T. (2023b). [Specializing smaller language models towards multi-step reasoning](https://md.research.elamur.ai/free/en/papers/fu-2023-specializing-smaller-language-models-towards/). arXiv preprint arXiv:2301.12726. 12
57. Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., and Wang, H. (2023). [Retrieval-augmented generation for large language models: A survey](https://md.research.elamur.ai/free/en/papers/gao-2023-retrieval-augmented-generation-large-language/). arXiv preprint arXiv:2312.10997. 27
58. Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F. A. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673. 25, 26 [find](https://scholar.google.com/scholar?q=Geirhos%2C+R%2E%2C+Jacobsen%2C+J%2E%2DH%2E%2C+Michaelis%2C+C%2E%2C+Zemel%2C+R%2E%2C+Brendel%2C+W%2E%2C+Bethge%2C+M%2E%2C+and+Wichmann%2C+F%2E+A%2E+%282020%29%2E+Shortcut+learning+in+deep+neural+networks%2E+Nature+Machine+Intelligence%2C+2%2811%29%3A665%E2%80%93673%2E+25%2C+)
59. Geva, M., Caciularu, A., Wang, K. R., and Goldberg, Y. (2022). [Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space](https://md.research.elamur.ai/free/en/papers/geva-2022-transformer-feed-forward-layers-build/). 13
60. Ghazvininejad, M., Levy, O., Liu, Y., and Zettlemoyer, L. (2019). [Mask-predict: Parallel decoding of conditional masked language models](https://md.research.elamur.ai/free/en/papers/ghazvininejad-2019-mask-predict-parallel-decoding-conditional/). 19
61. Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., and Keutzer, K. (2022). A survey of quantization methods for efficient neural network inference. In Low-Power Computer Vision, pages 291–326. Chapman and Hall/CRC. 6, 24 [find](https://scholar.google.com/scholar?q=Gholami%2C+A%2E%2C+Kim%2C+S%2E%2C+Dong%2C+Z%2E%2C+Yao%2C+Z%2E%2C+Mahoney%2C+M%2E+W%2E%2C+and+Keutzer%2C+K%2E+%282022%29%2E+A+survey+of+quantization+methods+for+efficient+neural+network+inference%2E+In+Low%2DPower+Computer+Vision%2C+pages+291%E2%80%93326%2E+C)
62. Gou, J., Yu, B., Maybank, S. J., and Tao, D. (2021). Knowledge distillation: A survey. International Journal of Computer Vision, 129:1789–1819. 11 [find](https://scholar.google.com/scholar?q=Gou%2C+J%2E%2C+Yu%2C+B%2E%2C+Maybank%2C+S%2E+J%2E%2C+and+Tao%2C+D%2E+%282021%29%2E+Knowledge+distillation%3A+A+survey%2E+International+Journal+of+Computer+Vision%2C+129%3A1789%E2%80%931819%2E+11)
63. Graphcore (2024). Tasks and tutorials using graphcore’s ipu with hugging face. [https://github.com/graphcore/Gradient-HuggingFace](https://github.com/graphcore/Gradient-HuggingFace). Accessed on March 10, 2024. 23 [↗](https://github.com/graphcore/Gradient-HuggingFace)
64. Groq (2023). Groq sets new large language model performance record of 300 tokens per second per user on meta ai foundational llm llama 2.70b. [https://wow.groq.com/groq](https://wow.groq.com/groq) - sets - new - large - language - model - performance - record - of - 300 - tokens - per - second- per- user- on- meta- ai- found… [↗](https://wow.groq.com/groq)
65. Gu, J., Bradbury, J., Xiong, C., Li, V. O., and Socher, R. (2017). [Non-autoregressive neural machine translation](https://md.research.elamur.ai/free/en/papers/gu-2017-non-autoregressive-neural-machine-translation/). arXiv preprint arXiv:1711.02281. 19
66. Gu, J. and Kong, X. (2020). [Fully non-autoregressive neural machine translation: Tricks of the trade](https://md.research.elamur.ai/free/en/papers/gu-2021-fully-non-autoregressive-neural-machine/). 19
67. Gu, Y., Dong, L., Wei, F., and Huang, M. (2023). [Knowledge distillation of large language models](https://md.research.elamur.ai/free/en/papers/gu-2023-minillm-policy-distillation-large-language/). arXiv preprint arXiv:2306.08543. 11
68. Guo, C., Tang, J., Hu, W., Leng, J., Zhang, C., Yang, F., Liu, Y., Guo, M., and Zhu, Y. (2023a). Olive: Accelerating large language models via hardware-friendly outlier-victim pair quantization. In Proceedings of the 50th Annual International Symposium on Computer Architecture, pages 1–15. 9, 24 [find](https://scholar.google.com/scholar?q=Guo%2C+C%2E%2C+Tang%2C+J%2E%2C+Hu%2C+W%2E%2C+Leng%2C+J%2E%2C+Zhang%2C+C%2E%2C+Yang%2C+F%2E%2C+Liu%2C+Y%2E%2C+Guo%2C+M%2E%2C+and+Zhu%2C+Y%2E+%282023a%29%2E+Olive%3A+Accelerating+large+language+models+via+hardware%2Dfriendly+outlier%2Dvictim+pair+quantization%2E+In+Pr)
69. Guo, C., Zhang, C., Leng, J., Liu, Z., Yang, F., Liu, Y., Guo, M., and Zhu, Y. (2022a). Ant: Exploiting adaptive numerical data type for low-bit deep neural network quantization. In 2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 1414–1433. IEEE. 24 [find](https://scholar.google.com/scholar?q=Guo%2C+C%2E%2C+Zhang%2C+C%2E%2C+Leng%2C+J%2E%2C+Liu%2C+Z%2E%2C+Yang%2C+F%2E%2C+Liu%2C+Y%2E%2C+Guo%2C+M%2E%2C+and+Zhu%2C+Y%2E+%282022a%29%2E+Ant%3A+Exploiting+adaptive+numerical+data+type+for+low%2Dbit+deep+neural+network+quantization%2E+In+2022+55th+IEEE%2FACM)
70. Guo, H., Greengard, P., Xing, E. P., and Kim, Y. (2023b). Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. arXiv preprint arXiv:2311.12023. 10 [find](https://scholar.google.com/scholar?q=Guo%2C+H%2E%2C+Greengard%2C+P%2E%2C+Xing%2C+E%2E+P%2E%2C+and+Kim%2C+Y%2E+%282023b%29%2E+Lq%2Dlora%3A+Low%2Drank+plus+quantized+matrix+decomposition+for+efficient+language+model+finetuning%2E+arXiv+preprint+arXiv%3A2311%2E12023%2E+10)
71. Guo, J., Xu, L., and Chen, E. (2020). Jointly masked sequence-to-sequence model for non-autoregressive neural machine translation. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J., editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 376–3… [find](https://scholar.google.com/scholar?q=Guo%2C+J%2E%2C+Xu%2C+L%2E%2C+and+Chen%2C+E%2E+%282020%29%2E+Jointly+masked+sequence%2Dto%2Dsequence+model+for+non%2Dautoregressive+neural+machine+translation%2E+In+Jurafsky%2C+D%2E%2C+Chai%2C+J%2E%2C+Schluter%2C+N%2E%2C+and+Tetreault%2C+J%2E%2C+editors%2C+)
72. Guo, M., Ainslie, J., Uthus, D., Ontanon, S., Ni, J., Sung, Y.-H., and Yang, Y. (2022b). [Longt5: Efficient text-to-text transformer for long sequences](https://md.research.elamur.ai/free/en/papers/guo-2022-longt5-efficient-text-text-transformer/). 27
73. Guo, S., Xu, J., Zhang, L. L., and Yang, M. (2023c). Compresso: Structured pruning with collaborative prompting learns compact large language models. arXiv preprint arXiv:2310.05015. 11 — [arXiv:2310.05015](https://arxiv.org/abs/2310.05015) [find](https://scholar.google.com/scholar?q=Guo%2C+S%2E%2C+Xu%2C+J%2E%2C+Zhang%2C+L%2E+L%2E%2C+and+Yang%2C+M%2E+%282023c%29%2E+Compresso%3A+Structured+pruning+with+collaborative+prompting+learns+compact+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E05015%2E+11+%E2%80%94+%5BarXiv%3A2310%2E)
74. Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. (2020). [Retrieval augmented language model pre-training](https://md.research.elamur.ai/free/en/papers/guu-2020-realm-retrieval-augmented-language-model/). In International conference on machine learning, pages 3929–3938. PMLR. 27
75. Han, C., Wang, Q., Xiong, W., Chen, Y., Ji, H., and Wang, S. (2023). Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308.16137. 27 — [arXiv:2308.16137](https://arxiv.org/abs/2308.16137) [find](https://scholar.google.com/scholar?q=Han%2C+C%2E%2C+Wang%2C+Q%2E%2C+Xiong%2C+W%2E%2C+Chen%2C+Y%2E%2C+Ji%2C+H%2E%2C+and+Wang%2C+S%2E+%282023%29%2E+Lm%2Dinfinite%3A+Simple+on%2Dthe%2Dfly+length+generalization+for+large+language+models%2E+arXiv+preprint+arXiv%3A2308%2E16137%2E+27+%E2%80%94+%5BarXiv%3A2308%2E1)
76. He, Z., Zhong, Z., Cai, T., Lee, J. D., and He, D. (2023). [Rest: Retrieval-based speculative decoding](https://md.research.elamur.ai/free/en/papers/he-2023-rest-retrieval-based-speculative-decoding/). arXiv preprint arXiv:2311.08252. 18
77. Hinton, G., Vinyals, O., and Dean, J. (2015). [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). arXiv preprint arXiv:1503.02531. 11
78. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L.… — [Training Compute-Optimal Large Language Models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
79. Holmes, C., Tanaka, M., Wyatt, M., Awan, A. A., Rasley, J., Rajbhandari, S., Aminabadi, R. Y., Qin, H., Bakhtiari, A., Kurilenko, L., et al. (2024). [Deepspeed-fastgen: High-throughput text generation for llms via mii and deepspeed-inference](https://md.research.elamur.ai/free/en/papers/holmes-2024-deepspeed-fastgen-high-throughput-text/). arXiv preprint arXiv:2401.08671. 22
80. Hong, K., Dai, G., Xu, J., Mao, Q., Li, X., Liu, J., Chen, K., Dong, H., and Wang, Y. (2023). Flashdecoding++: Faster large language model inference on gpus. arXiv preprint arXiv:2311.01282. 21 [find](https://scholar.google.com/scholar?q=Hong%2C+K%2E%2C+Dai%2C+G%2E%2C+Xu%2C+J%2E%2C+Mao%2C+Q%2E%2C+Li%2C+X%2E%2C+Liu%2C+J%2E%2C+Chen%2C+K%2E%2C+Dong%2C+H%2E%2C+and+Wang%2C+Y%2E+%282023%29%2E+Flashdecoding%2B%2B%3A+Faster+large+language+model+inference+on+gpus%2E+arXiv+preprint+arXiv%3A2311%2E01282%2E+21)
81. Hong, S., Moon, S., Kim, J., Lee, S., Kim, M., Lee, D., and Kim, J.-Y. (2022). Dfx: A low-latency multi-fpga appliance for accelerating transformer-based text generation. In 2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 616–630. IEEE. 25 [find](https://scholar.google.com/scholar?q=Hong%2C+S%2E%2C+Moon%2C+S%2E%2C+Kim%2C+J%2E%2C+Lee%2C+S%2E%2C+Kim%2C+M%2E%2C+Lee%2C+D%2E%2C+and+Kim%2C+J%2E%2DY%2E+%282022%29%2E+Dfx%3A+A+low%2Dlatency+multi%2Dfpga+appliance+for+accelerating+transformer%2Dbased+text+generation%2E+In+2022+55th+IEEE%2FACM+Interna)
82. Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., and Gholami, A. (2024). [Kvquant: Towards 10 million context length llm inference with kv cache quantization](https://md.research.elamur.ai/free/en/papers/hooper-2024-kvquant-towards-10-million-context/). arXiv preprint arXiv:2401.18079. 9
83. Hou, L., Huang, Z., Shang, L., Jiang, X., Chen, X., and Liu, Q. (2020). Dynabert: Dynamic bert with adaptive width and depth. 13 [find](https://scholar.google.com/scholar?q=Hou%2C+L%2E%2C+Huang%2C+Z%2E%2C+Shang%2C+L%2E%2C+Jiang%2C+X%2E%2C+Chen%2C+X%2E%2C+and+Liu%2C+Q%2E+%282020%29%2E+Dynabert%3A+Dynamic+bert+with+adaptive+width+and+depth%2E+13)
84. Hsieh, C.-Y., Li, C.-L., Yeh, C.-K., Nakhost, H., Fujii, Y., Ratner, A., Krishna, R., Lee, C.-Y., and Pfister, T. (2023). [Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes](https://md.research.elamur.ai/free/en/papers/hsieh-2023-distilling-step-step-outperforming-larger/). arXiv preprint arXiv:2305.02301. 12
85. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2021). [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). arXiv preprint arXiv:2106.09685. 10
86. Hu, Z., Lan, Y., Wang, L., Xu, W., Lim, E.-P., Lee, R. K.-W., Bing, L., and Poria, S. (2023). [Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models](https://md.research.elamur.ai/free/en/papers/hu-2023-llm-adapters-adapter-family-parameter/). arXiv preprint arXiv:2304.01933. 10
87. Huang, J. and Yau, H.-T. (2019). Dynamics of deep neural networks and neural tangent hierarchy. 28 [find](https://scholar.google.com/scholar?q=Huang%2C+J%2E+and+Yau%2C+H%2E%2DT%2E+%282019%29%2E+Dynamics+of+deep+neural+networks+and+neural+tangent+hierarchy%2E+28)
88. Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023a). [A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/). arXiv preprint arXiv:2311.05232. 25
89. Huang, W., Liu, Y., Qin, H., Li, Y., Zhang, S., Liu, X., Magno, M., and Qi, X. (2024). Billm: Pushing the limit of post-training quantization for llms. arXiv preprint arXiv:2402.04291. 9 [find](https://scholar.google.com/scholar?q=Huang%2C+W%2E%2C+Liu%2C+Y%2E%2C+Qin%2C+H%2E%2C+Li%2C+Y%2E%2C+Zhang%2C+S%2E%2C+Liu%2C+X%2E%2C+Magno%2C+M%2E%2C+and+Qi%2C+X%2E+%282024%29%2E+Billm%3A+Pushing+the+limit+of+post%2Dtraining+quantization+for+llms%2E+arXiv+preprint+arXiv%3A2402%2E04291%2E+9)
90. Huang, Y., Chen, Y., Yu, Z., and McKeown, K. (2022). In-context learning distillation: Transferring few-shot learning ability of pre-trained language models. arXiv preprint arXiv:2212.10670. 12 — [arXiv:2212.10670](https://arxiv.org/abs/2212.10670) [find](https://scholar.google.com/scholar?q=Huang%2C+Y%2E%2C+Chen%2C+Y%2E%2C+Yu%2C+Z%2E%2C+and+McKeown%2C+K%2E+%282022%29%2E+In%2Dcontext+learning+distillation%3A+Transferring+few%2Dshot+learning+ability+of+pre%2Dtrained+language+models%2E+arXiv+preprint+arXiv%3A2212%2E10670%2E+12+%E2%80%94+%5BarX)
91. Huang, Y., Xu, J., Jiang, Z., Lai, J., Li, Z., Yao, Y., Chen, T., Yang, L., Xin, Z., and Ma, X. (2023b). Advancing transformer architecture in long-context large language models: A comprehensive survey. arXiv preprint arXiv:2311.12351. 26 — [arXiv:2311.12351](https://arxiv.org/abs/2311.12351) [find](https://scholar.google.com/scholar?q=Huang%2C+Y%2E%2C+Xu%2C+J%2E%2C+Jiang%2C+Z%2E%2C+Lai%2C+J%2E%2C+Li%2C+Z%2E%2C+Yao%2C+Y%2E%2C+Chen%2C+T%2E%2C+Yang%2C+L%2E%2C+Xin%2C+Z%2E%2C+and+Ma%2C+X%2E+%282023b%29%2E+Advancing+transformer+architecture+in+long%2Dcontext+large+language+models%3A+A+comprehensive+surve)
92. Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., and Bengio, Y. (2016). Binarized neural networks. Advances in neural information processing systems, 29. 9 [find](https://scholar.google.com/scholar?q=Hubara%2C+I%2E%2C+Courbariaux%2C+M%2E%2C+Soudry%2C+D%2E%2C+El%2DYaniv%2C+R%2E%2C+and+Bengio%2C+Y%2E+%282016%29%2E+Binarized+neural+networks%2E+Advances+in+neural+information+processing+systems%2C+29%2E+9)
93. HuggingFace (2022). Hugging face accelerate. 21 [find](https://scholar.google.com/scholar?q=HuggingFace+%282022%29%2E+Hugging+face+accelerate%2E+21)
94. Javaheripi, M., Bubeck, S., Abdin, M., Aneja, J., Bubeck, S., Mendes, C. C. T., Chen, W., Del Giorno, A., Eldan, R., Gopi, S., et al. (2023). Phi-2: The surprising power of small language models. Microsoft Research Blog. 26 [find](https://scholar.google.com/scholar?q=Javaheripi%2C+M%2E%2C+Bubeck%2C+S%2E%2C+Abdin%2C+M%2E%2C+Aneja%2C+J%2E%2C+Bubeck%2C+S%2E%2C+Mendes%2C+C%2E+C%2E+T%2E%2C+Chen%2C+W%2E%2C+Del+Giorno%2C+A%2E%2C+Eldan%2C+R%2E%2C+Gopi%2C+S%2E%2C+et+al%2E+%282023%29%2E+Phi%2D2%3A+The+surprising+power+of+small+language+models%2E+Micr)
95. Jawahar, G., Mukherjee, S., Liu, X., Kim, Y. J., Mageed, M. A., Laks Lakshmanan, V., Hassan, A., Bubeck, S., and Gao, J. (2023). Automoe: Heterogeneous mixture-of-experts with adaptive computation for efficient neural machine translation. In Findings of the Association for Computational Linguistics:… [find](https://scholar.google.com/scholar?q=Jawahar%2C+G%2E%2C+Mukherjee%2C+S%2E%2C+Liu%2C+X%2E%2C+Kim%2C+Y%2E+J%2E%2C+Mageed%2C+M%2E+A%2E%2C+Laks+Lakshmanan%2C+V%2E%2C+Hassan%2C+A%2E%2C+Bubeck%2C+S%2E%2C+and+Gao%2C+J%2E+%282023%29%2E+Automoe%3A+Heterogeneous+mixture%2Dof%2Dexperts+with+adaptive+computation+for)
96. Ji, Y., Wang, J., Li, J., Chen, Q., Chen, W., and Zhang, M. (2023). Early exit with disentangled representation and equiangular tight frame. In Rogers, A., Boyd-Graber, J., and Okazaki, N., editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 14128–14142, Toronto, Cana… [find](https://scholar.google.com/scholar?q=Ji%2C+Y%2E%2C+Wang%2C+J%2E%2C+Li%2C+J%2E%2C+Chen%2C+Q%2E%2C+Chen%2C+W%2E%2C+and+Zhang%2C+M%2E+%282023%29%2E+Early+exit+with+disentangled+representation+and+equiangular+tight+frame%2E+In+Rogers%2C+A%2E%2C+Boyd%2DGraber%2C+J%2E%2C+and+Okazaki%2C+N%2E%2C+editors%2C+F)
97. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. (2024). Mixtral of experts. arXiv preprint arXiv:2401.04088. 16 [find](https://scholar.google.com/scholar?q=Jiang%2C+A%2E+Q%2E%2C+Sablayrolles%2C+A%2E%2C+Roux%2C+A%2E%2C+Mensch%2C+A%2E%2C+Savary%2C+B%2E%2C+Bamford%2C+C%2E%2C+Chaplot%2C+D%2E+S%2E%2C+Casas%2C+D%2E+d%2E+l%2E%2C+Hanna%2C+E%2E+B%2E%2C+Bressand%2C+F%2E%2C+et+al%2E+%282024%29%2E+Mixtral+of+experts%2E+arXiv+preprint+arXiv%3A2401)
98. Jiang, Y., Chan, C., Chen, M., and Wang, W. (2023). Lion: Adversarial distillation of closed-source large language model. arXiv preprint arXiv:2305.12870. 12 — [arXiv:2305.12870](https://arxiv.org/abs/2305.12870) [find](https://scholar.google.com/scholar?q=Jiang%2C+Y%2E%2C+Chan%2C+C%2E%2C+Chen%2C+M%2E%2C+and+Wang%2C+W%2E+%282023%29%2E+Lion%3A+Adversarial+distillation+of+closed%2Dsource+large+language+model%2E+arXiv+preprint+arXiv%3A2305%2E12870%2E+12+%E2%80%94+%5BarXiv%3A2305%2E12870%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
99. Jonatan, G., Cho, H., Son, H., Wu, X., Livesay, N., Mora, E., Shivdikar, K., Abellán, J. L., Joshi, A., Kaeli, D., et al. (2024). Scalability limitations of processing-in-memory using real system evaluations. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 8(1):1–28. 23 [find](https://scholar.google.com/scholar?q=Jonatan%2C+G%2E%2C+Cho%2C+H%2E%2C+Son%2C+H%2E%2C+Wu%2C+X%2E%2C+Livesay%2C+N%2E%2C+Mora%2C+E%2E%2C+Shivdikar%2C+K%2E%2C+Abella%CC%81n%2C+J%2E+L%2E%2C+Joshi%2C+A%2E%2C+Kaeli%2C+D%2E%2C+et+al%2E+%282024%29%2E+Scalability+limitations+of+processing%2Din%2Dmemory+using+real+system+eva)
100. Kao, S.-C., Subramanian, S., Agrawal, G., Yazdanbakhsh, A., and Krishna, T. (2023). Flat: An optimized dataflow for mitigating attention bottlenecks. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pages 295–… [find](https://scholar.google.com/scholar?q=Kao%2C+S%2E%2DC%2E%2C+Subramanian%2C+S%2E%2C+Agrawal%2C+G%2E%2C+Yazdanbakhsh%2C+A%2E%2C+and+Krishna%2C+T%2E+%282023%29%2E+Flat%3A+An+optimized+dataflow+for+mitigating+attention+bottlenecks%2E+In+Proceedings+of+the+28th+ACM+International+Confe)
101. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). 15
102. Kim, J., Lee, J. H., Kim, S., Park, J., Yoo, K. M., Kwon, S. J., and Lee, D. (2023a). Memory-efficient fine-tuning of compressed large language models via sub-4-bit integer quantization. arXiv preprint arXiv:2305.14152. 10 — [arXiv:2305.14152](https://arxiv.org/abs/2305.14152) [find](https://scholar.google.com/scholar?q=Kim%2C+J%2E%2C+Lee%2C+J%2E+H%2E%2C+Kim%2C+S%2E%2C+Park%2C+J%2E%2C+Yoo%2C+K%2E+M%2E%2C+Kwon%2C+S%2E+J%2E%2C+and+Lee%2C+D%2E+%282023a%29%2E+Memory%2Defficient+fine%2Dtuning+of+compressed+large+language+models+via+sub%2D4%2Dbit+integer+quantization%2E+arXiv+preprin)
103. Kim, M., Lee, S., Lee, J., Hong, S., Chang, D.-S., Sung, W., and Choi, J. (2023b). Token-scaled logit distillation for ternary weight generative language models. arXiv preprint arXiv:2308.06744. 8 — [arXiv:2308.06744](https://arxiv.org/abs/2308.06744) [find](https://scholar.google.com/scholar?q=Kim%2C+M%2E%2C+Lee%2C+S%2E%2C+Lee%2C+J%2E%2C+Hong%2C+S%2E%2C+Chang%2C+D%2E%2DS%2E%2C+Sung%2C+W%2E%2C+and+Choi%2C+J%2E+%282023b%29%2E+Token%2Dscaled+logit+distillation+for+ternary+weight+generative+language+models%2E+arXiv+preprint+arXiv%3A2308%2E06744%2E+8+%E2%80%94+%5B)
104. Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. (2023c). [Squeezellm: Dense-and-sparse quantization](https://md.research.elamur.ai/free/en/papers/kim-2023-squeezellm-dense-sparse-quantization/). arXiv preprint arXiv:2306.07629. 8
105. Kim, S., Hooper, C., Wattanawong, T., Kang, M., Yan, R., Genc, H., Dinh, G., Huang, Q., Keutzer, K., Mahoney, M. W., Shao, Y. S., and Gholami, A. (2023d). Full stack optimization of transformer inference: a survey. 13 [find](https://scholar.google.com/scholar?q=Kim%2C+S%2E%2C+Hooper%2C+C%2E%2C+Wattanawong%2C+T%2E%2C+Kang%2C+M%2E%2C+Yan%2C+R%2E%2C+Genc%2C+H%2E%2C+Dinh%2C+G%2E%2C+Huang%2C+Q%2E%2C+Keutzer%2C+K%2E%2C+Mahoney%2C+M%2E+W%2E%2C+Shao%2C+Y%2E+S%2E%2C+and+Gholami%2C+A%2E+%282023d%29%2E+Full+stack+optimization+of+transformer+infere)
106. Kim, S., Mangalam, K., Moon, S., Malik, J., Mahoney, M. W., Gholami, A., and Keutzer, K. (2023e). Speculative decoding with big little decoder. 17 [find](https://scholar.google.com/scholar?q=Kim%2C+S%2E%2C+Mangalam%2C+K%2E%2C+Moon%2C+S%2E%2C+Malik%2C+J%2E%2C+Mahoney%2C+M%2E+W%2E%2C+Gholami%2C+A%2E%2C+and+Keutzer%2C+K%2E+%282023e%29%2E+Speculative+decoding+with+big+little+decoder%2E+17)
107. Kishore Kumar, N. and Schneider, J. (2017). Literature survey on low rank approximation of matrices. Linear and Multilinear Algebra, 65(11):2212–2244. 12 [find](https://scholar.google.com/scholar?q=Kishore+Kumar%2C+N%2E+and+Schneider%2C+J%2E+%282017%29%2E+Literature+survey+on+low+rank+approximation+of+matrices%2E+Linear+and+Multilinear+Algebra%2C+65%2811%29%3A2212%E2%80%932244%2E+12)
108. Kossmann, F., Jia, Z., and Aiken, A. (2022). Optimizing mixture of experts using dynamic recompilations. arXiv preprint arXiv:2205.01848. 16 — [arXiv:2205.01848](https://arxiv.org/abs/2205.01848) [find](https://scholar.google.com/scholar?q=Kossmann%2C+F%2E%2C+Jia%2C+Z%2E%2C+and+Aiken%2C+A%2E+%282022%29%2E+Optimizing+mixture+of+experts+using+dynamic+recompilations%2E+arXiv+preprint+arXiv%3A2205%2E01848%2E+16+%E2%80%94+%5BarXiv%3A2205%2E01848%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2205%2E01848%29)
109. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. (2023). [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626. 21
110. Kwon, Y., Lee, S. H., Lee, J., Kwon, S., Ryu, J., Son, J., O, S., Yu, H., Lee, H., Kim, S. Y., Cho, Y., Kim, J. G., Choi, J., Shin, H., Kim, J., Phuah, B., Kim, H., Song, M. J., Choi, A., Kim, D., Kim, S., Kim, E., Wang, D., Kang, S., Ro, Y., Seo, S., Song, J., Youn, J., Sohn, K., and Kim, N. S. (20… [find](https://scholar.google.com/scholar?q=Kwon%2C+Y%2E%2C+Lee%2C+S%2E+H%2E%2C+Lee%2C+J%2E%2C+Kwon%2C+S%2E%2C+Ryu%2C+J%2E%2C+Son%2C+J%2E%2C+O%2C+S%2E%2C+Yu%2C+H%2E%2C+Lee%2C+H%2E%2C+Kim%2C+S%2E+Y%2E%2C+Cho%2C+Y%2E%2C+Kim%2C+J%2E+G%2E%2C+Choi%2C+J%2E%2C+Shin%2C+H%2E%2C+Kim%2C+J%2E%2C+Phuah%2C+B%2E%2C+Kim%2C+H%2E%2C+Song%2C+M%2E+J%2E%2C+Choi%2C+A%2E%2C+Kim%2C+D%2E%2C+Kim)
111. Kwon, Y., Vladimir, K., Kim, N., Shin, W., Won, J., Lee, M., Joo, H., Choi, H., Kim, G., An, B., et al. (2022). System architecture and software stack for gddr6-aim. In 2022 IEEE Hot Chips 34 Symposium (HCS), pages 1–25. IEEE. 23 [find](https://scholar.google.com/scholar?q=Kwon%2C+Y%2E%2C+Vladimir%2C+K%2E%2C+Kim%2C+N%2E%2C+Shin%2C+W%2E%2C+Won%2C+J%2E%2C+Lee%2C+M%2E%2C+Joo%2C+H%2E%2C+Choi%2C+H%2E%2C+Kim%2C+G%2E%2C+An%2C+B%2E%2C+et+al%2E+%282022%29%2E+System+architecture+and+software+stack+for+gddr6%2Daim%2E+In+2022+IEEE+Hot+Chips+34+Symposiu)
112. Kynoch, B., Latapie, H., and van der Sluis, D. (2023). Recallm: An adaptable memory mechanism with temporal understanding for large language models. 27 [find](https://scholar.google.com/scholar?q=Kynoch%2C+B%2E%2C+Latapie%2C+H%2E%2C+and+van+der+Sluis%2C+D%2E+%282023%29%2E+Recallm%3A+An+adaptable+memory+mechanism+with+temporal+understanding+for+large+language+models%2E+27)
113. Lan, T., Cai, D., Wang, Y., Huang, H., and Mao, X.-L. (2023). Copy is all you need. 18 [find](https://scholar.google.com/scholar?q=Lan%2C+T%2E%2C+Cai%2C+D%2E%2C+Wang%2C+Y%2E%2C+Huang%2C+H%2E%2C+and+Mao%2C+X%2E%2DL%2E+%282023%29%2E+Copy+is+all+you+need%2E+18)
114. Langroudi, H. F., Karia, V., Carmichael, Z., Zyarah, A., Pandit, T., Gustafson, J. L., and Kudithipudi, D. (2021). Alps: Adaptive quantization of deep neural networks with generalized posits. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3100–3109. 24 [find](https://scholar.google.com/scholar?q=Langroudi%2C+H%2E+F%2E%2C+Karia%2C+V%2E%2C+Carmichael%2C+Z%2E%2C+Zyarah%2C+A%2E%2C+Pandit%2C+T%2E%2C+Gustafson%2C+J%2E+L%2E%2C+and+Kudithipudi%2C+D%2E+%282021%29%2E+Alps%3A+Adaptive+quantization+of+deep+neural+networks+with+generalized+posits%2E+In+Proce)
115. LeCun, Y., Denker, J., and Solla, S. (1989). Optimal brain damage. Advances in neural information processing systems, 2. 11 [find](https://scholar.google.com/scholar?q=LeCun%2C+Y%2E%2C+Denker%2C+J%2E%2C+and+Solla%2C+S%2E+%281989%29%2E+Optimal+brain+damage%2E+Advances+in+neural+information+processing+systems%2C+2%2E+11)
116. Lee, C., Jin, J., Kim, T., Kim, H., and Park, E. (2023). [Owq: Lessons learned from activation outliers for weight quantization in large language models](https://md.research.elamur.ai/free/en/papers/lee-2023-owq-outlier-aware-weight-quantization/). arXiv preprint arXiv:2306.02272. 8
117. Lee, J., Mansimov, E., and Cho, K. (2018). [Deterministic non-autoregressive neural sequence modeling by iterative refinement](https://md.research.elamur.ai/free/en/papers/lee-2018-deterministic-non-autoregressive-neural-sequence/). 19
118. Lee-Thorp, J. and Ainslie, J. (2022). Sparse mixers: Combining moe and mixing to build a more efficient bert. arXiv preprint arXiv:2205.12399. 16 [find](https://scholar.google.com/scholar?q=Lee%2DThorp%2C+J%2E+and+Ainslie%2C+J%2E+%282022%29%2E+Sparse+mixers%3A+Combining+moe+and+mixing+to+build+a+more+efficient+bert%2E+arXiv+preprint+arXiv%3A2205%2E12399%2E+16)
119. Lefaudeux, B., Massa, F., Liskovich, D., Xiong, W., Caggiano, V., Naren, S., Xu, M., Hu, J., Tintore, M., Zhang, S., Labatut, P., Haziza, D., Wehrstedt, L., Reizenstein, J., and Sizov, G. (2022). xformers: A modular and hackable transformer modelling library. [https://github.com/facebookresearch/xfor…](https://github.com/facebookresearch/xfor…) [↗](https://github.com/facebookresearch/xfor…)
120. Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. (2020). Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668. 15 [find](https://scholar.google.com/scholar?q=Lepikhin%2C+D%2E%2C+Lee%2C+H%2E%2C+Xu%2C+Y%2E%2C+Chen%2C+D%2E%2C+Firat%2C+O%2E%2C+Huang%2C+Y%2E%2C+Krikun%2C+M%2E%2C+Shazeer%2C+N%2E%2C+and+Chen%2C+Z%2E+%282020%29%2E+Gshard%3A+Scaling+giant+models+with+conditional+computation+and+automatic+sharding%2E+arXiv+pre)
121. Leviathan, Y., Kalman, M., and Matias, Y. (2023). Fast inference from transformers via speculative decoding. 17 [find](https://scholar.google.com/scholar?q=Leviathan%2C+Y%2E%2C+Kalman%2C+M%2E%2C+and+Matias%2C+Y%2E+%282023%29%2E+Fast+inference+from+transformers+via+speculative+decoding%2E+17)
122. Li, L., Li, Q., Zhang, B., and Chu, X. (2023a). Norm tweaking: High-performance low-bit quantization of large language models. arXiv preprint arXiv:2309.02784. 9 — [arXiv:2309.02784](https://arxiv.org/abs/2309.02784) [find](https://scholar.google.com/scholar?q=Li%2C+L%2E%2C+Li%2C+Q%2E%2C+Zhang%2C+B%2E%2C+and+Chu%2C+X%2E+%282023a%29%2E+Norm+tweaking%3A+High%2Dperformance+low%2Dbit+quantization+of+large+language+models%2E+arXiv+preprint+arXiv%3A2309%2E02784%2E+9+%E2%80%94+%5BarXiv%3A2309%2E02784%5D%28https%3A%2F%2Farxiv%2Eorg)
123. Li, L., Lin, Y., Chen, D., Ren, S., Li, P., Zhou, J., and Sun, X. (2021). Cascadebert: Accelerating inference of pre-trained language models via calibrated complete models cascade. 13 [find](https://scholar.google.com/scholar?q=Li%2C+L%2E%2C+Lin%2C+Y%2E%2C+Chen%2C+D%2E%2C+Ren%2C+S%2E%2C+Li%2C+P%2E%2C+Zhou%2C+J%2E%2C+and+Sun%2C+X%2E+%282021%29%2E+Cascadebert%3A+Accelerating+inference+of+pre%2Dtrained+language+models+via+calibrated+complete+models+cascade%2E+13)
124. Li, L. H., Hessel, J., Yu, Y., Ren, X., Chang, K.-W., and Choi, Y. (2023b). [Symbolic chain-of-thought distillation: Small models can also” think” step-by-step](https://md.research.elamur.ai/free/en/papers/li-2023-symbolic-chain-thought-distillation-small/). arXiv preprint arXiv:2306.14050. 12
125. Li, Q., Zhang, Y., Li, L., Yao, P., Zhang, B., Chu, X., Sun, Y., Du, L., and Xie, Y. (2023c). Fptq: Fine-grained post-training quantization for large language models. arXiv preprint arXiv:2308.15987. 9 — [arXiv:2308.15987](https://arxiv.org/abs/2308.15987) [find](https://scholar.google.com/scholar?q=Li%2C+Q%2E%2C+Zhang%2C+Y%2E%2C+Li%2C+L%2E%2C+Yao%2C+P%2E%2C+Zhang%2C+B%2E%2C+Chu%2C+X%2E%2C+Sun%2C+Y%2E%2C+Du%2C+L%2E%2C+and+Xie%2C+Y%2E+%282023c%29%2E+Fptq%3A+Fine%2Dgrained+post%2Dtraining+quantization+for+large+language+models%2E+arXiv+preprint+arXiv%3A2308%2E15987%2E+)
126. Li, Y., Wei, F., Zhang, C., and Zhang, H. (2024). [Eagle: Speculative sampling requires rethinking feature uncertainty](https://md.research.elamur.ai/free/en/papers/li-2024-eagle-speculative-sampling-requires-rethinking/). arXiv preprint arXiv:2401.15077. 18
127. Li, Y., Yu, Y., Liang, C., He, P., Karampatziakis, N., Chen, W., and Zhao, T. (2023d). Loftq: Lora-fine-tuning-aware quantization for large language models. arXiv preprint arXiv:2310.08659. 10 — [arXiv:2310.08659](https://arxiv.org/abs/2310.08659) [find](https://scholar.google.com/scholar?q=Li%2C+Y%2E%2C+Yu%2C+Y%2E%2C+Liang%2C+C%2E%2C+He%2C+P%2E%2C+Karampatziakis%2C+N%2E%2C+Chen%2C+W%2E%2C+and+Zhao%2C+T%2E+%282023d%29%2E+Loftq%3A+Lora%2Dfine%2Dtuning%2Daware+quantization+for+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E08659%2E+10+%E2%80%94+%5BarXi)
128. Li, Z., Lin, Z., He, D., Tian, F., Qin, T., Wang, L., and Liu, T.-Y. (2019). Hint-based training for non-autoregressive machine translation. arXiv preprint arXiv:1909.06708. 19 [find](https://scholar.google.com/scholar?q=Li%2C+Z%2E%2C+Lin%2C+Z%2E%2C+He%2C+D%2E%2C+Tian%2C+F%2E%2C+Qin%2C+T%2E%2C+Wang%2C+L%2E%2C+and+Liu%2C+T%2E%2DY%2E+%282019%29%2E+Hint%2Dbased+training+for+non%2Dautoregressive+machine+translation%2E+arXiv+preprint+arXiv%3A1909%2E06708%2E+19)
129. Li, Z., Liu, X., Zhu, B., Dong, Z., Gu, Q., and Keutzer, K. (2023e). Qft: Quantized full-parameter tuning of llms with affordable resources. arXiv preprint arXiv:2310.07147. 10 [find](https://scholar.google.com/scholar?q=Li%2C+Z%2E%2C+Liu%2C+X%2E%2C+Zhu%2C+B%2E%2C+Dong%2C+Z%2E%2C+Gu%2C+Q%2E%2C+and+Keutzer%2C+K%2E+%282023e%29%2E+Qft%3A+Quantized+full%2Dparameter+tuning+of+llms+with+affordable+resources%2E+arXiv+preprint+arXiv%3A2310%2E07147%2E+10)
130. Liang, C., Jiang, H., Li, Z., Tang, X., Yin, B., and Zhao, T. (2023a). Homodistil: Homotopic task-agnostic distillation of pre-trained transformers. arXiv preprint arXiv:2302.09632. 12 — [arXiv:2302.09632](https://arxiv.org/abs/2302.09632) [find](https://scholar.google.com/scholar?q=Liang%2C+C%2E%2C+Jiang%2C+H%2E%2C+Li%2C+Z%2E%2C+Tang%2C+X%2E%2C+Yin%2C+B%2E%2C+and+Zhao%2C+T%2E+%282023a%29%2E+Homodistil%3A+Homotopic+task%2Dagnostic+distillation+of+pre%2Dtrained+transformers%2E+arXiv+preprint+arXiv%3A2302%2E09632%2E+12+%E2%80%94+%5BarXiv%3A2302%2E0)
131. Liang, C., Zuo, S., Zhang, Q., He, P., Chen, W., and Zhao, T. (2023b). Less is more: Task-aware layer-wise distillation for language model compression. In International Conference on Machine Learning, pages 20852–20867. PMLR. 12 [find](https://scholar.google.com/scholar?q=Liang%2C+C%2E%2C+Zuo%2C+S%2E%2C+Zhang%2C+Q%2E%2C+He%2C+P%2E%2C+Chen%2C+W%2E%2C+and+Zhao%2C+T%2E+%282023b%29%2E+Less+is+more%3A+Task%2Daware+layer%2Dwise+distillation+for+language+model+compression%2E+In+International+Conference+on+Machine+Learning%2C)
132. Liang, T., Glossner, J., Wang, L., Shi, S., and Zhang, X. (2021). Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing, 461:370–403. 11 [find](https://scholar.google.com/scholar?q=Liang%2C+T%2E%2C+Glossner%2C+J%2E%2C+Wang%2C+L%2E%2C+Shi%2C+S%2E%2C+and+Zhang%2C+X%2E+%282021%29%2E+Pruning+and+quantization+for+deep+neural+network+acceleration%3A+A+survey%2E+Neurocomputing%2C+461%3A370%E2%80%93403%2E+11)
133. Lin, B., Tang, Z., Ye, Y., Cui, J., Zhu, B., Jin, P., Zhang, J., Ning, M., and Yuan, L. (2024). Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947. 26 — [arXiv:2401.15947](https://arxiv.org/abs/2401.15947) [find](https://scholar.google.com/scholar?q=Lin%2C+B%2E%2C+Tang%2C+Z%2E%2C+Ye%2C+Y%2E%2C+Cui%2C+J%2E%2C+Zhu%2C+B%2E%2C+Jin%2C+P%2E%2C+Zhang%2C+J%2E%2C+Ning%2C+M%2E%2C+and+Yuan%2C+L%2E+%282024%29%2E+Moe%2Dllava%3A+Mixture+of+experts+for+large+vision%2Dlanguage+models%2E+arXiv+preprint+arXiv%3A2401%2E15947%2E+26+%E2%80%94+%5Ba)
134. Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., and Han, S. (2023). Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978. 8 [find](https://scholar.google.com/scholar?q=Lin%2C+J%2E%2C+Tang%2C+J%2E%2C+Tang%2C+H%2E%2C+Yang%2C+S%2E%2C+Dang%2C+X%2E%2C+and+Han%2C+S%2E+%282023%29%2E+Awq%3A+Activation%2Daware+weight+quantization+for+llm+compression+and+acceleration%2E+arXiv+preprint+arXiv%3A2306%2E00978%2E+8)
135. Liu, M., Zeng, A., Wang, B., Zhang, P., Tang, J., and Dong, Y. (2024a). Apar: Llms can do auto-parallel auto-regressive decoding. 19 [find](https://scholar.google.com/scholar?q=Liu%2C+M%2E%2C+Zeng%2C+A%2E%2C+Wang%2C+B%2E%2C+Zhang%2C+P%2E%2C+Tang%2C+J%2E%2C+and+Dong%2C+Y%2E+%282024a%29%2E+Apar%3A+Llms+can+do+auto%2Dparallel+auto%2Dregressive+decoding%2E+19)
136. Liu, W., Zhou, P., Zhao, Z., Wang, Z., Deng, H., and Ju, Q. (2020). [Fastbert: a self-distilling bert with adaptive inference time](https://md.research.elamur.ai/free/en/papers/liu-2020-fastbert-self-distilling-bert-adaptive/). 13
137. Liu, X., Hu, L., Bailis, P., Stoica, I., Deng, Z., Cheung, A., and Zhang, H. (2023a). [Online speculative decoding](https://md.research.elamur.ai/free/en/papers/liu-2023-online-speculative-decoding/). arXiv preprint arXiv:2310.07177. 18
138. Liu, X., Sun, T., He, J., Wu, J., Wu, L., Zhang, X., Jiang, H., Cao, Z., Huang, X., and Qiu, X. (2022). Towards efficient NLP: A standard evaluation and a strong baseline. In Carpuat, M., de Marneffe, M.-C., and Meza Ruiz, I. V., editors, Proceedings of the 2022 Conference of the North American Chap… [find](https://scholar.google.com/scholar?q=Liu%2C+X%2E%2C+Sun%2C+T%2E%2C+He%2C+J%2E%2C+Wu%2C+J%2E%2C+Wu%2C+L%2E%2C+Zhang%2C+X%2E%2C+Jiang%2C+H%2E%2C+Cao%2C+Z%2E%2C+Huang%2C+X%2E%2C+and+Qiu%2C+X%2E+%282022%29%2E+Towards+efficient+NLP%3A+A+standard+evaluation+and+a+strong+baseline%2E+In+Carpuat%2C+M%2E%2C+de+Marneffe%2C)
139. Liu, Z., Oguz, B., Zhao, C., Chang, E., Stock, P., Mehdad, Y., Shi, Y., Krishnamoorthi, R., and Chandra, V. (2023b). [Llm-qat: Data-free quantization aware training for large language models](https://md.research.elamur.ai/free/en/papers/liu-2024-llm-qat-data-free-quantization/). arXiv preprint arXiv:2305.17888. 8, 24
140. Liu, Z., Wang, J., Dao, T., Zhou, T., Yuan, B., Song, Z., Shrivastava, A., Zhang, C., Tian, Y., Re, C., and Chen, B. (2023c). [Deja vu: Contextual sparsity for efficient llms at inference time](https://md.research.elamur.ai/free/en/papers/liu-2023-deja-vu-contextual-sparsity-efficient/). 15
141. Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. (2024b). [Kivi: A tuning-free asymmetric 2bit quantization for kv cache](https://md.research.elamur.ai/free/en/papers/liu-2024-kivi-tuning-free-asymmetric-2bit/). arXiv preprint arXiv:2402.02750. 9
142. Ma, H., Zhu, Y., Zhang, C., Zhao, P., Wu, B., Huang, L.-K., Hu, Q., and Wu, B. (2024). Invariant test-time adaptation for vision-language model generalization. arXiv preprint arXiv:2403.00376. 26 — [arXiv:2403.00376](https://arxiv.org/abs/2403.00376) [find](https://scholar.google.com/scholar?q=Ma%2C+H%2E%2C+Zhu%2C+Y%2E%2C+Zhang%2C+C%2E%2C+Zhao%2C+P%2E%2C+Wu%2C+B%2E%2C+Huang%2C+L%2E%2DK%2E%2C+Hu%2C+Q%2E%2C+and+Wu%2C+B%2E+%282024%29%2E+Invariant+test%2Dtime+adaptation+for+vision%2Dlanguage+model+generalization%2E+arXiv+preprint+arXiv%3A2403%2E00376%2E+26+%E2%80%94+%5Ba)
143. Ma, X., Fang, G., and Wang, X. (2023). [Llm-pruner: On the structural pruning of large language models](https://md.research.elamur.ai/free/en/papers/ma-2023-llm-pruner-structural-pruning-large/). arXiv preprint arXiv:2305.11627. 11
144. Meng, K., Bau, D., Andonian, A., and Belinkov, Y. (2022). Locating and editing factual associations in GPT. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A., editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Sy… [find](https://scholar.google.com/scholar?q=Meng%2C+K%2E%2C+Bau%2C+D%2E%2C+Andonian%2C+A%2E%2C+and+Belinkov%2C+Y%2E+%282022%29%2E+Locating+and+editing+factual+associations+in+GPT%2E+In+Koyejo%2C+S%2E%2C+Mohamed%2C+S%2E%2C+Agarwal%2C+A%2E%2C+Belgrave%2C+D%2E%2C+Cho%2C+K%2E%2C+and+Oh%2C+A%2E%2C+editors%2C+Advance)
145. Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Jin, H., Chen, T., and Jia, Z. (2023a). [Towards efficient generative large language model serving: A survey from algorithms to systems](https://md.research.elamur.ai/free/en/papers/miao-2023-towards-efficient-generative-large-language/). arXiv preprint arXiv:2312.15234. 2
146. Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Wong, R. Y. Y., Chen, Z., Arfeen, D., Abhyankar, R., and Jia, Z. (2023b). [Specinfer: Accelerating generative llm serving with speculative inference and token tree verification](https://md.research.elamur.ai/free/en/papers/miao-2023-specinfer-accelerating-generative-large-language/). arXiv preprint arXiv:2305.09781. 17
147. Micikevicius, P., Stosic, D., Burgess, N., Cornea, M., Dubey, P., Grisenthwaite, R., Ha, S., Heinecke, A., Judd, P., Kamalu, J., et al. (2022). Fp8 formats for deep learning. arXiv preprint arXiv:2209.05433. 24 [find](https://scholar.google.com/scholar?q=Micikevicius%2C+P%2E%2C+Stosic%2C+D%2E%2C+Burgess%2C+N%2E%2C+Cornea%2C+M%2E%2C+Dubey%2C+P%2E%2C+Grisenthwaite%2C+R%2E%2C+Ha%2C+S%2E%2C+Heinecke%2C+A%2E%2C+Judd%2C+P%2E%2C+Kamalu%2C+J%2E%2C+et+al%2E+%282022%29%2E+Fp8+formats+for+deep+learning%2E+arXiv+preprint+arXiv%3A2209)
148. Modarressi, A., Imani, A., Fayyaz, M., and Schütze, H. (2023). Ret-llm: Towards a general read-write memory for large language models. 27 [find](https://scholar.google.com/scholar?q=Modarressi%2C+A%2E%2C+Imani%2C+A%2E%2C+Fayyaz%2C+M%2E%2C+and+Schu%CC%88tze%2C+H%2E+%282023%29%2E+Ret%2Dllm%3A+Towards+a+general+read%2Dwrite+memory+for+large+language+models%2E+27)
149. ModelTC (2024). Lightllm: A python-based llm inference and serving framework. [https://github.com/ModelTC/lightllm](https://github.com/ModelTC/lightllm). 22 [↗](https://github.com/ModelTC/lightllm)
150. Mohtashami, A. and Jaggi, M. (2023). [Landmark attention: Random-access infinite context length for transformers](https://md.research.elamur.ai/free/en/papers/mohtashami-2023-landmark-attention-random-access-infinite/). arXiv preprint arXiv:2305.16300. 27
151. Monea, G., Joulin, A., and Grave, E. (2023). Pass: Parallel speculative sampling. arXiv preprint arXiv:2311.13581. 18 — [arXiv:2311.13581](https://arxiv.org/abs/2311.13581) [find](https://scholar.google.com/scholar?q=Monea%2C+G%2E%2C+Joulin%2C+A%2E%2C+and+Grave%2C+E%2E+%282023%29%2E+Pass%3A+Parallel+speculative+sampling%2E+arXiv+preprint+arXiv%3A2311%2E13581%2E+18+%E2%80%94+%5BarXiv%3A2311%2E13581%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2311%2E13581%29)
152. Nawrot, P., Łańcucki, A., Chochowski, M., Tarjan, D., and Ponti, E. M. (2024). Dynamic memory compression: Retrofitting llms for accelerated inference. arXiv preprint arXiv:2403.09636. 27 [find](https://scholar.google.com/scholar?q=Nawrot%2C+P%2E%2C+%C5%81an%CC%81cucki%2C+A%2E%2C+Chochowski%2C+M%2E%2C+Tarjan%2C+D%2E%2C+and+Ponti%2C+E%2E+M%2E+%282024%29%2E+Dynamic+memory+compression%3A+Retrofitting+llms+for+accelerated+inference%2E+arXiv+preprint+arXiv%3A2403%2E09636%2E+27)
153. Ning, X., Lin, Z., Zhou, Z., Wang, Z., Yang, H., and Wang, Y. (2023). Skeleton-of-thought: Large language models can do parallel decoding. 19 [find](https://scholar.google.com/scholar?q=Ning%2C+X%2E%2C+Lin%2C+Z%2E%2C+Zhou%2C+Z%2E%2C+Wang%2C+Z%2E%2C+Yang%2C+H%2E%2C+and+Wang%2C+Y%2E+%282023%29%2E+Skeleton%2Dof%2Dthought%3A+Large+language+models+can+do+parallel+decoding%2E+19)
154. NVIDIA (2022). Inside the nvidia hopper architecture. [https://www.nvidia.com/en-us/data-center/technologies/hopper-architecture/](https://www.nvidia.com/en-us/data-center/technologies/hopper-architecture/). Accessed on March 11, 2024. 24 [↗](https://www.nvidia.com/en-us/data-center/technologies/hopper-architecture/)
155. Ortega, J. M. and Rheinboldt, W. C. (2000). Iterative solution of nonlinear equations in several variables. SIAM. 19 [find](https://scholar.google.com/scholar?q=Ortega%2C+J%2E+M%2E+and+Rheinboldt%2C+W%2E+C%2E+%282000%29%2E+Iterative+solution+of+nonlinear+equations+in+several+variables%2E+SIAM%2E+19)
156. Oseledets, I. V. (2011). Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5):2295–2317. 13 [find](https://scholar.google.com/scholar?q=Oseledets%2C+I%2E+V%2E+%282011%29%2E+Tensor%2Dtrain+decomposition%2E+SIAM+Journal+on+Scientific+Computing%2C+33%285%29%3A2295%E2%80%932317%2E+13)
157. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022). [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744. 1, 25
158. Pal, A., Karkhanis, D., Roberts, M., Dooley, S., Sundararajan, A., and Naidu, S. (2023). Giraffe: Adventures in expanding context lengths in llms. 28 [find](https://scholar.google.com/scholar?q=Pal%2C+A%2E%2C+Karkhanis%2C+D%2E%2C+Roberts%2C+M%2E%2C+Dooley%2C+S%2E%2C+Sundararajan%2C+A%2E%2C+and+Naidu%2C+S%2E+%282023%29%2E+Giraffe%3A+Adventures+in+expanding+context+lengths+in+llms%2E+28)
159. Pandya, K. and Holia, M. (2023). Automating customer service using langchain: Building custom open-source gpt chatbot for organizations. arXiv preprint arXiv:2310.05421. 27 — [arXiv:2310.05421](https://arxiv.org/abs/2310.05421) [find](https://scholar.google.com/scholar?q=Pandya%2C+K%2E+and+Holia%2C+M%2E+%282023%29%2E+Automating+customer+service+using+langchain%3A+Building+custom+open%2Dsource+gpt+chatbot+for+organizations%2E+arXiv+preprint+arXiv%3A2310%2E05421%2E+27+%E2%80%94+%5BarXiv%3A2310%2E05421%5D%28https%3A)
160. Park, E. et al. (2018). Energy-efficient neural network accelerator based on outlier-aware low-precision computation. In ISCA, pages 688–698. IEEE. 24 [find](https://scholar.google.com/scholar?q=Park%2C+E%2E+et+al%2E+%282018%29%2E+Energy%2Defficient+neural+network+accelerator+based+on+outlier%2Daware+low%2Dprecision+computation%2E+In+ISCA%2C+pages+688%E2%80%93698%2E+IEEE%2E+24)
161. Park, G., Park, B., Kim, M., Lee, S., Kim, J., Kwon, B., Kwon, S. J., Kim, B., Lee, Y., and Lee, D. (2023). [Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models](https://md.research.elamur.ai/free/en/papers/park-2022-lut-gemm-quantized-matrix-multiplication/). arXiv preprint arXiv:2206.09557. 8
162. Peng, B., Quesnelle, J., Fan, H., and Shippole, E. (2023). Yarn: Efficient context window extension of large language models. 28 [find](https://scholar.google.com/scholar?q=Peng%2C+B%2E%2C+Quesnelle%2C+J%2E%2C+Fan%2C+H%2E%2C+and+Shippole%2C+E%2E+%282023%29%2E+Yarn%3A+Efficient+context+window+extension+of+large+language+models%2E+28)
163. Press, O., Smith, N. A., and Lewis, M. (2021). Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409. 28 [find](https://scholar.google.com/scholar?q=Press%2C+O%2E%2C+Smith%2C+N%2E+A%2E%2C+and+Lewis%2C+M%2E+%282021%29%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2E+arXiv+preprint+arXiv%3A2108%2E12409%2E+28)
164. Qin, Y., Wang, Y., Deng, D., Zhao, Z., Yang, X., Liu, L., Wei, S., Hu, Y., and Yin, S. (2023). Fact: Ffn-attention co-optimized transformer architecture with eager correlation prediction. In Proceedings of the 50th Annual International Symposium on Computer Architecture, pages 1–14. 25 [find](https://scholar.google.com/scholar?q=Qin%2C+Y%2E%2C+Wang%2C+Y%2E%2C+Deng%2C+D%2E%2C+Zhao%2C+Z%2E%2C+Yang%2C+X%2E%2C+Liu%2C+L%2E%2C+Wei%2C+S%2E%2C+Hu%2C+Y%2E%2C+and+Yin%2C+S%2E+%282023%29%2E+Fact%3A+Ffn%2Dattention+co%2Doptimized+transformer+architecture+with+eager+correlation+prediction%2E+In+Proceedin)
165. Rae, J. W., Potapenko, A., Jayakumar, S. M., and Lillicrap, T. P. (2019). [Compressive transformers for long-range sequence modelling](https://md.research.elamur.ai/free/en/papers/rae-2019-compressive-transformers-long-range-sequence/). 27
166. Rajbhandari, S., Li, C., Yao, Z., Zhang, M., Aminabadi, R. Y., Awan, A. A., Rasley, J., and He, Y. (2022). [Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale](https://md.research.elamur.ai/free/en/papers/rajbhandari-2022-deepspeed-moe-advancing-mixture-experts/). In International Conference on Machine Learning, pages 18332–18346. PMLR. 16
167. Rotem, D., Hassid, M., Mamou, J., and Schwartz, R. (2023). Finding the sweet spot: Analysis and improvement of adaptive inference in low resource settings. 15 [find](https://scholar.google.com/scholar?q=Rotem%2C+D%2E%2C+Hassid%2C+M%2E%2C+Mamou%2C+J%2E%2C+and+Schwartz%2C+R%2E+%282023%29%2E+Finding+the+sweet+spot%3A+Analysis+and+improvement+of+adaptive+inference+in+low+resource+settings%2E+15)
168. Sahu, G., Vechtomova, O., Bahdanau, D., and Laradji, I. H. (2023). Promptmix: A class boundary augmentation method for large language model distillation. arXiv preprint arXiv:2310.14192. 12 [find](https://scholar.google.com/scholar?q=Sahu%2C+G%2E%2C+Vechtomova%2C+O%2E%2C+Bahdanau%2C+D%2E%2C+and+Laradji%2C+I%2E+H%2E+%282023%29%2E+Promptmix%3A+A+class+boundary+augmentation+method+for+large+language+model+distillation%2E+arXiv+preprint+arXiv%3A2310%2E14192%2E+12)
169. Santilli, A., Severino, S., Postolache, E., Maiorca, V., Mancusi, M., Marin, R., and Rodolà, E. (2023). Accelerating transformer inference for translation via parallel decoding. arXiv preprint arXiv:2305.10427. 19 [find](https://scholar.google.com/scholar?q=Santilli%2C+A%2E%2C+Severino%2C+S%2E%2C+Postolache%2C+E%2E%2C+Maiorca%2C+V%2E%2C+Mancusi%2C+M%2E%2C+Marin%2C+R%2E%2C+and+Rodola%CC%80%2C+E%2E+%282023%29%2E+Accelerating+transformer+inference+for+translation+via+parallel+decoding%2E+arXiv+preprint+arXiv%3A)
170. Savinov, N., Chung, J., Binkowski, M., Elsen, E., and Oord, A. v. d. (2021). Step-unrolled denoising autoencoders for text generation. arXiv preprint arXiv:2112.06749. 19 — [arXiv:2112.06749](https://arxiv.org/abs/2112.06749) [find](https://scholar.google.com/scholar?q=Savinov%2C+N%2E%2C+Chung%2C+J%2E%2C+Binkowski%2C+M%2E%2C+Elsen%2C+E%2E%2C+and+Oord%2C+A%2E+v%2E+d%2E+%282021%29%2E+Step%2Dunrolled+denoising+autoencoders+for+text+generation%2E+arXiv+preprint+arXiv%3A2112%2E06749%2E+19+%E2%80%94+%5BarXiv%3A2112%2E06749%5D%28https%3A%2F%2F)
171. Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., et al. (2022). Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. 1 [find](https://scholar.google.com/scholar?q=Scao%2C+T%2E+L%2E%2C+Fan%2C+A%2E%2C+Akiki%2C+C%2E%2C+Pavlick%2C+E%2E%2C+Ilic%CC%81%2C+S%2E%2C+Hesslow%2C+D%2E%2C+Castagne%CC%81%2C+R%2E%2C+Luccioni%2C+A%2E+S%2E%2C+Yvon%2C+F%2E%2C+Galle%CC%81%2C+M%2E%2C+et+al%2E+%282022%29%2E+Bloom%3A+A+176b%2Dparameter+open%2Daccess+multilingual+language+mod)
172. Schotthöfer, S., Zangrando, E., Kusch, J., Ceruti, G., and Tudisco, F. (2022). Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations. Advances in Neural Information Processing Systems, 35:20051–20063. 12 [find](https://scholar.google.com/scholar?q=Schottho%CC%88fer%2C+S%2E%2C+Zangrando%2C+E%2E%2C+Kusch%2C+J%2E%2C+Ceruti%2C+G%2E%2C+and+Tudisco%2C+F%2E+%282022%29%2E+Low%2Drank+lottery+tickets%3A+finding+efficient+low%2Drank+neural+networks+via+matrix+differential+equations%2E+Advances+in+Neur)
173. Schuster, T., Fisch, A., Gupta, J., Dehghani, M., Bahri, D., Tran, V. Q., Tay, Y., and Metzler, D. (2022). Confident adaptive language modeling. 13, 14, 15 [find](https://scholar.google.com/scholar?q=Schuster%2C+T%2E%2C+Fisch%2C+A%2E%2C+Gupta%2C+J%2E%2C+Dehghani%2C+M%2E%2C+Bahri%2C+D%2E%2C+Tran%2C+V%2E+Q%2E%2C+Tay%2C+Y%2E%2C+and+Metzler%2C+D%2E+%282022%29%2E+Confident+adaptive+language+modeling%2E+13%2C+14%2C+15)
174. Schuster, T., Fisch, A., Jaakkola, T., and Barzilay, R. (2021). Consistent accelerated inference via confident adaptive transformers. 13 [find](https://scholar.google.com/scholar?q=Schuster%2C+T%2E%2C+Fisch%2C+A%2E%2C+Jaakkola%2C+T%2E%2C+and+Barzilay%2C+R%2E+%282021%29%2E+Consistent+accelerated+inference+via+confident+adaptive+transformers%2E+13)
175. Schwartz, R., Stanovsky, G., Swayamdipta, S., Dodge, J., and Smith, N. A. (2020). [The right tool for the job: Matching model and instance complexities](https://md.research.elamur.ai/free/en/papers/schwartz-2020-right-tool-job-matching-model/). In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J., editors, Proceedings of the 58th Annual Meeting of the Association for Computational Lin…
176. Shang, Y., Duan, B., Zong, Z., Nie, L., and Yan, Y. (2021). Lipschitz continuity guided knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10675–10684. 11 [find](https://scholar.google.com/scholar?q=Shang%2C+Y%2E%2C+Duan%2C+B%2E%2C+Zong%2C+Z%2E%2C+Nie%2C+L%2E%2C+and+Yan%2C+Y%2E+%282021%29%2E+Lipschitz+continuity+guided+knowledge+distillation%2E+In+Proceedings+of+the+IEEE%2FCVF+International+Conference+on+Computer+Vision%2C+pages+10675%E2%80%93)
177. Shang, Y., Yuan, Z., and Dong, Z. (2024). Pb-llm: Partially binarized large language models. In ICLR. 8, 9 [find](https://scholar.google.com/scholar?q=Shang%2C+Y%2E%2C+Yuan%2C+Z%2E%2C+and+Dong%2C+Z%2E+%282024%29%2E+Pb%2Dllm%3A+Partially+binarized+large+language+models%2E+In+ICLR%2E+8%2C+9)
178. Shao, C., Zhang, J., Feng, Y., Meng, F., and Zhou, J. (2020). Minimizing the bag-of-ngrams difference for non-autoregressive neural machine translation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 198–205. 19 [find](https://scholar.google.com/scholar?q=Shao%2C+C%2E%2C+Zhang%2C+J%2E%2C+Feng%2C+Y%2E%2C+Meng%2C+F%2E%2C+and+Zhou%2C+J%2E+%282020%29%2E+Minimizing+the+bag%2Dof%2Dngrams+difference+for+non%2Dautoregressive+neural+machine+translation%2E+In+Proceedings+of+the+AAAI+conference+on+artifi)
179. Sharma, P., Ash, J. T., and Misra, D. (2023). The truth is in there: Improving reasoning in language models with layer-selective rank reduction. arXiv preprint arXiv:2312.13558. 12 — [arXiv:2312.13558](https://arxiv.org/abs/2312.13558) [find](https://scholar.google.com/scholar?q=Sharma%2C+P%2E%2C+Ash%2C+J%2E+T%2E%2C+and+Misra%2C+D%2E+%282023%29%2E+The+truth+is+in+there%3A+Improving+reasoning+in+language+models+with+layer%2Dselective+rank+reduction%2E+arXiv+preprint+arXiv%3A2312%2E13558%2E+12+%E2%80%94+%5BarXiv%3A2312%2E13558)
180. Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., Ré, C., Stoica, I., and Zhang, C. (2023). [Flexgen: High-throughput generative inference of large language models with a single GPU](https://md.research.elamur.ai/free/en/papers/sheng-2023-flexgen-high-throughput-generative-inference/). In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J., editors…
181. Simoulin, A. and Crabbé, B. (2021). How many layers and why? An analysis of the model depth in transformers. In Kabbara, J., Lin, H., Paullada, A., and Vamvas, J., editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confe… [find](https://scholar.google.com/scholar?q=Simoulin%2C+A%2E+and+Crabbe%CC%81%2C+B%2E+%282021%29%2E+How+many+layers+and+why%3F+An+analysis+of+the+model+depth+in+transformers%2E+In+Kabbara%2C+J%2E%2C+Lin%2C+H%2E%2C+Paullada%2C+A%2E%2C+and+Vamvas%2C+J%2E%2C+editors%2C+Proceedings+of+the+59th+An)
182. Song, Y., Meng, C., Liao, R., and Ermon, S. (2021). Accelerating feedforward computation via parallel nonlinear equation solving. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9… [find](https://scholar.google.com/scholar?q=Song%2C+Y%2E%2C+Meng%2C+C%2E%2C+Liao%2C+R%2E%2C+and+Ermon%2C+S%2E+%282021%29%2E+Accelerating+feedforward+computation+via+parallel+nonlinear+equation+solving%2E+In+Meila%2C+M%2E+and+Zhang%2C+T%2E%2C+editors%2C+Proceedings+of+the+38th+Internati)
183. Song, Y., Mi, Z., Xie, H., and Chen, H. (2023). [Powerinfer: Fast large language model serving with a consumer-grade gpu](https://md.research.elamur.ai/free/en/papers/song-2023-powerinfer-fast-large-language-model/). 15
184. Stern, M., Shazeer, N., and Uszkoreit, J. (2018). Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems, 31. 18 [find](https://scholar.google.com/scholar?q=Stern%2C+M%2E%2C+Shazeer%2C+N%2E%2C+and+Uszkoreit%2C+J%2E+%282018%29%2E+Blockwise+parallel+decoding+for+deep+autoregressive+models%2E+Advances+in+Neural+Information+Processing+Systems%2C+31%2E+18)
185. Stickland, A. C. and Murray, I. (2019). Bert and pals: Projected attention layers for efficient adaptation in multi-task learning. 13 [find](https://scholar.google.com/scholar?q=Stickland%2C+A%2E+C%2E+and+Murray%2C+I%2E+%282019%29%2E+Bert+and+pals%3A+Projected+attention+layers+for+efficient+adaptation+in+multi%2Dtask+learning%2E+13)
186. Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. (2023a). A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695. 11 [find](https://scholar.google.com/scholar?q=Sun%2C+M%2E%2C+Liu%2C+Z%2E%2C+Bair%2C+A%2E%2C+and+Kolter%2C+J%2E+Z%2E+%282023a%29%2E+A+simple+and+effective+pruning+approach+for+large+language+models%2E+arXiv+preprint+arXiv%3A2306%2E11695%2E+11)
187. Sun, X., Choi, J., Chen, C.-Y., Wang, N., Venkataramani, S., Srinivasan, V. V., Cui, X., Zhang, W., and Gopalakrishnan, K. (2019a). Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. Advances in neural information processing systems, 32. 24 [find](https://scholar.google.com/scholar?q=Sun%2C+X%2E%2C+Choi%2C+J%2E%2C+Chen%2C+C%2E%2DY%2E%2C+Wang%2C+N%2E%2C+Venkataramani%2C+S%2E%2C+Srinivasan%2C+V%2E+V%2E%2C+Cui%2C+X%2E%2C+Zhang%2C+W%2E%2C+and+Gopalakrishnan%2C+K%2E+%282019a%29%2E+Hybrid+8%2Dbit+floating+point+%28hfp8%29+training+and+inference+for+deep+n)
188. Sun, Y., Dong, L., Patra, B., Ma, S., Huang, S., Benhaim, A., Chaudhary, V., Song, X., and Wei, F. (2022). [A length-extrapolatable transformer](https://md.research.elamur.ai/free/en/papers/sun-2023-length-extrapolatable-transformer/). 28
189. Sun, Z., Li, Z., Wang, H., He, D., Lin, Z., and Deng, Z. (2019b). Fast structured decoding for sequence models. Advances in Neural Information Processing Systems, 32. 19 [find](https://scholar.google.com/scholar?q=Sun%2C+Z%2E%2C+Li%2C+Z%2E%2C+Wang%2C+H%2E%2C+He%2C+D%2E%2C+Lin%2C+Z%2E%2C+and+Deng%2C+Z%2E+%282019b%29%2E+Fast+structured+decoding+for+sequence+models%2E+Advances+in+Neural+Information+Processing+Systems%2C+32%2E+19)
190. Sun, Z., Suresh, A. T., Ro, J. H., Beirami, A., Jain, H., and Yu, F. (2023b). Spectr: Fast speculative decoding via optimal transport. arXiv preprint arXiv:2310.15141. 17 — [arXiv:2310.15141](https://arxiv.org/abs/2310.15141) [find](https://scholar.google.com/scholar?q=Sun%2C+Z%2E%2C+Suresh%2C+A%2E+T%2E%2C+Ro%2C+J%2E+H%2E%2C+Beirami%2C+A%2E%2C+Jain%2C+H%2E%2C+and+Yu%2C+F%2E+%282023b%29%2E+Spectr%3A+Fast+speculative+decoding+via+optimal+transport%2E+arXiv+preprint+arXiv%3A2310%2E15141%2E+17+%E2%80%94+%5BarXiv%3A2310%2E15141%5D%28https%3A%2F%2F)
191. Sundar Pichai, D. H. (2024). Our next-generation model: Gemini 1.5. 9 [find](https://scholar.google.com/scholar?q=Sundar+Pichai%2C+D%2E+H%2E+%282024%29%2E+Our+next%2Dgeneration+model%3A+Gemini+1%2E5%2E+9)
192. Tambe, T., Yang, E.-Y., Wan, Z., Deng, Y., Reddi, V. J., Rush, A., Brooks, D., and Wei, G.-Y. (2020). Algorithm-hardware co-design of adaptive floating-point encodings for resilient deep learning inference. In 2020 57th ACM/IEEE Design Automation Conference (DAC), pages 1–6. IEEE. 25 [find](https://scholar.google.com/scholar?q=Tambe%2C+T%2E%2C+Yang%2C+E%2E%2DY%2E%2C+Wan%2C+Z%2E%2C+Deng%2C+Y%2E%2C+Reddi%2C+V%2E+J%2E%2C+Rush%2C+A%2E%2C+Brooks%2C+D%2E%2C+and+Wei%2C+G%2E%2DY%2E+%282020%29%2E+Algorithm%2Dhardware+co%2Ddesign+of+adaptive+floating%2Dpoint+encodings+for+resilient+deep+learning+infe)
193. Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. (2023). [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805. 26
194. Tesla (2023). Tesla dojo technology: A guide to tesla’s configurable floating point formats & arithmetic. [https://cdn.motor1.com/pdf-files/535242876-tesla-dojo-technology.pdf/](https://cdn.motor1.com/pdf-files/535242876-tesla-dojo-technology.pdf/). 24 [↗](https://cdn.motor1.com/pdf-files/535242876-tesla-dojo-technology.pdf/)
195. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023a). [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971. 1, 6
196. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023b). [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288. 1, 16
197. Vaidya, N., Oh, F., and Comly, N. (2023). Optimizing inference on large language models with nvidia tensorrt-llm, now publicly available. 20 [find](https://scholar.google.com/scholar?q=Vaidya%2C+N%2E%2C+Oh%2C+F%2E%2C+and+Comly%2C+N%2E+%282023%29%2E+Optimizing+inference+on+large+language+models+with+nvidia+tensorrt%2Dllm%2C+now+publicly+available%2E+20)
198. Valipour, M., Rezagholizadeh, M., Kobyzev, I., and Ghodsi, A. (2022). [Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation](https://md.research.elamur.ai/free/en/papers/valipour-2023-dylora-parameter-efficient-tuning-pre/). arXiv preprint arXiv:2210.07558. 10
199. Wang, P., Wang, Z., Li, Z., Gao, Y., Yin, B., and Ren, X. (2023). [Scott: Self-consistent chain-of-thought distillation](https://md.research.elamur.ai/free/en/papers/wang-2023-scott-self-consistent-chain-thought/). arXiv preprint arXiv:2305.01879. 12
200. Wang, W., Chen, W., Luo, Y., Long, Y., Lin, Z., Zhang, L., Lin, B., Cai, D., and He, X. (2024a). Model compression and efficient inference for [large language models: A survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/). arXiv preprint arXiv:2402.09748. 3 — [arXiv:2402.09748](https://arxiv.org/abs/2402.09748)
201. Wang, W., Dong, L., Cheng, H., Liu, X., Yan, X., Gao, J., and Wei, F. (2024b). Augmenting language models with long-term memory. Advances in Neural Information Processing Systems, 36. 27 [find](https://scholar.google.com/scholar?q=Wang%2C+W%2E%2C+Dong%2C+L%2E%2C+Cheng%2C+H%2E%2C+Liu%2C+X%2E%2C+Yan%2C+X%2E%2C+Gao%2C+J%2E%2C+and+Wei%2C+F%2E+%282024b%29%2E+Augmenting+language+models+with+long%2Dterm+memory%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%2E+27)
202. Wang, Y., Tian, F., He, D., Qin, T., Zhai, C., and Liu, T.-Y. (2019). Non-autoregressive machine translation with auxiliary regularization. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 5377–5384. 19 [find](https://scholar.google.com/scholar?q=Wang%2C+Y%2E%2C+Tian%2C+F%2E%2C+He%2C+D%2E%2C+Qin%2C+T%2E%2C+Zhai%2C+C%2E%2C+and+Liu%2C+T%2E%2DY%2E+%282019%29%2E+Non%2Dautoregressive+machine+translation+with+auxiliary+regularization%2E+In+Proceedings+of+the+AAAI+conference+on+artificial+intellig)
203. Wei, B., Wang, M., Zhou, H., Lin, J., Xie, J., and Sun, X. (2019). Imitation learning for non-autoregressive neural machine translation. arXiv preprint arXiv:1906.02041. 19 [find](https://scholar.google.com/scholar?q=Wei%2C+B%2E%2C+Wang%2C+M%2E%2C+Zhou%2C+H%2E%2C+Lin%2C+J%2E%2C+Xie%2C+J%2E%2C+and+Sun%2C+X%2E+%282019%29%2E+Imitation+learning+for+non%2Dautoregressive+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1906%2E02041%2E+19)
204. Wei, H., Kong, L., Chen, J., Zhao, L., Ge, Z., Yu, E., Sun, J., Han, C., and Zhang, X. (2024). Small language model meets with reinforced vision vocabulary. arXiv preprint arXiv:2401.12503. 26 [find](https://scholar.google.com/scholar?q=Wei%2C+H%2E%2C+Kong%2C+L%2E%2C+Chen%2C+J%2E%2C+Zhao%2C+L%2E%2C+Ge%2C+Z%2E%2C+Yu%2C+E%2E%2C+Sun%2C+J%2E%2C+Han%2C+C%2E%2C+and+Zhang%2C+X%2E+%282024%29%2E+Small+language+model+meets+with+reinforced+vision+vocabulary%2E+arXiv+preprint+arXiv%3A2401%2E12503%2E+26)
205. Wu, M., Waheed, A., Zhang, C., Abdul-Mageed, M., and Aji, A. F. (2023a). [Lamini-lm: A diverse herd of distilled models from large-scale instructions](https://md.research.elamur.ai/free/en/papers/wu-2023-lamini-lm-diverse-herd-distilled/). arXiv preprint arXiv:2304.14402. 12
206. Wu, Q., Lan, Z., Qian, K., Gu, J., Geramifard, A., and Yu, Z. (2020). Memformer: A memory-augmented transformer for sequence modeling. arXiv preprint arXiv:2010.06891. 27 [find](https://scholar.google.com/scholar?q=Wu%2C+Q%2E%2C+Lan%2C+Z%2E%2C+Qian%2C+K%2E%2C+Gu%2C+J%2E%2C+Geramifard%2C+A%2E%2C+and+Yu%2C+Z%2E+%282020%29%2E+Memformer%3A+A+memory%2Daugmented+transformer+for+sequence+modeling%2E+arXiv+preprint+arXiv%3A2010%2E06891%2E+27)
207. Wu, S., Chen, H., Quan, X., Wang, Q., and Wang, R. (2023b). Ad-kd: Attribution-driven knowledge distillation for language model compression. arXiv preprint arXiv:2305.10010. 12 [find](https://scholar.google.com/scholar?q=Wu%2C+S%2E%2C+Chen%2C+H%2E%2C+Quan%2C+X%2E%2C+Wang%2C+Q%2E%2C+and+Wang%2C+R%2E+%282023b%29%2E+Ad%2Dkd%3A+Attribution%2Ddriven+knowledge+distillation+for+language+model+compression%2E+arXiv+preprint+arXiv%3A2305%2E10010%2E+12)
208. Wu, X., Xia, H., Youn, S., Zheng, Z., Chen, S., Bakhtiari, A., Wyatt, M., He, Y., Ruwase, O., Song, L., et al. (2023c). Zeroquant (4+ 2): Redefining llms quantization with a new fp6-centric strategy for diverse generative tasks. arXiv preprint arXiv:2312.08583. 24 — [arXiv:2312.08583](https://arxiv.org/abs/2312.08583) [find](https://scholar.google.com/scholar?q=Wu%2C+X%2E%2C+Xia%2C+H%2E%2C+Youn%2C+S%2E%2C+Zheng%2C+Z%2E%2C+Chen%2C+S%2E%2C+Bakhtiari%2C+A%2E%2C+Wyatt%2C+M%2E%2C+He%2C+Y%2E%2C+Ruwase%2C+O%2E%2C+Song%2C+L%2E%2C+et+al%2E+%282023c%29%2E+Zeroquant+%284%2B+2%29%3A+Redefining+llms+quantization+with+a+new+fp6%2Dcentric+strategy+f)
209. Wu, X., Yao, Z., and He, Y. (2023d). Zeroquant-fp: A leap forward in llms post-training w4a8 quantization using floating-point formats. arXiv preprint arXiv:2307.09782. 9, 24 [find](https://scholar.google.com/scholar?q=Wu%2C+X%2E%2C+Yao%2C+Z%2E%2C+and+He%2C+Y%2E+%282023d%29%2E+Zeroquant%2Dfp%3A+A+leap+forward+in+llms+post%2Dtraining+w4a8+quantization+using+floating%2Dpoint+formats%2E+arXiv+preprint+arXiv%3A2307%2E09782%2E+9%2C+24)
210. Wu, Y., Rabe, M. N., Hutchins, D., and Szegedy, C. (2022). Memorizing transformers. arXiv preprint arXiv:2203.08913. 27 [find](https://scholar.google.com/scholar?q=Wu%2C+Y%2E%2C+Rabe%2C+M%2E+N%2E%2C+Hutchins%2C+D%2E%2C+and+Szegedy%2C+C%2E+%282022%29%2E+Memorizing+transformers%2E+arXiv+preprint+arXiv%3A2203%2E08913%2E+27)
211. Xia, H., Zheng, Z., Li, Y., Zhuang, D., Zhou, Z., Qiu, X., Li, Y., Lin, W., and Song, S. L. (2023a). Flash-llm: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity. arXiv preprint arXiv:2309.10285. 11 [find](https://scholar.google.com/scholar?q=Xia%2C+H%2E%2C+Zheng%2C+Z%2E%2C+Li%2C+Y%2E%2C+Zhuang%2C+D%2E%2C+Zhou%2C+Z%2E%2C+Qiu%2C+X%2E%2C+Li%2C+Y%2E%2C+Lin%2C+W%2E%2C+and+Song%2C+S%2E+L%2E+%282023a%29%2E+Flash%2Dllm%3A+Enabling+cost%2Deffective+and+highly%2Defficient+large+generative+model+inference+with+unstr)
212. Xia, H., Zheng, Z., Wu, X., Chen, S., Yao, Z., Youn, S., Bakhtiari, A., Wyatt, M., Zhuang, D., Zhou, Z., et al. (2024). Fp6-llm: Efficiently serving large language models through fp6-centric algorithm-system co-design. arXiv preprint arXiv:2401.14112. 24 [find](https://scholar.google.com/scholar?q=Xia%2C+H%2E%2C+Zheng%2C+Z%2E%2C+Wu%2C+X%2E%2C+Chen%2C+S%2E%2C+Yao%2C+Z%2E%2C+Youn%2C+S%2E%2C+Bakhtiari%2C+A%2E%2C+Wyatt%2C+M%2E%2C+Zhuang%2C+D%2E%2C+Zhou%2C+Z%2E%2C+et+al%2E+%282024%29%2E+Fp6%2Dllm%3A+Efficiently+serving+large+language+models+through+fp6%2Dcentric+algorithm)
213. Xia, M., Gao, T., Zeng, Z., and Chen, D. (2023b). Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694. 11 [find](https://scholar.google.com/scholar?q=Xia%2C+M%2E%2C+Gao%2C+T%2E%2C+Zeng%2C+Z%2E%2C+and+Chen%2C+D%2E+%282023b%29%2E+Sheared+llama%3A+Accelerating+language+model+pre%2Dtraining+via+structured+pruning%2E+arXiv+preprint+arXiv%3A2310%2E06694%2E+11)
214. Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. (2023a). Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087–38099. PMLR. 9 [find](https://scholar.google.com/scholar?q=Xiao%2C+G%2E%2C+Lin%2C+J%2E%2C+Seznec%2C+M%2E%2C+Wu%2C+H%2E%2C+Demouth%2C+J%2E%2C+and+Han%2C+S%2E+%282023a%29%2E+Smoothquant%3A+Accurate+and+efficient+post%2Dtraining+quantization+for+large+language+models%2E+In+International+Conference+on+Machin)
215. Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. (2023b). Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453. 27 [find](https://scholar.google.com/scholar?q=Xiao%2C+G%2E%2C+Tian%2C+Y%2E%2C+Chen%2C+B%2E%2C+Han%2C+S%2E%2C+and+Lewis%2C+M%2E+%282023b%29%2E+Efficient+streaming+language+models+with+attention+sinks%2E+arXiv+preprint+arXiv%3A2309%2E17453%2E+27)
216. Xiao, Y., Wu, L., Guo, J., Li, J., Zhang, M., Qin, T., and Liu, T.-y. (2023c). A survey on non-autoregressive generation for neural machine translation and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence. 19 [find](https://scholar.google.com/scholar?q=Xiao%2C+Y%2E%2C+Wu%2C+L%2E%2C+Guo%2C+J%2E%2C+Li%2C+J%2E%2C+Zhang%2C+M%2E%2C+Qin%2C+T%2E%2C+and+Liu%2C+T%2E%2Dy%2E+%282023c%29%2E+A+survey+on+non%2Dautoregressive+generation+for+neural+machine+translation+and+beyond%2E+IEEE+Transactions+on+Pattern+Analysi)
217. Xin, J., Tang, R., Lee, J., Yu, Y., and Lin, J. (2020). [Deebert: Dynamic early exiting for accelerating bert inference](https://md.research.elamur.ai/free/en/papers/xin-2020-deebert-dynamic-early-exiting-accelerating/). 13
218. Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., Khabsa, M., Fang, H., Mehdad, Y., Narang, S., Malik, K., Fan, A., Bhosale, S., Edunov, S., Lewis, M., Wang, S., and Ma, H. (2023). [Effective long-context scaling of foundation mo…](https://md.research.elamur.ai/free/en/papers/xiong-2023-effective-long-context-scaling-foundation/)
219. Xu, D., Yin, W., Jin, X., Zhang, Y., Wei, S., Xu, M., and Liu, X. (2023a). Llmcad: Fast and scalable on-device large language model inference. arXiv preprint arXiv:2309.04255. 17 — [arXiv:2309.04255](https://arxiv.org/abs/2309.04255) [find](https://scholar.google.com/scholar?q=Xu%2C+D%2E%2C+Yin%2C+W%2E%2C+Jin%2C+X%2E%2C+Zhang%2C+Y%2E%2C+Wei%2C+S%2E%2C+Xu%2C+M%2E%2C+and+Liu%2C+X%2E+%282023a%29%2E+Llmcad%3A+Fast+and+scalable+on%2Ddevice+large+language+model+inference%2E+arXiv+preprint+arXiv%3A2309%2E04255%2E+17+%E2%80%94+%5BarXiv%3A2309%2E04255%5D%28)
220. Xu, M., Xu, Y. L., and Mandic, D. P. (2023b). Tensorgpt: Efficient compression of the embedding layer in llms based on the tensor-train decomposition. arXiv preprint arXiv:2307.00526. 13 — [arXiv:2307.00526](https://arxiv.org/abs/2307.00526) [find](https://scholar.google.com/scholar?q=Xu%2C+M%2E%2C+Xu%2C+Y%2E+L%2E%2C+and+Mandic%2C+D%2E+P%2E+%282023b%29%2E+Tensorgpt%3A+Efficient+compression+of+the+embedding+layer+in+llms+based+on+the+tensor%2Dtrain+decomposition%2E+arXiv+preprint+arXiv%3A2307%2E00526%2E+13+%E2%80%94+%5BarXiv%3A2307)
221. Xu, X., Li, M., Tao, C., Shen, T., Cheng, R., Li, J., Xu, C., Tao, D., and Zhou, T. (2024). A survey on knowledge distillation of large language models. 11 [find](https://scholar.google.com/scholar?q=Xu%2C+X%2E%2C+Li%2C+M%2E%2C+Tao%2C+C%2E%2C+Shen%2C+T%2E%2C+Cheng%2C+R%2E%2C+Li%2C+J%2E%2C+Xu%2C+C%2E%2C+Tao%2C+D%2E%2C+and+Zhou%2C+T%2E+%282024%29%2E+A+survey+on+knowledge+distillation+of+large+language+models%2E+11)
222. Xu, Y., Xie, L., Gu, X., Chen, X., Chang, H., Zhang, H., Chen, Z., Zhang, X., and Tian, Q. (2023c). Qa-lora: Quantization-aware low-rank adaptation of large language models. arXiv preprint arXiv:2309.14717. 10 — [arXiv:2309.14717](https://arxiv.org/abs/2309.14717) [find](https://scholar.google.com/scholar?q=Xu%2C+Y%2E%2C+Xie%2C+L%2E%2C+Gu%2C+X%2E%2C+Chen%2C+X%2E%2C+Chang%2C+H%2E%2C+Zhang%2C+H%2E%2C+Chen%2C+Z%2E%2C+Zhang%2C+X%2E%2C+and+Tian%2C+Q%2E+%282023c%29%2E+Qa%2Dlora%3A+Quantization%2Daware+low%2Drank+adaptation+of+large+language+models%2E+arXiv+preprint+arXiv%3A2309%2E)
223. Yang, N., Ge, T., Wang, L., Jiao, B., Jiang, D., Yang, L., Majumder, R., and Wei, F. (2023). Inference with reference: Lossless acceleration of large language models. arXiv preprint arXiv:2304.04487. 18 — [arXiv:2304.04487](https://arxiv.org/abs/2304.04487) [find](https://scholar.google.com/scholar?q=Yang%2C+N%2E%2C+Ge%2C+T%2E%2C+Wang%2C+L%2E%2C+Jiao%2C+B%2E%2C+Jiang%2C+D%2E%2C+Yang%2C+L%2E%2C+Majumder%2C+R%2E%2C+and+Wei%2C+F%2E+%282023%29%2E+Inference+with+reference%3A+Lossless+acceleration+of+large+language+models%2E+arXiv+preprint+arXiv%3A2304%2E04487%2E+)
224. Yao, Z., Yazdani Aminabadi, R., Zhang, M., Wu, X., Li, C., and He, Y. (2022). [Zeroquant: Efficient and affordable post-training quantization for large-scale transformers](https://md.research.elamur.ai/free/en/papers/yao-2022-zeroquant-efficient-affordable-post-training/). Advances in Neural Information Processing Systems, 35:27168–27183. 8
225. Yi, R., Guo, L., Wei, S., Zhou, A., Wang, S., and Xu, M. (2023). Edgemoe: Fast on-device inference of moe-based large language models. arXiv preprint arXiv:2308.14352. 16 — [arXiv:2308.14352](https://arxiv.org/abs/2308.14352) [find](https://scholar.google.com/scholar?q=Yi%2C+R%2E%2C+Guo%2C+L%2E%2C+Wei%2C+S%2E%2C+Zhou%2C+A%2E%2C+Wang%2C+S%2E%2C+and+Xu%2C+M%2E+%282023%29%2E+Edgemoe%3A+Fast+on%2Ddevice+inference+of+moe%2Dbased+large+language+models%2E+arXiv+preprint+arXiv%3A2308%2E14352%2E+16+%E2%80%94+%5BarXiv%3A2308%2E14352%5D%28https%3A%2F%2F)
226. Yin, L., Wu, Y., Zhang, Z., Hsieh, C.-Y., Wang, Y., Jia, Y., Pechenizkiy, M., Liang, Y., Wang, Z., and Liu, S. (2023a). Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity. arXiv preprint arXiv:2310.05175. 11 — [arXiv:2310.05175](https://arxiv.org/abs/2310.05175) [find](https://scholar.google.com/scholar?q=Yin%2C+L%2E%2C+Wu%2C+Y%2E%2C+Zhang%2C+Z%2E%2C+Hsieh%2C+C%2E%2DY%2E%2C+Wang%2C+Y%2E%2C+Jia%2C+Y%2E%2C+Pechenizkiy%2C+M%2E%2C+Liang%2C+Y%2E%2C+Wang%2C+Z%2E%2C+and+Liu%2C+S%2E+%282023a%29%2E+Outlier+weighed+layerwise+sparsity+%28owl%29%3A+A+missing+secret+sauce+for+pruning+llm)
227. Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., and Chen, E. (2023b). [A survey on multimodal large language models](https://md.research.elamur.ai/free/en/papers/yin-2023-survey-multimodal-large-language-models/). arXiv preprint arXiv:2306.13549. 26
228. Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.-G. (2022). [Orca: A distributed serving system for {Transformer-Based} generative models](https://md.research.elamur.ai/free/en/papers/yu-2022-orca-distributed-serving-system-transformer/). In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538. 21
229. Yuan, Z., Li, Z., and Sun, L. (2023a). Tinygpt-v: Efficient multimodal large language model via small backbones. arXiv preprint arXiv:2312.16862. 26 — [arXiv:2312.16862](https://arxiv.org/abs/2312.16862) [find](https://scholar.google.com/scholar?q=Yuan%2C+Z%2E%2C+Li%2C+Z%2E%2C+and+Sun%2C+L%2E+%282023a%29%2E+Tinygpt%2Dv%3A+Efficient+multimodal+large+language+model+via+small+backbones%2E+arXiv+preprint+arXiv%3A2312%2E16862%2E+26+%E2%80%94+%5BarXiv%3A2312%2E16862%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2312%2E168)
230. Yuan, Z., Liu, J., Wu, J., Yang, D., Wu, Q., Sun, G., Liu, W., Wang, X., and Wu, B. (2023b). Benchmarking the reliability of post-training quantization: a particular focus on worst-case performance. arXiv preprint arXiv:2303.13003. 26 — [arXiv:2303.13003](https://arxiv.org/abs/2303.13003) [find](https://scholar.google.com/scholar?q=Yuan%2C+Z%2E%2C+Liu%2C+J%2E%2C+Wu%2C+J%2E%2C+Yang%2C+D%2E%2C+Wu%2C+Q%2E%2C+Sun%2C+G%2E%2C+Liu%2C+W%2E%2C+Wang%2C+X%2E%2C+and+Wu%2C+B%2E+%282023b%29%2E+Benchmarking+the+reliability+of+post%2Dtraining+quantization%3A+a+particular+focus+on+worst%2Dcase+performance%2E+a)
231. Yuan, Z., Niu, L., Liu, J., Liu, W., Wang, X., Shang, Y., Sun, G., Wu, Q., Wu, J., and Wu, B. (2023c). Rptq: Reorder-based post-training quantization for large language models. arXiv preprint arXiv:2304.01089. 9, 25 — [arXiv:2304.01089](https://arxiv.org/abs/2304.01089) [find](https://scholar.google.com/scholar?q=Yuan%2C+Z%2E%2C+Niu%2C+L%2E%2C+Liu%2C+J%2E%2C+Liu%2C+W%2E%2C+Wang%2C+X%2E%2C+Shang%2C+Y%2E%2C+Sun%2C+G%2E%2C+Wu%2C+Q%2E%2C+Wu%2C+J%2E%2C+and+Wu%2C+B%2E+%282023c%29%2E+Rptq%3A+Reorder%2Dbased+post%2Dtraining+quantization+for+large+language+models%2E+arXiv+preprint+arXiv%3A23)
232. Yuan, Z., Shang, Y., Song, Y., Wu, Q., Yan, Y., and Sun, G. (2023d). Asvd: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821. 12 — [arXiv:2312.05821](https://arxiv.org/abs/2312.05821) [find](https://scholar.google.com/scholar?q=Yuan%2C+Z%2E%2C+Shang%2C+Y%2E%2C+Song%2C+Y%2E%2C+Wu%2C+Q%2E%2C+Yan%2C+Y%2E%2C+and+Sun%2C+G%2E+%282023d%29%2E+Asvd%3A+Activation%2Daware+singular+value+decomposition+for+compressing+large+language+models%2E+arXiv+preprint+arXiv%3A2312%2E05821%2E+12+%E2%80%94+%5Ba)
233. Yue, Y., Yuan, Z., Duanmu, H., Zhou, S., Wu, J., and Nie, L. (2024). Wkvquant: Quantizing weight and key/value cache for large language models gains more. arXiv preprint arXiv:2402.12065. 10 — [arXiv:2402.12065](https://arxiv.org/abs/2402.12065) [find](https://scholar.google.com/scholar?q=Yue%2C+Y%2E%2C+Yuan%2C+Z%2E%2C+Duanmu%2C+H%2E%2C+Zhou%2C+S%2E%2C+Wu%2C+J%2E%2C+and+Nie%2C+L%2E+%282024%29%2E+Wkvquant%3A+Quantizing+weight+and+key%2Fvalue+cache+for+large+language+models+gains+more%2E+arXiv+preprint+arXiv%3A2402%2E12065%2E+10+%E2%80%94+%5BarXiv%3A)
234. Yuksel, S. E., Wilson, J. N., and Gader, P. D. (2012). Twenty years of mixture of experts. IEEE Transactions on Neural Networks and Learning Systems, 23(8):1177–1193. 15, 26 [find](https://scholar.google.com/scholar?q=Yuksel%2C+S%2E+E%2E%2C+Wilson%2C+J%2E+N%2E%2C+and+Gader%2C+P%2E+D%2E+%282012%29%2E+Twenty+years+of+mixture+of+experts%2E+IEEE+Transactions+on+Neural+Networks+and+Learning+Systems%2C+23%288%29%3A1177%E2%80%931193%2E+15%2C+26)
235. Zadeh, A. H. et al. (2020). Gobo: Quantizing attention-based nlp models for low latency and energy efficient inference. In MI-CRO, pages 811–824. IEEE. 24 [find](https://scholar.google.com/scholar?q=Zadeh%2C+A%2E+H%2E+et+al%2E+%282020%29%2E+Gobo%3A+Quantizing+attention%2Dbased+nlp+models+for+low+latency+and+energy+efficient+inference%2E+In+MI%2DCRO%2C+pages+811%E2%80%93824%2E+IEEE%2E+24)
236. Zeng, S., Liu, J., Dai, G., Yang, X., Fu, T., Wang, H., Ma, W., Sun, H., Li, S., Huang, Z., et al. (2024). Flightllm: Efficient large language model inference with a complete mapping flow on fpga. arXiv preprint arXiv:2401.03868. 25 [find](https://scholar.google.com/scholar?q=Zeng%2C+S%2E%2C+Liu%2C+J%2E%2C+Dai%2C+G%2E%2C+Yang%2C+X%2E%2C+Fu%2C+T%2E%2C+Wang%2C+H%2E%2C+Ma%2C+W%2E%2C+Sun%2C+H%2E%2C+Li%2C+S%2E%2C+Huang%2C+Z%2E%2C+et+al%2E+%282024%29%2E+Flightllm%3A+Efficient+large+language+model+inference+with+a+complete+mapping+flow+on+fpga%2E+arX)
237. Zeng, Z., Hong, Y., Dai, H., Zhuang, H., and Chen, C. (2023). Consistentee: A consistent and hardness-guided early exiting method for accelerating language models inference. 14 [find](https://scholar.google.com/scholar?q=Zeng%2C+Z%2E%2C+Hong%2C+Y%2E%2C+Dai%2C+H%2E%2C+Zhuang%2C+H%2E%2C+and+Chen%2C+C%2E+%282023%29%2E+Consistentee%3A+A+consistent+and+hardness%2Dguided+early+exiting+method+for+accelerating+language+models+inference%2E+14)
238. Zhang, D., Yu, Y., Li, C., Dong, J., Su, D., Chu, C., and Yu, D. (2024a). [Mm-llms: Recent advances in multimodal large language models](https://md.research.elamur.ai/free/en/papers/zhang-2024-mm-llms-recent-advances-multimodal/). arXiv preprint arXiv:2401.13601. 26
239. Zhang, J., Wang, J., Li, H., Shou, L., Chen, K., Chen, G., and Mehrotra, S. (2023). [Draft & verify: Lossless large language model acceleration via self-speculative decoding](https://md.research.elamur.ai/free/en/papers/zhang-2024-draft-verify-lossless-large-language/). arXiv preprint arXiv:2309.08168. 18
240. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022). [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv preprint arXiv:2205.01068. 1
241. Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., et al. (2024b). H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36. 27 [find](https://scholar.google.com/scholar?q=Zhang%2C+Z%2E%2C+Sheng%2C+Y%2E%2C+Zhou%2C+T%2E%2C+Chen%2C+T%2E%2C+Zheng%2C+L%2E%2C+Cai%2C+R%2E%2C+Song%2C+Z%2E%2C+Tian%2C+Y%2E%2C+Re%CC%81%2C+C%2E%2C+Barrett%2C+C%2E%2C+et+al%2E+%282024b%29%2E+H2o%3A+Heavy%2Dhitter+oracle+for+efficient+generative+inference+of+large+language+mo)
242. Zhao, P., Zhang, H., Yu, Q., Wang, Z., Geng, Y., Fu, F., Yang, L., Zhang, W., and Cui, B. (2024). [Retrieval-augmented generation for ai-generated content: A survey](https://md.research.elamur.ai/free/en/papers/zhao-2024-retrieval-augmented-generation-ai-generated/). arXiv preprint arXiv:2402.19473. 27
243. Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023). [A survey of large language models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/). arXiv preprint arXiv:2303.18223. 1, 3
244. Zhong, W., Guo, L., Gao, Q., and Wang, Y. (2023). Memorybank: Enhancing large language models with long-term memory. arXiv preprint arXiv:2305.10250. 27 [find](https://scholar.google.com/scholar?q=Zhong%2C+W%2E%2C+Guo%2C+L%2E%2C+Gao%2C+Q%2E%2C+and+Wang%2C+Y%2E+%282023%29%2E+Memorybank%3A+Enhancing+large+language+models+with+long%2Dterm+memory%2E+arXiv+preprint+arXiv%3A2305%2E10250%2E+27)
245. Zhou, B., Hu, Y., Weng, X., Jia, J., Luo, J., Liu, X., Wu, J., and Huang, L. (2024). Tinyllava: A framework of small-scale large multimodal models. arXiv preprint arXiv:2402.14289. 26 — [arXiv:2402.14289](https://arxiv.org/abs/2402.14289) [find](https://scholar.google.com/scholar?q=Zhou%2C+B%2E%2C+Hu%2C+Y%2E%2C+Weng%2C+X%2E%2C+Jia%2C+J%2E%2C+Luo%2C+J%2E%2C+Liu%2C+X%2E%2C+Wu%2C+J%2E%2C+and+Huang%2C+L%2E+%282024%29%2E+Tinyllava%3A+A+framework+of+small%2Dscale+large+multimodal+models%2E+arXiv+preprint+arXiv%3A2402%2E14289%2E+26+%E2%80%94+%5BarXiv%3A2402%2E14)
246. Zhou, J., Wu, J., Gao, Y., Ding, Y., Tao, C., Li, B., Tu, F., Cheng, K.-T., So, H. K.-H., and Wong, N. (2023a). Dybit: Dynamic bit-precision numbers for efficient quantized neural network inference. arXiv preprint arXiv:2302.12510. 24 — [arXiv:2302.12510](https://arxiv.org/abs/2302.12510) [find](https://scholar.google.com/scholar?q=Zhou%2C+J%2E%2C+Wu%2C+J%2E%2C+Gao%2C+Y%2E%2C+Ding%2C+Y%2E%2C+Tao%2C+C%2E%2C+Li%2C+B%2E%2C+Tu%2C+F%2E%2C+Cheng%2C+K%2E%2DT%2E%2C+So%2C+H%2E+K%2E%2DH%2E%2C+and+Wong%2C+N%2E+%282023a%29%2E+Dybit%3A+Dynamic+bit%2Dprecision+numbers+for+efficient+quantized+neural+network+inference%2E+a)
247. Zhou, W., Jiang, Y. E., Cui, P., Wang, T., Xiao, Z., Hou, Y., Cotterell, R., and Sachan, M. (2023b). [Recurrentgpt: Interactive generation of (arbitrarily) long text](https://md.research.elamur.ai/free/en/papers/zhou-2023-recurrentgpt-interactive-generation-arbitrarily-long/). arXiv preprint arXiv:2305.13304. 27
248. Zhou, W., Xu, C., Ge, T., McAuley, J., Xu, K., and Wei, F. (2020). Bert loses patience: Fast and robust inference with early exit. 13 [find](https://scholar.google.com/scholar?q=Zhou%2C+W%2E%2C+Xu%2C+C%2E%2C+Ge%2C+T%2E%2C+McAuley%2C+J%2E%2C+Xu%2C+K%2E%2C+and+Wei%2C+F%2E+%282020%29%2E+Bert+loses+patience%3A+Fast+and+robust+inference+with+early+exit%2E+13)
249. Zhou, Y., Lyu, K., Rawat, A. S., Menon, A. K., Rostamizadeh, A., Kumar, S., Kagy, J.-F., and Agarwal, R. (2023c). Distillspec: Improving speculative decoding via knowledge distillation. arXiv preprint arXiv:2310.08461. 18 — [arXiv:2310.08461](https://arxiv.org/abs/2310.08461) [find](https://scholar.google.com/scholar?q=Zhou%2C+Y%2E%2C+Lyu%2C+K%2E%2C+Rawat%2C+A%2E+S%2E%2C+Menon%2C+A%2E+K%2E%2C+Rostamizadeh%2C+A%2E%2C+Kumar%2C+S%2E%2C+Kagy%2C+J%2E%2DF%2E%2C+and+Agarwal%2C+R%2E+%282023c%29%2E+Distillspec%3A+Improving+speculative+decoding+via+knowledge+distillation%2E+arXiv+preprint)
250. Zhou, Z., Li, C., Yang, F., and Suny, G. (2023d). Dimm-link: Enabling efficient inter-dimm communication for near-memory processing. In 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 302–316. IEEE. 23 [find](https://scholar.google.com/scholar?q=Zhou%2C+Z%2E%2C+Li%2C+C%2E%2C+Yang%2C+F%2E%2C+and+Suny%2C+G%2E+%282023d%29%2E+Dimm%2Dlink%3A+Enabling+efficient+inter%2Ddimm+communication+for+near%2Dmemory+processing%2E+In+2023+IEEE+International+Symposium+on+High%2DPerformance+Computer+A)
251. Zhu, W. (2021). LeeBERT: Learned early exit for BERT with cross-level optimization. In Zong, C., Xia, F., Li, W., and Navigli, R., editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processi… [find](https://scholar.google.com/scholar?q=Zhu%2C+W%2E+%282021%29%2E+LeeBERT%3A+Learned+early+exit+for+BERT+with+cross%2Dlevel+optimization%2E+In+Zong%2C+C%2E%2C+Xia%2C+F%2E%2C+Li%2C+W%2E%2C+and+Navigli%2C+R%2E%2C+editors%2C+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+fo)
252. Zhu, X., Li, J., Liu, Y., Ma, C., and Wang, W. (2023). [A survey on model compression for large language models](https://md.research.elamur.ai/free/en/papers/zhu-2024-survey-model-compression-large-language/). arXiv preprint arXiv:2308.07633. 2
253. Zhu, Y., Zhu, M., Liu, N., Ou, Z., Mou, X., and Tang, J. (2024). Llava-phi: Efficient multi-modal assistant with small language model. arXiv preprint arXiv:2401.02330. 26 — [arXiv:2401.02330](https://arxiv.org/abs/2401.02330) [find](https://scholar.google.com/scholar?q=Zhu%2C+Y%2E%2C+Zhu%2C+M%2E%2C+Liu%2C+N%2E%2C+Ou%2C+Z%2E%2C+Mou%2C+X%2E%2C+and+Tang%2C+J%2E+%282024%29%2E+Llava%2Dphi%3A+Efficient+multi%2Dmodal+assistant+with+small+language+model%2E+arXiv+preprint+arXiv%3A2401%2E02330%2E+26+%E2%80%94+%5BarXiv%3A2401%2E02330%5D%28https%3A%2F%2F)
254. Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y., Dean, J., Shazeer, N., and Fedus, W. (2022). St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906. 16 [find](https://scholar.google.com/scholar?q=Zoph%2C+B%2E%2C+Bello%2C+I%2E%2C+Kumar%2C+S%2E%2C+Du%2C+N%2E%2C+Huang%2C+Y%2E%2C+Dean%2C+J%2E%2C+Shazeer%2C+N%2E%2C+and+Fedus%2C+W%2E+%282022%29%2E+St%2Dmoe%3A+Designing+stable+and+transferable+sparse+expert+models%2E+arXiv+preprint+arXiv%3A2202%2E08906%2E+16)

*Updated 2 November 2026*