---
title: "Puzzle: Distillation-Based NAS for Inference-Optimized LLMs"
description: "Serving large language models is costly, and standard transformers stack identical layers that suit… 63 references, cited by 8 papers here."
authors: ["Akhiad Bercovich", "Tomer Ronen", "Talor Abramovich", "Nir Ailon", "Nave Assaf", "Mohammad Dabbah", "Ido Galil", "Amnon Geifman", "Yonatan Geifman", "Izhak Golan", "Netanel Haber", "Ehud Karpas", "Roi Koren", "Itay Levy", "Pavlo Molchanov", "Shahar Mor", "Zach Moshe", "Najeeb Nabwani", "Omri Puny", "Ran Rubin", "Itamar Schen", "Ido Shahaf", "Oren Tropp", "Omer Ullman Argov", "Ran Zilberstein", "Ran El-Yaniv"]
year: 2024
venue: "arXiv:2411.19146"
source: "https://arxiv.org/abs/2411.19146"
doi: "10.48550/arxiv.2411.19146"
arxiv: "2411.19146"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bercovich-2024-puzzle-distillation-based-nas-inference/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bercovich-2024-puzzle-distillation-based-nas-inference/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/14/) / Puzzle: Distillation-Based NAS for Inference-Optimized LLMs

# Puzzle: Distillation-Based NAS for Inference-Optimized LLMs

**Authors:** [Akhiad Bercovich](https://md.research.elamur.ai/free/en/authors/akhiad-bercovich/), [Tomer Ronen](https://md.research.elamur.ai/free/en/authors/tomer-ronen/), Talor Abramovich, Nir Ailon, Nave Assaf, [Mohammad Dabbah](https://md.research.elamur.ai/free/en/authors/mohammad-dabbah/), [Ido Galil](https://md.research.elamur.ai/free/en/authors/ido-galil/), Amnon Geifman, Yonatan Geifman, Izhak Golan, Netanel Haber, [Ehud Karpas](https://md.research.elamur.ai/free/en/authors/ehud-karpas/), Roi Koren, [Itay Levy](https://md.research.elamur.ai/free/en/authors/itay-levy/), [Pavlo Molchanov](https://md.research.elamur.ai/free/en/authors/pavlo-molchanov/), Shahar Mor, Zach Moshe, Najeeb Nabwani, Omri Puny, Ran Rubin, Itamar Schen, Ido Shahaf, Oren Tropp, Omer Ullman Argov, Ran Zilberstein and [Ran El-Yaniv](https://md.research.elamur.ai/free/en/authors/ran-el-yaniv/) · **Year:** 2024 · **Venue:** arXiv:2411.19146 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2411.19146) · [PDF](https://arxiv.org/pdf/2411.19146) · [DOI 10.48550/arxiv.2411.19146](https://doi.org/10.48550/arxiv.2411.19146) · [arXiv 2411.19146](https://arxiv.org/abs/2411.19146) · [OpenAlex](https://openalex.org/W4405033040)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [8](https://md.research.elamur.ai/free/en/papers/bercovich-2024-puzzle-distillation-based-nas-inference/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Serving large language models is costly, and standard transformers stack identical layers that suit training stability rather than cheap inference. Puzzle, from NVIDIA, converts an already trained "parent" LLM into a heterogeneous architecture tuned for specific hardware and deployment limits. It has three stages. First, it trains many cheaper alternatives for each layer's attention and FFN subblocks in parallel using blockwise local distillation (BLD), then scores each for quality and cost. These alternatives include fewer KV heads, linear replacements, narrower FFNs, or skipping the subblock entirely. Second, mixed-integer programming selects a per-layer combination that maximizes estimated quality under throughput, latency and memory constraints. Third, global knowledge distillation polishes the assembled model. The method needs only the parent's weights, not its training data. Applied to Llama-3.1-70B-Instruct, it produced Nemotron-51B and Nemotron-49B, which fit on a single H100 and run about twice as fast with minimal accuracy loss. The authors argue that inference efficiency on target hardware, not parameter count, should guide model selection.

## Key points
- Nemotron-51B and Llama-3.3-Nemotron-49B achieve a 2.17× throughput speedup on a single NVIDIA H100 (FP8) and keep 98.4% of Llama-70B-Instruct's benchmark accuracy.
- The full pipeline used under 50B tokens (at most 45B for the released models), compared with roughly 15T tokens for training the parent. Reasonable results remain possible with budgets as small as 4B tokens.
- Ablations show the MIP search matters. Randomly assembling blocks from the same trained library reaches only about 86.6% of Nemotron-51B's score, and skipping GKD uptraining lowers MMLU and MT-Bench.
- At similar throughput, Nemotron-51B keeps 99.49% of the parent's average MMLU/MT-Bench accuracy. Wanda 2:4 sparsity keeps 92.23% and low-rank approximation keeps 88.96%.
- The authors extended TensorRT-LLM to handle non-uniform layers and varying KV-head counts. Nemotron-49B retains the parent's 128K context, and after RLHF it beats Nemotron-51B on some tasks.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,621 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 328 works

## Cite

Cite as: Akhiad Bercovich, Tomer Ronen, Talor Abramovich, Nir Ailon, Nave Assaf, Mohammad Dabbah, Ido Galil, Amnon Geifman, Yonatan Geifman, Izhak Golan, Netanel Haber, Ehud Karpas, Roi Koren, Itay Levy, Pavlo Molchanov, Shahar Mor, Zach Moshe, Najeeb Nabwani, Omri Puny, Ran Rubin, Itamar Schen, Ido Shahaf, Oren Tropp, Omer Ullman Argov, Ran Zilberstein and Ran El-Yaniv. 2024. Puzzle: Distillation-Based NAS for Inference-Optimized LLMs. arXiv:2411.19146.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{bercovich2024puzzle,
  title = {{Puzzle: Distillation-Based NAS for Inference-Optimized LLMs}},
  author = {Akhiad Bercovich, Tomer Ronen, Talor Abramovich, Nir Ailon, Nave Assaf, Mohammad Dabbah, Ido Galil, Amnon Geifman, Yonatan Geifman, Izhak Golan, Netanel Haber, Ehud Karpas, Roi Koren, Itay Levy, Pavlo Molchanov, Shahar Mor, Zach Moshe, Najeeb Nabwani, Omri Puny, Ran Rubin, Itamar Schen, Ido Shahaf, Oren Tropp, Omer Ullman Argov, Ran Zilberstein and Ran El-Yaniv},
  year = {2024},
  doi = {10.48550/arxiv.2411.19146},
  eprint = {2411.19146},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2411.19146}
}
```

- [← previous Progress or Regress? Self-Improvement Reversal in Post-training](https://md.research.elamur.ai/free/en/papers/wu-2024-progress-regress-self-improvement-reversal/)
- [next → QuantAgent: Seeking Holy Grail in Trading by Self-Improving Large Language Model](https://md.research.elamur.ai/free/en/papers/wang-2024-quantagent-seeking-holy-grail-trading/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/14/)

## References

63 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. The claude 3 model family: Opus, sonnet, haiku. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 268232499. [↗](https://api.semanticscholar.org/CorpusID:)
2. Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan M. Cohen, Sirshak Das, Ayush Dattagupta, Olivier Delalleau, Leon Derczynski, Yi Dong, Daniel Egert, Ellie Evans, Aleksander Ficek, Denys Fridman, Shaona Gho… — [Nemotron-4 340B Technical Report](https://md.research.elamur.ai/free/en/papers/adler-2024-nemotron-4-340b-technical-report/)
3. Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. [Intrinsic dimensionality explains the effectiveness of language model fine-tuning](https://md.research.elamur.ai/free/en/papers/aghajanyan-2021-intrinsic-dimensionality-explains-effectiveness-language/). In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the…
4. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. [GQA: training generalized multi-query transformer models from multi-head checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/). In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Method…
5. Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via overparameterization. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume… [find](https://scholar.google.com/scholar?q=Zeyuan+Allen%2DZhu%2C+Yuanzhi+Li%2C+and+Zhao+Song%2E+A+convergence+theory+for+deep+learning+via+overparameterization%2E+In+Kamalika+Chaudhuri+and+Ruslan+Salakhutdinov%2C+editors%2C+Proceedings+of+the+36th+Internati)
6. Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap,… — [Gemini: A Family of Highly Capable Multimodal Models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/)
7. Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari Do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.n… [find](https://scholar.google.com/scholar?q=Saleh+Ashkboos%2C+Maximilian+L%2E+Croci%2C+Marcelo+Gennari+Do+Nascimento%2C+Torsten+Hoefler%2C+and+James+Hensman%2E+Slicegpt%3A+Compress+large+language+models+by+deleting+rows+and+columns%2E+In+The+Twelfth+Internatio)
8. Mikhail Belkin, Daniel J. Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias–variance trade-off. Proceedings of the National Academy of Sciences, 116:15849 – 15854, 2018. URL [https://api.semanticscholar.org/CorpusID:198496504](https://api.semanticscholar.org/CorpusID:198496504). [↗](https://api.semanticscholar.org/CorpusID:198496504)
9. Iz Beltagy, Matthew E. Peters, and Arman Cohan. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). CoRR, abs/2004.05150, 2020. URL https://arxiv.org/abs/2004.05150. [↗](https://arxiv.org/abs/2004.05150)
10. Akhiad Bercovich, Mohammad Dabbah, Omri Puny, Ido Galil, Amnon Geifman, Yonatan Geifman, Izhak Golan, Ehud Karpas, Itay Levy, Zach Moshe, Najeeb Nabwani, Tomer Ronen, Itamar Schen, Elad Segal, Ido Shahaf, Oren Tropp, Ran Zilberstein, and Ran El-Yaniv. FFN fusion: Rethinking sequential computation in… — [doi:10.48550/arxiv.2503.18908](https://doi.org/10.48550/arxiv.2503.18908) [find](https://scholar.google.com/scholar?q=Akhiad+Bercovich%2C+Mohammad+Dabbah%2C+Omri+Puny%2C+Ido+Galil%2C+Amnon+Geifman%2C+Yonatan+Geifman%2C+Izhak+Golan%2C+Ehud+Karpas%2C+Itay+Levy%2C+Zach+Moshe%2C+Najeeb+Nabwani%2C+Tomer+Ronen%2C+Itamar+Schen%2C+Elad+Segal%2C+Ido+Sha)
11. Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adithya Renduchintala, Akhiad Bercovich, Aleksander Ficek, Alexis Bjorlin, Ali Taghibakhshi, Amala Sanjay Deshmukh, Ameya Sunil Mahabaleshwarkar, Andrew Tao, Anna Shors, Ashwath Aithal, Ashwin Poojary, Ayush Dattagupta, Balaram Buddharaju, Bobby Chen, B… — [Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models](https://md.research.elamur.ai/free/en/papers/blakeman-2025-nemotron-h-family-accurate-efficient/)
12. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikha… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
13. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. [Think you have solved question answering? try arc, the AI2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). CoRR, abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457. [↗](http://arxiv.org/abs/1803.05457)
14. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). CoRR, abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.1416… [↗](https://arxiv.org/abs/2110.1416…)
15. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
16. Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts. [The unreasonable ineffectiveness of the deeper layers](https://md.research.elamur.ai/free/en/papers/gromov-2024-unreasonable-ineffectiveness-deeper-layers/). CoRR, abs/2403.17887, 2024. doi: 10.48550/ARXIV.2403.17887. URL https://doi.org/10.48550/arXiv.2403.17887. [↗](https://doi.org/10.48550/arXiv.2403.17887)
17. Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024. URL [https://openreview.net/forum?id=tEYskw1VY2](https://openreview.net/forum?id=tEYskw1VY2). [↗](https://openreview.net/forum?id=tEYskw1VY2)
18. Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022. URL [https://openreview.net/forum?id=](https://openreview.net/forum?id=) uYLFoz1vlAC. [↗](https://openreview.net/forum?id=)
19. Moritz Hardt and Yu Sun. Test-time training on nearest neighbors for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL [https://openreview.net/forum?id=CNL2bku4ra](https://openreview.net/forum?id=CNL2bku4ra). [↗](https://openreview.net/forum?id=CNL2bku4ra)
20. Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Samin Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. Xl-sum: Large-scale multilingual abstractive summarization for 44 languages. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Findings of… [find](https://scholar.google.com/scholar?q=Tahmid+Hasan%2C+Abhik+Bhattacharjee%2C+Md%2E+Saiful+Islam%2C+Kazi+Samin+Mubasshir%2C+Yuan%2DFang+Li%2C+Yong%2DBin+Kang%2C+M%2E+Sohel+Rahman%2C+and+Rifat+Shahriyar%2E+Xl%2Dsum%3A+Large%2Dscale+multilingual+abstractive+summarization)
21. Shwai He, Guoheng Sun, Zheyu Shen, and Ang Li. What matters in transformers? not all attention is needed. CoRR, abs/2406.15786, 2024. doi: 10.48550/ARXIV.2406.15786. URL https://doi.org/10. 48550/arXiv.2406.15786. — [doi:10.48550/arxiv.2406.15786](https://doi.org/10.48550/arxiv.2406.15786) [find](https://scholar.google.com/scholar?q=Shwai+He%2C+Guoheng+Sun%2C+Zheyu+Shen%2C+and+Ang+Li%2E+What+matters+in+transformers%3F+not+all+attention+is+needed%2E+CoRR%2C+abs%2F2406%2E15786%2C+2024%2E+doi%3A+10%2E48550%2FARXIV%2E2406%2E15786%2E+URL+https%3A%2F%2Fdoi%2Eorg%2F10%2E+48550%2FarXi)
22. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https: /…
23. Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). CoRR, abs/1503.02531, 2015. URL http://arxiv.org/abs/1503.02531. [↗](http://arxiv.org/abs/1503.02531)
24. Hive-Digital-Technologies. [https://huggingface.co/datasets/H-D-T/Buzz-V1.2](https://huggingface.co/datasets/H-D-T/Buzz-V1.2). [↗](https://huggingface.co/datasets/H-D-T/Buzz-V1.2)
25. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — [Training Compute-Optimal Large Language Models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
26. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: what’s the real context size of your long-context language models? CoRR, abs/2404.[06654, 2024](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/). doi: 10.48550/ARXIV.2404.06654. URL https://doi.org/10.48550/arXiv.2404. 06654. [↗](https://doi.org/10.48550/arXiv.2404)
27. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL…
28. COIN-OR Foundation Inc. Python-mip: collection of python tools for the modeling and solution of mixed-integer linear programs, 2023. URL [https://github.com/coin-or/python-mip](https://github.com/coin-or/python-mip). [↗](https://github.com/coin-or/python-mip)
29. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). CoRR, abs/2001.08361, 2020. URL https://arxiv.org/abs/2001.08361. [↗](https://arxiv.org/abs/2001.08361)
30. Mikhail Khodak, Neil A. Tenenholtz, Lester Mackey, and Nicolò Fusi. Initialization and regularization of factorized neural layers. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL [https://openreview.net/forum?id…](https://openreview.net/forum?id…) [↗](https://openreview.net/forum?id…)
31. Eldar Kurtic, Denis Kuznedelev, Elias Frantar, Michael Goin, and Dan Alistarh. Sparse fine-tuning for inference acceleration of large language models. CoRR, abs/2310.06927, 2023. doi: 10.48550/ARXIV.2310.06927. URL https://doi.org/10.48550/arXiv.2310.06927. [↗](https://doi.org/10.48550/arXiv.2310.06927)
32. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/), 2023. URL https://arxiv.org/abs/2309.06180. [↗](https://arxiv.org/abs/2309.06180)
33. Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. [Retrieval-augmented generation for knowledge-intensive nlp tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). ArXiv, abs/2005.11401, 2020. URL https://a…
34. Changlin Li, Jiefeng Peng, Liuchun Yuan, Guangrun Wang, Xiaodan Liang, Liang Lin, and Xiaojun Chang. Block-wisely supervised neural architecture search with knowledge distillation. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1986–1995, 2019. URL https://api. sem… [find](https://scholar.google.com/scholar?q=Changlin+Li%2C+Jiefeng+Peng%2C+Liuchun+Yuan%2C+Guangrun+Wang%2C+Xiaodan+Liang%2C+Liang+Lin%2C+and+Xiaojun+Chang%2E+Block%2Dwisely+supervised+neural+architecture+search+with+knowledge+distillation%2E+2020+IEEE%2FCVF+Confe)
35. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. [From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline](https://md.research.elamur.ai/free/en/papers/li-2024-crowdsourced-data-high-quality-benchmarks/). CoRR, abs/2406.11939, 2024. doi: 10.48550/ARXIV.2406.11939. URL https://doi.org/10.48550/arXiv.240… [↗](https://doi.org/10.48550/arXiv.240…)
36. Stephanie Lin, Jacob Hilton, and Owain Evans. [Truthfulqa: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dubl…
37. Gui Ling, Ziyang Wang, Yuliang Yan, and Qingwen Liu. Slimgpt: Layer-wise structured pruning for large language models. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: An… [find](https://scholar.google.com/scholar?q=Gui+Ling%2C+Ziyang+Wang%2C+Yuliang+Yan%2C+and+Qingwen+Liu%2E+Slimgpt%3A+Layer%2Dwise+structured+pruning+for+large+language+models%2E+In+Amir+Globersons%2C+Lester+Mackey%2C+Danielle+Belgrave%2C+Angela+Fan%2C+Ulrich+Paquet%2C+)
38. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). ArXiv, abs/2304.08485, 2023. URL [https://api.semanticscholar.org/CorpusID:258179774](https://api.semanticscholar.org/CorpusID:258179774). [↗](https://api.semanticscholar.org/CorpusID:258179774)
39. Chengqiang Lu, Jianwei Zhang, Yunfei Chu, Zhengyu Chen, Jingren Zhou, Fei Wu, Haiqing Chen, and Hongxia Yang. Knowledge distillation of transformer-based language models revisited. arXiv preprint arXiv:2206.14366, 2022. [find](https://scholar.google.com/scholar?q=Chengqiang+Lu%2C+Jianwei+Zhang%2C+Yunfei+Chu%2C+Zhengyu+Chen%2C+Jingren+Zhou%2C+Fei+Wu%2C+Haiqing+Chen%2C+and+Hongxia+Yang%2E+Knowledge+distillation+of+transformer%2Dbased+language+models+revisited%2E+arXiv+preprint+arXi)
40. Xinyin Ma, Gongfan Fang, and Xinchao Wang. [Llm-pruner: On the structural pruning of large language models](https://md.research.elamur.ai/free/en/papers/ma-2023-llm-pruner-structural-pruning-large/). In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informatio…
41. Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. [Shortgpt: Layers in large language models are more redundant than you expect](https://md.research.elamur.ai/free/en/papers/men-2024-shortgpt-layers-large-language-models/). CoRR, abs/2403.03853, 2024. doi: 10.48550/ARXIV.2403.03853. URL https://doi.org/10.48550/arXiv.2403.03853. [↗](https://doi.org/10.48550/arXiv.2403.03853)
42. Pavlo Molchanov, Jimmy Hall, Hongxu Yin, Jan Kautz, Nicolò Fusi, and Arash Vahdat. LANA: latency aware network acceleration. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Is… [find](https://scholar.google.com/scholar?q=Pavlo+Molchanov%2C+Jimmy+Hall%2C+Hongxu+Yin%2C+Jan+Kautz%2C+Nicolo%CC%80+Fusi%2C+and+Arash+Vahdat%2E+LANA%3A+latency+aware+network+acceleration%2E+In+Shai+Avidan%2C+Gabriel+J%2E+Brostow%2C+Moustapha+Cisse%CC%81%2C+Giovanni+Maria+Farin)
43. Bert Moons, Parham Noorzad, Andrii Skliar, Giovanni Mariani, Dushyant Mehta, Chris Lott, and Tijmen Blankevoort. Distilling optimal neural networks: Rapid search in diverse spaces. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 12209–12218, 2020. URL [https://api.semanticscho…](https://api.semanticscho…) [↗](https://api.semanticscho…)
44. Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation. CoRR, abs/2407.14679, 2024. doi: 10.48550/ARXIV.2407.14679. URL htt… [find](https://scholar.google.com/scholar?q=Saurav+Muralidharan%2C+Sharath+Turuvekere+Sreenivas%2C+Raviraj+Joshi%2C+Marcin+Chochowski%2C+Mostofa+Patwary%2C+Mohammad+Shoeybi%2C+Bryan+Catanzaro%2C+Jan+Kautz%2C+and+Pavlo+Molchanov%2E+Compact+language+models+via+pru)
45. OpenAI. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). CoRR, abs/2303.08774, 2023. doi: 10.48550/ARXIV.2303.08774. URL https://doi.org/10.48550/arXiv.2303.08774. [↗](https://doi.org/10.48550/arXiv.2303.08774)
46. Guilherme Penedo, Hynek Kydlı́cek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. [The fineweb datasets: Decanting the web for the finest text data at scale](https://md.research.elamur.ai/free/en/papers/penedo-2024-fineweb-datasets-decanting-web-finest/). CoRR, abs/2406.17557, 2024. doi: 10.48550/ARXIV.2406.17557. URL https://doi.org/10.48550…
47. Project Gutenberg. [https://www.gutenberg.org](https://www.gutenberg.org). [↗](https://www.gutenberg.org)
48. Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V. Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In International Conference on Machine Learning, 2017. URL [https://api.semanticscholar.org/CorpusID:743641](https://api.semanticscholar.org/CorpusID:743641). [↗](https://api.semanticscholar.org/CorpusID:743641)
49. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020… [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+In+The+Thirty%2DFourth+AAAI+Conference+on+Artificial+Intelligence%2C+A)
50. Sunny Sanyal, Ravid Shwartz-Ziv, Alexandros G. Dimakis, and Sujay Sanghavi. Inheritune: Training smaller yet more attentive language models, 2024. URL https://arxiv.org/abs/2404.08634. — [arXiv:2404.08634](https://arxiv.org/abs/2404.08634) [↗](https://arxiv.org/abs/2404.08634)
51. Noam Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019. URL http://arxiv.org/abs/1911.02150. [↗](http://arxiv.org/abs/1911.02150)
52. Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Raghavi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aaka… [find](https://scholar.google.com/scholar?q=Luca+Soldaini%2C+Rodney+Kinney%2C+Akshita+Bhagia%2C+Dustin+Schwenk%2C+David+Atkinson%2C+Russell+Authur%2C+Ben+Bogin%2C+Khyathi+Raghavi+Chandu%2C+Jennifer+Dumas%2C+Yanai+Elazar%2C+Valentin+Hofmann%2C+Ananya+Harsh+Jha%2C+Sachi)
53. Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL [https://openreview.net/forum?id=PxoFut…](https://openreview.net/forum?id=PxoFut…) [↗](https://openreview.net/forum?id=PxoFut…)
54. Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. [Helpsteer2-preference: Complementing ratings with preferences](https://md.research.elamur.ai/free/en/papers/wang-2024-helpsteer2-preference-complementing-ratings-preferences/). CoRR, abs/2410.01257, 2024. doi: 10.48550/ARXIV.2410.01257. URL https://doi.org/10.48550/arXiv.2410.01257. [↗](https://doi.org/10.48550/arXiv.2410.01257)
55. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Info…
56. Mengzhou Xia, Zexuan Zhong, and Danqi Chen. [Structured pruning learns compact and accurate models](https://md.research.elamur.ai/free/en/papers/xia-2022-structured-pruning-learns-compact-accurate/). In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin,…
57. Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL [https://openreview.net/fo…](https://openreview.net/fo…) [↗](https://openreview.net/fo…)
58. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL [https://openreview.net/forum?id=NG…](https://openreview.net/forum?id=NG…) [↗](https://openreview.net/forum?id=NG…)
59. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Informat… [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Tom+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+In+Alice+Oh%2C+Tristan+Naumann)
60. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Anna Korhonen, David R. Traum, and Lluı́s Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy… [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Anna+Korhonen%2C+David+R%2E+Traum%2C+and+Llu%C4%B1%CC%81s+Ma%CC%80rquez%2C+editors%2C+Proceeding)
61. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Morit…
62. Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multimodal model. CoRR, abs/2408.[11039, 2024](https://md.research.elamur.ai/free/en/papers/zhou-2024-transfusion-predict-next-token-diffuse/). doi: 10.48550/ARXIV.2408.11039. URL https://doi…
63. Barret Zoph and Quoc V. Le. [Neural architecture search with reinforcement learning](https://md.research.elamur.ai/free/en/papers/zoph-2016-neural-architecture-search-reinforcement-learning/). ArXiv, abs/1611.01578, 2016. URL [https://api.semanticscholar.org/CorpusID:12713052](https://api.semanticscholar.org/CorpusID:12713052). [↗](https://api.semanticscholar.org/CorpusID:12713052)

*Updated 3 November 2026*