---
title: "Densing Law of LLMs — paper by Xiao et al. (2024)"
description: "They introduce \"capability density.\" A family of reference models is used to fit a scaling function that… 58 references, cited by 16 papers here."
authors: ["Chaojun Xiao", "Jie Cai", "Weilin Zhao", "Guoyang Zeng", "Biyuan Lin", "Jie Zhou", "Zhi Zheng", "Xu Han", "Zhiyuan Liu", "Maosong Sun"]
year: 2024
venue: "arXiv:2412.04315"
source: "https://arxiv.org/abs/2412.04315"
doi: "10.48550/arxiv.2412.04315"
arxiv: "2412.04315"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xiao-2024-densing-law-llms/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xiao-2024-densing-law-llms/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/12/) / Densing Law of LLMs

# Densing Law of LLMs

**Authors:** [Chaojun Xiao](https://md.research.elamur.ai/free/en/authors/chaojun-xiao/), [Jie Cai](https://md.research.elamur.ai/free/en/authors/jie-cai/), [Weilin Zhao](https://md.research.elamur.ai/free/en/authors/weilin-zhao/), [Guoyang Zeng](https://md.research.elamur.ai/free/en/authors/guoyang-zeng/), [Biyuan Lin](https://md.research.elamur.ai/free/en/authors/biyuan-lin/), [Jie Zhou](https://md.research.elamur.ai/free/en/authors/jie-zhou/), [Zhi Zheng](https://md.research.elamur.ai/free/en/authors/zhi-zheng/), [Xu Han](https://md.research.elamur.ai/free/en/authors/xu-han/), [Zhiyuan Liu](https://md.research.elamur.ai/free/en/authors/zhiyuan-liu/) and [Maosong Sun](https://md.research.elamur.ai/free/en/authors/maosong-sun/) · **Year:** 2024 · **Venue:** arXiv:2412.04315 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2412.04315) · [PDF](https://arxiv.org/pdf/2412.04315) · [DOI 10.48550/arxiv.2412.04315](https://doi.org/10.48550/arxiv.2412.04315) · [arXiv 2412.04315](https://arxiv.org/abs/2412.04315) · [OpenAlex](https://openalex.org/W4405095461)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [16](https://md.research.elamur.ai/free/en/papers/xiao-2024-densing-law-llms/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Progress in LLMs has pulled in two directions: very large models that perform better, and compact models that are cheaper to run and fit on devices like phones. The authors ask whether models of different sizes can be compared on training quality, and whether efficiency follows a predictable trend over time. They introduce "capability density." A family of reference models is used to fit a scaling function that predicts downstream benchmark scores from parameter count. For any target model, the *effective parameter size* is the reference-model size that would reach the same score. Density is the effective size divided by the real size. They apply this to 29 open-source base LLMs released since 2023, scored on MMLU, BBH, MATH, HumanEval and MBPP. The highest density grows exponentially with release date, a pattern they call the "Densing Law." The authors argue that developers should optimize for density rather than raw performance. This would make capability cheaper to serve and reduce wasted compute. They caution that the measured rates depend on the chosen benchmarks and reference models.

## Key points
- The fitted trend is ln(ρ_max) = A·t + B, with A ≈ 0.007 and R² ≈ 0.93. This means maximum density doubles roughly every 3.3 months.
- In one example, MiniCPM-1-2.4B (Feb 2024) roughly matches Mistral-7B (Sep 2023) with about 35% of its parameters, four months later.
- The inference cost of reaching GPT-3.5-level performance fell by a factor of about 266.7 between January 2023 and the time of writing. Density growth was about 50% faster after ChatGPT's release.
- Combined with Moore's Law, the effective parameter size that fits on a chip of fixed cost grows faster than either trend alone. Common pruning and distillation methods, however, typically produce models with *lower* density than the originals.
- Stated limitations and future directions:
  - benchmark contamination and over-optimization to benchmarks;
  - extending density to multimodal models;
  - an inference-FLOPs-based density for reasoning-heavy models.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 439 works · 94,783 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,621 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works

## Cite

Cite as: Chaojun Xiao, Jie Cai, Weilin Zhao, Guoyang Zeng, Biyuan Lin, Jie Zhou, Zhi Zheng, Xu Han, Zhiyuan Liu and Maosong Sun. 2024. Densing Law of LLMs. arXiv:2412.04315.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{xiao2024densing,
  title = {{Densing Law of LLMs}},
  author = {Chaojun Xiao, Jie Cai, Weilin Zhao, Guoyang Zeng, Biyuan Lin, Jie Zhou, Zhi Zheng, Xu Han, Zhiyuan Liu and Maosong Sun},
  year = {2024},
  doi = {10.48550/arxiv.2412.04315},
  eprint = {2412.04315},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.04315}
}
```

- [← previous Cultural Commonsense Knowledge for Intercultural Dialogues](https://md.research.elamur.ai/free/en/papers/nguyen-2024-cultural-commonsense-knowledge-intercultural-dialogues/)
- [next → FormalAlign: Automated Alignment Evaluation for Autoformalization](https://md.research.elamur.ai/free/en/papers/lu-2024-formalalign-automated-alignment-evaluation-autoformalization/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/12/)

## References

58 references, 36 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. [Phi-3 technical report: A highly capable language model locally on your phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/). arXiv preprint arXiv:2404.14219, 2024.
2. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. [Gqa: Training generalized multi-query transformer models from multi-head checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4895–4901, 2023…
3. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. [The falcon series of open language models](https://md.research.elamur.ai/free/en/papers/almazrouei-2023-falcon-series-open-language-models/). arXiv preprint arXiv:2311.16867, 2023.
4. Guido Appenzeller. Welcome to llmflation – llm inference cost is going down fast. Blog, 2024. URL [https://a16z.com/llmflation-llm-inference-cost/](https://a16z.com/llmflation-llm-inference-cost/). [↗](https://a16z.com/llmflation-llm-inference-cost/)
5. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732, 2021.
6. Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, et al. Stable lm 2 1.6 b technical report. arXiv preprint arXiv:2402.[17834, 2024](https://md.research.elamur.ai/free/en/papers/bellagente-2024-stable-lm-2-1-6b/).
7. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Quincy Davis, D… — [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
8. Bradley C. A. Brown, Jordan Juravsky, Ryan Saul Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. [Large language monkeys: Scaling inference compute with repeated sampling](https://md.research.elamur.ai/free/en/papers/brown-2024-large-language-monkeys-scaling-inference/). CoRR, abs/2407.21787, 2024. doi: 10.48550/ARXIV.2407.21787. URL https://doi.org/10.48550/arXiv.2407.217… [↗](https://doi.org/10.48550/arXiv.2407.217…)
9. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
10. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374, 2021.
11. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). Journal of Machine Learning Research, 24(240):1–113, 2023.
12. Tri Dao. [Flashattention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). arXiv preprint arXiv:2307.08691, 2023.
13. Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359, 2022. [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Dan+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+Flashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A16344)
14. Ricardo Dominguez-Olmedo, Florian E Dorner, and Moritz Hardt. Training on the test task confounds evaluation and emergence. arXiv preprint arXiv:2407.07890, 2024. — [arXiv:2407.07890](https://arxiv.org/abs/2407.07890) [find](https://scholar.google.com/scholar?q=Ricardo+Dominguez%2DOlmedo%2C+Florian+E+Dorner%2C+and+Moritz+Hardt%2E+Training+on+the+test+task+confounds+evaluation+and+emergence%2E+arXiv+preprint+arXiv%3A2407%2E07890%2C+2024%2E+%E2%80%94+%5BarXiv%3A2407%2E07890%5D%28https%3A%2F%2Farxiv%2Eor)
15. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783, 2024.
16. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023. [find](https://scholar.google.com/scholar?q=Suriya+Gunasekar%2C+Yi+Zhang%2C+Jyoti+Aneja%2C+Caio+C%C3%A9sar+Teodoro+Mendes%2C+Allie+Del+Giorno%2C+Sivakanth+Gopi%2C+Mojan+Javaheripi%2C+Piero+Kauffmann%2C+Gustavo+de+Rosa%2C+Olli+Saarikivi%2C+et+al%2E+Textbooks+are+all+you+n)
17. Tom Gunter, Zirui Wang, Chong Wang, Ruoming Pang, Andy Narayanan, Aonan Zhang, Bowen Zhang, Chen Chen, Chung-Cheng Chiu, David Qiu, et al. Apple intelligence foundation language models. CoRR, abs/2407.21075, 2024. doi: 10.48550/ARXIV.2407.21075. URL https://doi.org/10.48550/arXiv.2407.21075. — [doi:10.48550/arxiv.2407.21075](https://doi.org/10.48550/arxiv.2407.21075) [↗](https://doi.org/10.48550/arXiv.2407.21075)
18. Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang, Wentao Han, Minlie Huang, Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song, Jie Tang, Ji-Rong Wen, Jinhui Yuan, Wayne Xin Zhao, and Jun Zhu. Pre-trained model… [find](https://scholar.google.com/scholar?q=Xu+Han%2C+Zhengyan+Zhang%2C+Ning+Ding%2C+Yuxian+Gu%2C+Xiao+Liu%2C+Yuqi+Huo%2C+Jiezhong+Qiu%2C+Yuan+Yao%2C+Ao+Zhang%2C+Liang+Zhang%2C+Wentao+Han%2C+Minlie+Huang%2C+Qin+Jin%2C+Yanyan+Lan%2C+Yang+Liu%2C+Zhiyuan+Liu%2C+Zhiwu+Lu%2C+Xipeng+)
19. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations, 2020.
20. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
21. Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for auto… — [Scaling Laws for Autoregressive Generative Modeling](https://md.research.elamur.ai/free/en/papers/henighan-2020-scaling-laws-autoregressive-generative-modeling/)
22. Marius Hobbhahn, Lennart Heim, and Gökçe Aydos. Trends in machine learning hardware, 2023. URL [https://epoch.ai/blog/trends-in-machine-learning-hardware](https://epoch.ai/blog/trends-in-machine-learning-hardware). Accessed: 2024-12-05. [↗](https://epoch.ai/blog/trends-in-machine-learning-hardware)
23. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — [Training Compute-Optimal Large Language Models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
24. Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. [Minicpm: Unveiling the potential of small language models with scalable training strategies](https://md.research.elamur.ai/free/en/papers/hu-2024-minicpm-unveiling-potential-small-language/). CoRR, abs/2404.06395, 2024. doi: 10.48550/ARXIV.2404.06395. URL https://doi.org/10.…
25. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+Mistral+7b)
26. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). CoRR, abs/2001.08361, 2020. URL https://arxiv.org/abs/2001.08361. [↗](https://arxiv.org/abs/2001.08361)
27. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611–626, 2023.
28. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. [Retrieval-augmented generation for knowledge-intensive nlp tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). Advances in Neural Information Processing Systems, 33: 9459–9474, 2020.
29. Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. [Textbooks are all you need ii: phi-1.5 technical report](https://md.research.elamur.ai/free/en/papers/li-2023-textbooks-all-need-ii-phi/). arXiv preprint arXiv:2309.05463, 2023.
30. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. [Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation](https://md.research.elamur.ai/free/en/papers/liu-2023-your-code-generated-chatgpt-really/). Advances in Neural Information Processing Systems, 36, 2024.
31. Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. [Deja vu: Contextual sparsity for efficient llms at inference time](https://md.research.elamur.ai/free/en/papers/liu-2023-deja-vu-contextual-sparsity-efficient/). In International Conference on Machine Learning, pp. 22137–22176. PMLR, 2023.
32. Xinyin Ma, Gongfan Fang, and Xinchao Wang. [Llm-pruner: On the structural pruning of large language models](https://md.research.elamur.ai/free/en/papers/ma-2023-llm-pruner-structural-pruning-large/). Advances in neural information processing systems, 36:21702–21720, 2023.
33. Gordon E Moore. Cramming more components onto integrated circuits. Electronics, 1965. [find](https://scholar.google.com/scholar?q=Gordon+E+Moore%2E+Cramming+more+components+onto+integrated+circuits%2E+Electronics%2C+1965%2E)
34. Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation. arXiv preprint arXiv:2407.14679, 2024. [find](https://scholar.google.com/scholar?q=Saurav+Muralidharan%2C+Sharath+Turuvekere+Sreenivas%2C+Raviraj+Joshi%2C+Marcin+Chochowski%2C+Mostofa+Patwary%2C+Mohammad+Shoeybi%2C+Bryan+Catanzaro%2C+Jan+Kautz%2C+and+Pavlo+Molchanov%2E+Compact+language+models+via+pru)
35. OpenAI. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). CoRR, abs/2303.08774, 2023.
36. OpenAI. Learning to reason with llms. Technical Report, 2024a. URL [https://openai.com/](https://openai.com/) index/gpt-4o-mini-advancing-cost-efficient-intelligence/. [↗](https://openai.com/)
37. OpenAI. Gpt-4o mini: advancing cost-efficient intelligence. Technical Report, 2024b. URL [https://openai.com/index/learning-to-reason-with-llms/](https://openai.com/index/learning-to-reason-with-llms/). [↗](https://openai.com/index/learning-to-reason-with-llms/)
38. OpenCompass. Opencompass: A universal evaluation platform for foundation models. [https://github.com/open-compass/opencompass](https://github.com/open-compass/opencompass), 2023. [↗](https://github.com/open-compass/opencompass)
39. Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. [Pre-trained models for natural language processing: A survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/). CoRR, abs/2003.08271, 2020.
40. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. [find](https://scholar.google.com/scholar?q=Jack+W+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+et+al%2E+Scaling+language+models%3A+Methods%2C+analysi)
41. Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. [Beyond chinchilla-optimal: Accounting for inference in language model scaling laws](https://md.research.elamur.ai/free/en/papers/sardana-2024-chinchilla-optimal-accounting-inference-language/). In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024.
42. OpenReview.net, 2024. URL [https://openreview.net/forum?id=0bmXrtTDUu](https://openreview.net/forum?id=0bmXrtTDUu). [↗](https://openreview.net/forum?id=0bmXrtTDUu)
43. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. [Scaling LLM test-time compute optimally can be more effective than scaling model parameters](https://md.research.elamur.ai/free/en/papers/snell-2024-scaling-llm-test-time-compute/). CoRR, abs/2408.03314, 2024. doi: 10.48550/ ARXIV.2408.03314. URL https://doi.org/10.48550/arXiv.2408.03314. [↗](https://doi.org/10.48550/arXiv.2408.03314)
44. Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. [Powerinfer: Fast large language model serving with a consumer-grade gpu](https://md.research.elamur.ai/free/en/papers/song-2023-powerinfer-fast-large-language-model/). arXiv preprint arXiv:2312.12456, 2023.
45. Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024. [find](https://scholar.google.com/scholar?q=Mingjie+Sun%2C+Zhuang+Liu%2C+Anna+Bair%2C+and+J+Zico+Kolter%2E+A+simple+and+effective+pruning+approach+for+large+language+models%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+2024%2E)
46. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, et al. [Challenging big-bench tasks and whether chain-of-thought can solve them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/). In Findings of the Association for Computational Linguistics: ACL 2023, pp. 1…
47. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024. [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Morgane+Riviere%2C+Shreya+Pathak%2C+Pier+Giuseppe+Sessa%2C+Cassidy+Hardin%2C+Surya+Bhupatiraju%2C+L%C3%A9onard+Hussenot%2C+Thomas+Mesnard%2C+Bobak+Shahriari%2C+Alexandre+Ram%C3%A9%2C+et+al%2E+Gemma+2%3A+Improving+open+la)
48. MosaicML NLP Team. Introducing mpt-30b: Raising the bar for open-source foundation models, 2023. URL www.mosaicml.com/blog/mpt-30b. Accessed: 2023-06-22. [find](https://scholar.google.com/scholar?q=MosaicML+NLP+Team%2E+Introducing+mpt%2D30b%3A+Raising+the+bar+for+open%2Dsource+foundation+models%2C+2023%2E+URL+www%2Emosaicml%2Ecom%2Fblog%2Fmpt%2D30b%2E+Accessed%3A+2023%2D06%2D22%2E)
49. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). CoRR, abs/2302…
50. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023b.
51. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). Transactions on Machine Learning Research, 2022a.
52. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824–24837, 2022b.
53. Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, et al. [Skywork: A more open bilingual foundation model](https://md.research.elamur.ai/free/en/papers/wei-2023-skywork-more-open-bilingual/). arXiv preprint arXiv:2310.19341, 2023.
54. Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116, 2024. — [arXiv:2402.13116](https://arxiv.org/abs/2402.13116) [find](https://scholar.google.com/scholar?q=Xiaohan+Xu%2C+Ming+Li%2C+Chongyang+Tao%2C+Tao+Shen%2C+Reynold+Cheng%2C+Jinyang+Li%2C+Can+Xu%2C+Dacheng+Tao%2C+and+Tianyi+Zhou%2E+A+survey+on+knowledge+distillation+of+large+language+models%2E+arXiv+preprint+arXiv%3A2402%2E13)
55. Zhenliang Xue, Yixin Song, Zeyu Mi, Le Chen, Yubin Xia, and Haibo Chen. Powerinfer-2: Fast large language model inference on a smartphone. CoRR, abs/2406.06282, 2024. doi: 10.48550/ ARXIV.2406.06282. URL https://doi.org/10.48550/arXiv.2406.06282. [↗](https://doi.org/10.48550/arXiv.2406.06282)
56. Chuanpeng Yang, Yao Zhu, Wang Lu, Yidong Wang, Qian Chen, Chenlong Gao, Bingjie Yan, and Yiqiang Chen. Survey on knowledge distillation for large language models: Methods, evaluation, and application. ACM Transactions on Intelligent Systems and Technology, 2024. [find](https://scholar.google.com/scholar?q=Chuanpeng+Yang%2C+Yao+Zhu%2C+Wang+Lu%2C+Yidong+Wang%2C+Qian+Chen%2C+Chenlong+Gao%2C+Bingjie+Yan%2C+and+Yiqiang+Chen%2E+Survey+on+knowledge+distillation+for+large+language+models%3A+Methods%2C+evaluation%2C+and+application%2E)
57. Longfei Yun, Yonghao Zhuang, Yao Fu, Eric P. Xing, and Hao Zhang. Toward inference-optimal mixture-of-expert large language models. CoRR, abs/2404.02852, 2024. doi: 10.48550/ARXIV. 2404.02852. URL https://doi.org/10.48550/arXiv.2404.02852. — [doi:10.48550/arxiv](https://doi.org/10.48550/arxiv) [↗](https://doi.org/10.48550/arXiv.2404.02852)
58. Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. [Tinyllama: An open-source small language model](https://md.research.elamur.ai/free/en/papers/zhang-2024-tinyllama-open-source-small-language/). arXiv preprint arXiv:2401.02385, 2024.

*Updated 3 November 2026*