---
title: "Distillation Scaling Laws — Busbridge 2025"
description: "Small language models are cheap to serve, but making them strong through heavy overtraining needs trillions of… 156 references, cited by 23 papers here."
authors: ["Dan Busbridge", "Amitis Shidani", "Floris Weers", "Jason Ramapuram", "Etai Littwin", "Russ Webb"]
year: 2025
venue: "arXiv:2502.08606"
source: "https://arxiv.org/abs/2502.08606"
doi: "10.48550/arxiv.2502.08606"
arxiv: "2502.08606"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/busbridge-2025-distillation-scaling-laws/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/busbridge-2025-distillation-scaling-laws/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/) / Distillation Scaling Laws

# Distillation Scaling Laws

**Authors:** Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin and Russ Webb · **Year:** 2025 · **Venue:** arXiv:2502.08606 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2502.08606) · [PDF](https://arxiv.org/pdf/2502.08606) · [DOI 10.48550/arxiv.2502.08606](https://doi.org/10.48550/arxiv.2502.08606) · [arXiv 2502.08606](https://arxiv.org/abs/2502.08606) · [OpenAlex](https://openalex.org/W4407570807)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [23](https://md.research.elamur.ai/free/en/papers/busbridge-2025-distillation-scaling-laws/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Small language models are cheap to serve, but making them strong through heavy overtraining needs trillions of tokens. Distillation, where a larger teacher supplies training targets for a smaller student, is a possible shortcut. Practitioners still lack guidance on how to split compute between teacher and student, or whether distillation beats ordinary supervised pretraining at the same budget. The authors, from Apple and the University of Oxford, ran a large controlled study. Transformer students and teachers ranged from 143M to 12.6B parameters and were trained on a few billion up to 512B tokens. From these runs they fitted a distillation scaling law that predicts student cross-entropy from student size, distillation tokens and teacher quality. The law also holds when extrapolated beyond the fitted range. They then derived compute-optimal recipes for two cases: a teacher is already available, or one must be trained first. The work matters because it lets teams plan distillation runs ahead of time and avoid wasting compute. It also shows clearly when distillation is the wrong choice.

## Key points
- The teacher's size N_T and its training tokens D_T affect the student only through the teacher's cross-entropy L_T. This removes two dimensions from hyperparameter search.
- The law captures the "capacity gap", where a stronger teacher can produce a worse student. The authors explain it as a gap in learning capacity, covering both hypothesis space and optimization ability, rather than in size alone. A student can sometimes outperform its teacher.
- Distillation beats supervised learning only under two conditions:
  - the student's compute or token budget stays below a threshold that grows with student size;
  - the teacher already exists or will be reused for more than one student.
- If a single student is needed and the teacher must be trained from scratch, supervised pretraining is usually the better option. With unlimited data or compute, distillation never achieves lower cross-entropy than supervised learning.
- The law was fitted on students with L_S > 2.3 and extrapolated to stronger students. The paper was published at ICML 2025 (arXiv:2502.08606).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Mathematics & statistics](https://md.research.elamur.ai/free/en/shelves/mathematics/)

Topics: [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/)

## Cite

Cite as: Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin and Russ Webb. 2025. Distillation Scaling Laws. arXiv:2502.08606.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{busbridge2025distillation,
  title = {{Distillation Scaling Laws}},
  author = {Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin and Russ Webb},
  year = {2025},
  doi = {10.48550/arxiv.2502.08606},
  eprint = {2502.08606},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.08606}
}
```

- [← previous An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)](https://md.research.elamur.ai/free/en/papers/burchell-2025-expanded-massive-multilingual-dataset-high/)
- [next → MedArabiQ: Benchmarking Large Language Models on Arabic Medical Tasks](https://md.research.elamur.ai/free/en/papers/daoud-2025-medarabiq-benchmarking-large-language-models/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/)

## References

156 references, 37 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abdin, M. I., Aneja, J., Behl, H. S., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmann, P., Lee, J. R., Lee, Y. T., Li, Y., Liu, W., Mendes, C. C. T., Nguyen, A., Price, E., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Wang, X., Ward, R., Wu, Y., Yu, D… [our card](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-4-technical-report/)
2. Abdin, M. I., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H. S., Benhaim, A., Bilenko, M., Bjorck, J., Bubeck, S., Cai, M., Mendes, C. C. T., Chen, W., Chaudhary, V., Chopra, P., Giorno, A. D., de Rosa, G., Dixon, M., Eldan, R., Iter… [our card](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/)
3. Abnar, S., Shah, H., Busbridge, D., Ali, A. M. E., Susskind, J., and Thilak, V. Parameters vs flops: Scaling laws for optimal sparsity for mixture-of-experts language models, 2025. URL https://arxiv.org/abs/2501.12370. — [arXiv:2501.12370](https://arxiv.org/abs/2501.12370) [↗](https://arxiv.org/abs/2501.12370)
4. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. GQA: training generalized multi-query transformer models from multi-head checkpoints. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Proce… [our card](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/)
5. Aitchison, L. Why you don’t overfit, and don’t need bayes if you only train for one epoch. CoRR, abs/2411.14478, 2024. doi: 10.48550/ARXIV.2411.14478. URL https://doi.org/10.48550/arXiv.2411.14478. — [doi:10.48550/arxiv.2411.14478](https://doi.org/10.48550/arxiv.2411.14478) [↗](https://doi.org/10.48550/arXiv.2411.14478)
6. Amara, I., Sepahvand, N. M., Meyer, B. H., Gross, W. J., and Clark, J. J. BD-KD: balancing the divergences for online knowledge distillation. CoRR, abs/2212.12965, 2022. doi: 10.48550/ARXIV.2212.12965. URL https://doi.org/10.48550/arXiv.2212.12965. — [doi:10.48550/arxiv.2212.12965](https://doi.org/10.48550/arxiv.2212.12965) [↗](https://doi.org/10.48550/arXiv.2212.12965)
7. Apple. The axlearn library for deep learning., 2023. URL [https://github.com/apple/axlearn](https://github.com/apple/axlearn). Accessed: 2025-02-11. [↗](https://github.com/apple/axlearn)
8. Bahri, Y., Dyer, E., Kaplan, J., Lee, J., and Sharma, U. Explaining neural scaling laws. CoRR, abs/2102.06701, 2021. URL https://arxiv.org/abs/2102.06701. — [arXiv:2102.06701](https://arxiv.org/abs/2102.06701) [↗](https://arxiv.org/abs/2102.06701)
9. Barnett, M. An empirical study of scaling laws for transfer. CoRR, abs/2408.16947, 2024. doi: 10.48550/ARXIV.2408.16947. URL https://doi.org/10.48550/arXiv.2408.16947. — [doi:10.48550/arxiv.2408.16947](https://doi.org/10.48550/arxiv.2408.16947) [↗](https://doi.org/10.48550/arXiv.2408.16947)
10. Berant, J., Chou, A., Frostig, R., and Liang, P. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a S… [our card](https://md.research.elamur.ai/free/en/papers/berant-2013-semantic-parsing-freebase-question-answer/)
11. ACL, 2013. URL https://aclanthology.org/D13-1160/. [↗](https://aclanthology.org/D13-1160/) [our card](https://md.research.elamur.ai/free/en/papers/berant-2013-semantic-parsing-freebase-question-answer/)
12. Besiroglu, T., Erdil, E., Barnett, M., and You, J. Chinchilla scaling: A replication attempt. CoRR, abs/2404.10102, 2024. doi: 10.48550/ARXIV.2404.10102. URL https://doi.org/10.48550/arXiv.2404.10102. [↗](https://doi.org/10.48550/arXiv.2404.10102) [our card](https://md.research.elamur.ai/free/en/papers/besiroglu-2024-chinchilla-scaling-replication-attempt/)
13. Beyer, L., Zhai, X., Royer, A., Markeeva, L., Anil, R., and Kolesnikov, A. Knowledge distillation: A good teacher is patient and consistent. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 10915–10924. [find](https://scholar.google.com/scholar?q=Beyer%2C+L%2E%2C+Zhai%2C+X%2E%2C+Royer%2C+A%2E%2C+Markeeva%2C+L%2E%2C+Anil%2C+R%2E%2C+and+Kolesnikov%2C+A%2E+Knowledge+distillation%3A+A+good+teacher+is+patient+and+consistent%2E+In+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recog)
14. IEEE, 2022. doi: 10.1109/CVPR52688.2022.01065. URL https://doi.org/10.1109/CVPR52688.2022.01065. — [doi:10.1109/cvpr52688.2022.01065](https://doi.org/10.1109/cvpr52688.2022.01065) [↗](https://doi.org/10.1109/CVPR52688.2022.01065)
15. Bhakthavatsalam, S., Khashabi, D., Khot, T., Mishra, B. D., Richardson, K., Sabharwal, A., Schoenick, C., Tafjord, O., and Clark, P. Think you have solved direct-answer question answering? try arc-da, the direct-answer AI2 reasoning challenge. CoRR, abs/2102.03315, 2021. URL https://arxiv.org/abs/21… — [arXiv:2102.03315](https://arxiv.org/abs/2102.03315) [↗](https://arxiv.org/abs/21…)
16. Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., Gao, H., Gao, K., Gao, W., Ge, R., Guan, K., Guo, D., Guo, J., Hao, G., Hao, Z., He, Y., Hu, W., Huang, P., Li, E., Li, G., Li, J., Li, Y., Li, Y. K., Liang, W., Lin, F., Liu, A. X., Liu, B., Liu, W., Liu, X… [find](https://scholar.google.com/scholar?q=Bi%2C+X%2E%2C+Chen%2C+D%2E%2C+Chen%2C+G%2E%2C+Chen%2C+S%2E%2C+Dai%2C+D%2E%2C+Deng%2C+C%2E%2C+Ding%2C+H%2E%2C+Dong%2C+K%2E%2C+Du%2C+Q%2E%2C+Fu%2C+Z%2E%2C+Gao%2C+H%2E%2C+Gao%2C+K%2E%2C+Gao%2C+W%2E%2C+Ge%2C+R%2E%2C+Guan%2C+K%2E%2C+Guo%2C+D%2E%2C+Guo%2C+J%2E%2C+Hao%2C+G%2E%2C+Hao%2C+Z%2E%2C+He%2C+Y%2E%2C+Hu%2C+W%2E%2C+Huang%2C+P%2E%2C)
17. Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth… [our card](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/)
18. AAAI Press, 2020. doi: 10.1609/AAAI.V34I05.6239. URL https://doi.org/10.1609/aaai. v34i05.6239. — [doi:10.1609/aaai.v34i05.6239](https://doi.org/10.1609/aaai.v34i05.6239) [↗](https://doi.org/10.1609/aaai)
19. Blasiok, J., Gopalan, P., Hu, L., and Nakkiran, P. When does optimizing a proper loss yield calibration? In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Syste… [find](https://scholar.google.com/scholar?q=Blasiok%2C+J%2E%2C+Gopalan%2C+P%2E%2C+Hu%2C+L%2E%2C+and+Nakkiran%2C+P%2E+When+does+optimizing+a+proper+loss+yield+calibration%3F+In+Oh%2C+A%2E%2C+Naumann%2C+T%2E%2C+Globerson%2C+A%2E%2C+Saenko%2C+K%2E%2C+Hardt%2C+M%2E%2C+and+Levine%2C+S%2E+%28eds%2E%29%2C+Advances+i)
20. Blondel, M. and Roulet, V. The elements of differentiable programming. CoRR, abs/2403.14606, 2024. doi: 10.48550/ARXIV.2403.14606. URL https://doi.org/10.48550/arXiv.2403.14606. — [doi:10.48550/arxiv.2403.14606](https://doi.org/10.48550/arxiv.2403.14606) [↗](https://doi.org/10.48550/arXiv.2403.14606)
21. Brown, B. C. A., Juravsky, J., Ehrlich, R. S., Clark, R., Le, Q. V., Ré, C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling. CoRR, abs/2407.21787, 2024. doi: 10.48550/ARXIV.2407.21787. URL https://doi.org/10.48550/arXiv.2407.21787. [↗](https://doi.org/10.48550/arXiv.2407.21787) [our card](https://md.research.elamur.ai/free/en/papers/brown-2024-large-language-monkeys-scaling-inference/)
22. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Ch… [our card](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
23. Bucila, C., Caruana, R., and Niculescu-Mizil, A. Model compression. In Eliassi-Rad, T., Ungar, L. H., Craven, M., and Gunopulos, D. (eds.), Proceedings of the Twelfth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Philadelphia, PA, USA, August 20-23, 2006, pp. 535–541. [find](https://scholar.google.com/scholar?q=Bucila%2C+C%2E%2C+Caruana%2C+R%2E%2C+and+Niculescu%2DMizil%2C+A%2E+Model+compression%2E+In+Eliassi%2DRad%2C+T%2E%2C+Ungar%2C+L%2E+H%2E%2C+Craven%2C+M%2E%2C+and+Gunopulos%2C+D%2E+%28eds%2E%29%2C+Proceedings+of+the+Twelfth+ACM+SIGKDD+International+Conferen)
24. ACM, 2006. doi: 10.1145/1150402.1150464. URL https://doi.org/10.1145/1150402.1150464. — [doi:10.1145/1150402.1150464](https://doi.org/10.1145/1150402.1150464) [↗](https://doi.org/10.1145/1150402.1150464)
25. Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., Sutskever, I., and Wu, J. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. In Forty-first International Conference on Machine Learning,… [our card](https://md.research.elamur.ai/free/en/papers/burns-2023-weak-strong-generalization-eliciting-strong/)
26. OpenReview.net, 2024. URL [https://openreview.net/forum?id=ghNRg2mEgN](https://openreview.net/forum?id=ghNRg2mEgN). [↗](https://openreview.net/forum?id=ghNRg2mEgN)
27. Caballero, E., Gupta, K., Rish, I., and Krueger, D. Broken neural scaling laws. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. [find](https://scholar.google.com/scholar?q=Caballero%2C+E%2E%2C+Gupta%2C+K%2E%2C+Rish%2C+I%2E%2C+and+Krueger%2C+D%2E+Broken+neural+scaling+laws%2E+In+The+Eleventh+International+Conference+on+Learning+Representations%2C+ICLR+2023%2C+Kigali%2C+Rwanda%2C+May+1%2D5%2C+2023%2E)
28. OpenReview.net, 2023. URL [https://openreview.net/forum?id=sckjveqlCZ](https://openreview.net/forum?id=sckjveqlCZ). [↗](https://openreview.net/forum?id=sckjveqlCZ)
29. Carrell, A. M., Mallinar, N., Lucas, J., and Nakkiran, P. The calibration generalization gap. CoRR, abs/2210.01964, 2022. doi: 10.48550/ARXIV.2210.01964. URL https://doi.org/10.48550/arXiv.2210.01964. — [doi:10.48550/arxiv.2210.01964](https://doi.org/10.48550/arxiv.2210.01964) [↗](https://doi.org/10.48550/arXiv.2210.01964)
30. CERN. Cern data centre: Key information, March 2018. URL [http://information-technology.web.cern.ch/sites/information-technology.web.cern.ch/files/CERNDataCentre_KeyInformation_02March2018V1.pdf](http://information-technology.web.cern.ch/sites/information-technology.web.cern.ch/files/CERNDataCentre_KeyInformation_02March2018V1.pdf). Accessed: 2025-01-29. [↗](http://information-technology.web.cern.ch/sites/information-technology.web.cern.ch/files/CERNDataCentre_KeyInformation_02March2018V1.pdf)
31. Chien, A. A., Lin, L., Nguyen, H., Rao, V., Sharma, T., and Wijayawardana, R. Reducing the carbon impact of generative AI inference (today and in 2035). In Porter, G., Anderson, T., Chien, A. A., Eilam, T., Josephson, C., and Park, J. (eds.), Proceedings of the 2nd Workshop on Sustainable Computer S… [find](https://scholar.google.com/scholar?q=Chien%2C+A%2E+A%2E%2C+Lin%2C+L%2E%2C+Nguyen%2C+H%2E%2C+Rao%2C+V%2E%2C+Sharma%2C+T%2E%2C+and+Wijayawardana%2C+R%2E+Reducing+the+carbon+impact+of+generative+AI+inference+%28today+and+in+2035%29%2E+In+Porter%2C+G%2E%2C+Anderson%2C+T%2E%2C+Chien%2C+A%2E+A%2E%2C+Eila)
32. ACM, 2023. doi: 10.1145/3604930.3605705. URL https://doi.org/10.1145/3604930.3605705. — [doi:10.1145/3604930.3605705](https://doi.org/10.1145/3604930.3605705) [↗](https://doi.org/10.1145/3604930.3605705)
33. Cho, J. H. and Hariharan, B. On the efficacy of knowledge distillation. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pp. 4793–4801. [find](https://scholar.google.com/scholar?q=Cho%2C+J%2E+H%2E+and+Hariharan%2C+B%2E+On+the+efficacy+of+knowledge+distillation%2E+In+2019+IEEE%2FCVF+International+Conference+on+Computer+Vision%2C+ICCV+2019%2C+Seoul%2C+Korea+%28South%29%2C+October+27+%2D+November+2%2C+2019%2C+pp)
34. IEEE, 2019. doi: 10.1109/ICCV.2019.00489. URL https://doi.org/10.1109/ICCV.2019.00489. — [doi:10.1109/iccv.2019.00489](https://doi.org/10.1109/iccv.2019.00489) [↗](https://doi.org/10.1109/ICCV.2019.00489)
35. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Aust… [our card](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
36. Clark, A., de Las Casas, D., Guy, A., Mensch, A., Paganini, M., Hoffmann, J., Damoc, B., Hechtman, B. A., Cai, T., Borgeaud, S., van den Driessche, G., Rutherford, E., Hennigan, T., Johnson, M. J., Cassirer, A., Jones, C., Buchatskaya, E., Budden, D., Sifre, L., Osindero, S., Vinyals, O., Ranzato, M… [find](https://scholar.google.com/scholar?q=Clark%2C+A%2E%2C+de+Las+Casas%2C+D%2E%2C+Guy%2C+A%2E%2C+Mensch%2C+A%2E%2C+Paganini%2C+M%2E%2C+Hoffmann%2C+J%2E%2C+Damoc%2C+B%2E%2C+Hechtman%2C+B%2E+A%2E%2C+Cai%2C+T%2E%2C+Borgeaud%2C+S%2E%2C+van+den+Driessche%2C+G%2E%2C+Rutherford%2C+E%2E%2C+Hennigan%2C+T%2E%2C+Johnson%2C+M%2E+J%2E%2C+Ca)
37. PMLR, 2022. URL [https://proceedings.mlr.press/v162/clark22a.html](https://proceedings.mlr.press/v162/clark22a.html). [↗](https://proceedings.mlr.press/v162/clark22a.html)
38. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.14168. [↗](https://arxiv.org/abs/2110.14168) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
39. Dao, T. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. [find](https://scholar.google.com/scholar?q=Dao%2C+T%2E+Flashattention%2D2%3A+Faster+attention+with+better+parallelism+and+work+partitioning%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+ICLR+2024%2C+Vienna%2C+Austria%2C+May+7%2D11%2C+202)
40. OpenReview.net, 2024. URL [https://openreview.net/forum?id=mZn2Xyh9Ec](https://openreview.net/forum?id=mZn2Xyh9Ec). [↗](https://openreview.net/forum?id=mZn2Xyh9Ec)
41. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Infor… [find](https://scholar.google.com/scholar?q=Dao%2C+T%2E%2C+Fu%2C+D%2E+Y%2E%2C+Ermon%2C+S%2E%2C+Rudra%2C+A%2E%2C+and+R%C3%A9%2C+C%2E+Flashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2E+In+Koyejo%2C+S%2E%2C+Mohamed%2C+S%2E%2C+Agarwal%2C+A%2E%2C+Belgrave%2C+D%2E%2C+Cho%2C+K%2E%2C+and+Oh)
42. DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang, H., Zhang, H., Ding, H., Xin, H., Gao, H., Li, H.,… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Liu%2C+A%2E%2C+Feng%2C+B%2E%2C+Xue%2C+B%2E%2C+Wang%2C+B%2E%2C+Wu%2C+B%2E%2C+Lu%2C+C%2E%2C+Zhao%2C+C%2E%2C+Deng%2C+C%2E%2C+Zhang%2C+C%2E%2C+Ruan%2C+C%2E%2C+Dai%2C+D%2E%2C+Guo%2C+D%2E%2C+Yang%2C+D%2E%2C+Chen%2C+D%2E%2C+Ji%2C+D%2E%2C+Li%2C+E%2E%2C+Lin%2C+F%2E%2C+Dai%2C+F%2E%2C+Luo%2C+F%2E%2C+Hao%2C+G%2E%2C+Ch)
43. Dehghani, M., Arnab, A., Beyer, L., Vaswani, A., and Tay, Y. The efficiency misnomer. CoRR, abs/2110.12894, 2021. URL https://arxiv.org/abs/2110.12894. — [arXiv:2110.12894](https://arxiv.org/abs/2110.12894) [↗](https://arxiv.org/abs/2110.12894)
44. Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248–255. [find](https://scholar.google.com/scholar?q=Deng%2C+J%2E%2C+Dong%2C+W%2E%2C+Socher%2C+R%2E%2C+Li%2C+L%2E%2C+Li%2C+K%2E%2C+and+Fei%2DFei%2C+L%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+2009+IEEE+Computer+Society+Conference+on+Computer+Vision+and+Pattern+Recognition)
45. IEEE Computer Society, 2009. doi: 10.1109/CVPR.2009.5206848. URL https://doi.org/10.1109/CVPR.2009.5206848. — [doi:10.1109/cvpr.2009.5206848](https://doi.org/10.1109/cvpr.2009.5206848) [↗](https://doi.org/10.1109/CVPR.2009.5206848)
46. Dey, N., Gosal, G., Chen, Z., Khachane, H., Marshall, W., Pathria, R., Tom, M., and Hestness, J. Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster. CoRR, abs/2304.03208, 2023. doi: 10.48550/ARXIV.2304.03208. URL https://doi.org/10.48550/arXiv.2304.03208. — [doi:10.48550/arxiv.2304.03208](https://doi.org/10.48550/arxiv.2304.03208) [↗](https://doi.org/10.48550/arXiv.2304.03208)
47. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Rozière, B., Biron, B., Tang… [find](https://scholar.google.com/scholar?q=Dubey%2C+A%2E%2C+Jauhri%2C+A%2E%2C+Pandey%2C+A%2E%2C+Kadian%2C+A%2E%2C+Al%2DDahle%2C+A%2E%2C+Letman%2C+A%2E%2C+Mathur%2C+A%2E%2C+Schelten%2C+A%2E%2C+Yang%2C+A%2E%2C+Fan%2C+A%2E%2C+Goyal%2C+A%2E%2C+Hartshorn%2C+A%2E%2C+Yang%2C+A%2E%2C+Mitra%2C+A%2E%2C+Sravankumar%2C+A%2E%2C+Korenev%2C+A%2E%2C+Hinsv)
48. Epoch AI. Key trends and figures in machine learning, 2023. URL [https://epoch.ai/trends](https://epoch.ai/trends). Accessed: 2025-02-11. [↗](https://epoch.ai/trends)
49. Fan, W., Lu, S., Li, X., Zhan, D., and Gan, L. Revisit the essence of distilling knowledge through calibration. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. [find](https://scholar.google.com/scholar?q=Fan%2C+W%2E%2C+Lu%2C+S%2E%2C+Li%2C+X%2E%2C+Zhan%2C+D%2E%2C+and+Gan%2C+L%2E+Revisit+the+essence+of+distilling+knowledge+through+calibration%2E+In+Forty%2Dfirst+International+Conference+on+Machine+Learning%2C+ICML+2024%2C+Vienna%2C+Austria%2C)
50. OpenReview.net, 2024. URL [https://openreview.net/forum?id=NZgbwzaOIx](https://openreview.net/forum?id=NZgbwzaOIx). [↗](https://openreview.net/forum?id=NZgbwzaOIx)
51. Furlanello, T., Lipton, Z. C., Tschannen, M., Itti, L., and Anandkumar, A. Born-again neural networks. In Dy, J. G. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings… [find](https://scholar.google.com/scholar?q=Furlanello%2C+T%2E%2C+Lipton%2C+Z%2E+C%2E%2C+Tschannen%2C+M%2E%2C+Itti%2C+L%2E%2C+and+Anandkumar%2C+A%2E+Born%2Dagain+neural+networks%2E+In+Dy%2C+J%2E+G%2E+and+Krause%2C+A%2E+%28eds%2E%29%2C+Proceedings+of+the+35th+International+Conference+on+Machine+L)
52. PMLR, 2018. URL [http://proceedings.mlr.press/v80/furlanello18a.html](http://proceedings.mlr.press/v80/furlanello18a.html). [↗](http://proceedings.mlr.press/v80/furlanello18a.html)
53. Gadre, S. Y., Smyrnis, G., Shankar, V., Gururangan, S., Wortsman, M., Shao, R., Mercat, J., Fang, A., Li, J., Keh, S., Xin, R., Nezhurina, M., Vasiljevic, I., Jitsev, J., Dimakis, A. G., Ilharco, G., Song, S., Kollar, T., Carmon, Y., Dave, A., Heckel, R., Muennighoff, N., and Schmidt, L. Language mo… [find](https://scholar.google.com/scholar?q=Gadre%2C+S%2E+Y%2E%2C+Smyrnis%2C+G%2E%2C+Shankar%2C+V%2E%2C+Gururangan%2C+S%2E%2C+Wortsman%2C+M%2E%2C+Shao%2C+R%2E%2C+Mercat%2C+J%2E%2C+Fang%2C+A%2E%2C+Li%2C+J%2E%2C+Keh%2C+S%2E%2C+Xin%2C+R%2E%2C+Nezhurina%2C+M%2E%2C+Vasiljevic%2C+I%2E%2C+Jitsev%2C+J%2E%2C+Dimakis%2C+A%2E+G%2E%2C+Ilharco%2C+G%2E%2C+)
54. Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework… [find](https://scholar.google.com/scholar?q=Gao%2C+L%2E%2C+Tow%2C+J%2E%2C+Abbasi%2C+B%2E%2C+Biderman%2C+S%2E%2C+Black%2C+S%2E%2C+DiPofi%2C+A%2E%2C+Foster%2C+C%2E%2C+Golding%2C+L%2E%2C+Hsu%2C+J%2E%2C+Le+Noac%E2%80%99h%2C+A%2E%2C+Li%2C+H%2E%2C+McDonell%2C+K%2E%2C+Muennighoff%2C+N%2E%2C+Ociepa%2C+C%2E%2C+Phang%2C+J%2E%2C+Reynolds%2C+L%2E%2C+Schoelko)
55. Gunter, T., Wang, Z., Wang, C., Pang, R., Narayanan, A., Zhang, A., Zhang, B., Chen, C., Chiu, C., Qiu, D., Gopinath, D., Yap, D. A., Yin, D., Nan, F., Weers, F., Yin, G., Huang, H., Wang, J., Lu, J., Peebles, J., Ye, K., Lee, M., Du, N., Chen, Q., Keunebroek, Q., Wiseman, S., Evans, S., Lei, T., Ra… — [doi:10.48550/arxiv.2407.21075](https://doi.org/10.48550/arxiv.2407.21075) [find](https://scholar.google.com/scholar?q=Gunter%2C+T%2E%2C+Wang%2C+Z%2E%2C+Wang%2C+C%2E%2C+Pang%2C+R%2E%2C+Narayanan%2C+A%2E%2C+Zhang%2C+A%2E%2C+Zhang%2C+B%2E%2C+Chen%2C+C%2E%2C+Chiu%2C+C%2E%2C+Qiu%2C+D%2E%2C+Gopinath%2C+D%2E%2C+Yap%2C+D%2E+A%2E%2C+Yin%2C+D%2E%2C+Nan%2C+F%2E%2C+Weers%2C+F%2E%2C+Yin%2C+G%2E%2C+Huang%2C+H%2E%2C+Wang%2C+J%2E%2C+Lu%2C+J%2E%2C)
56. Harutyunyan, H., Rawat, A. S., Menon, A. K., Kim, S., and Kumar, S. Supervision complexity and its role in knowledge distillation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. [find](https://scholar.google.com/scholar?q=Harutyunyan%2C+H%2E%2C+Rawat%2C+A%2E+S%2E%2C+Menon%2C+A%2E+K%2E%2C+Kim%2C+S%2E%2C+and+Kumar%2C+S%2E+Supervision+complexity+and+its+role+in+knowledge+distillation%2E+In+The+Eleventh+International+Conference+on+Learning+Representations%2C)
57. OpenReview.net, 2023. URL [https://openreview.net/forum?id=8jU7wy7N7mA](https://openreview.net/forum?id=8jU7wy7N7mA). [↗](https://openreview.net/forum?id=8jU7wy7N7mA)
58. Havrilla, A. and Liao, W. Understanding scaling laws with statistical and approximation theory for transformer neural networks on intrinsically low-dimensional data. CoRR, abs/2411.06646, 2024. doi: 10.48550/ARXIV.2411.06646. URL https://doi.org/10.48550/arXiv.2411.06646. — [doi:10.48550/arxiv.2411.06646](https://doi.org/10.48550/arxiv.2411.06646) [↗](https://doi.org/10.48550/arXiv.2411.06646)
59. Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., and Steinhardt, J. Aligning AI with shared human values. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-aligning-ai-shared-human-values/)
60. OpenReview.net, 2021a. URL [https://openreview.net/forum?id=dNy_RKzJacY](https://openreview.net/forum?id=dNy_RKzJacY). [↗](https://openreview.net/forum?id=dNy_RKzJacY)
61. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
62. Open-Review.net, 2021b. URL [https://openreview.net/forum?id=d7KBjmI3GmQ](https://openreview.net/forum?id=d7KBjmI3GmQ). [↗](https://openreview.net/forum?id=d7KBjmI3GmQ)
63. Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., Hallacy, C., Mann, B., Radford, A., Ramesh, A., Ryder, N., Ziegler, D. M., Schulman, J., Amodei, D., and McCandlish, S. Scaling laws for autoregressive generative modeling. CoRR, abs/… [our card](https://md.research.elamur.ai/free/en/papers/henighan-2020-scaling-laws-autoregressive-generative-modeling/)
64. Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. Scaling laws for transfer. CoRR, abs/2102.01293, 2021. URL https://arxiv.org/abs/2102.01293. [↗](https://arxiv.org/abs/2102.01293) [our card](https://md.research.elamur.ai/free/en/papers/hernandez-2021-scaling-laws-transfer/)
65. Hestness, J., Narang, S., Ardalani, N., Diamos, G. F., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically. CoRR, abs/1712.00409, 2017. URL http://arxiv.org/abs/1712.00409. [↗](http://arxiv.org/abs/1712.00409) [our card](https://md.research.elamur.ai/free/en/papers/hestness-2017-deep-learning-scaling-predictable-empirically/)
66. Hinton, G. E., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. CoRR, abs/1503.02531, 2015. URL http://arxiv.org/abs/1503.02531. [↗](http://arxiv.org/abs/1503.02531) [our card](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/)
67. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L.… [our card](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
68. Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y., Huang, Y., Zhao, W., Zhang, X., Thai, Z. L., Zhang, K., Wang, C., Yao, Y., Zhao, C., Zhou, J., Cai, J., Zhai, Z., Ding, N., Jia, C., Zeng, G., Li, D., Liu, Z., and Sun, M. Minicpm: Unveiling the potential of small language mode… [our card](https://md.research.elamur.ai/free/en/papers/hu-2024-minicpm-unveiling-potential-small-language/)
69. Ildiz, M. E., Gozeten, H. A., Taga, E. O., Mondelli, M., and Oymak, S. High-dimensional analysis of knowledge distillation: Weak-to-strong generalization and scaling laws. CoRR, abs/2410.18837, 2024. doi: 10.48550/ARXIV.2410.18837. URL https://doi.org/10.48550/arXiv.2410.18837. — [doi:10.48550/arxiv.2410.18837](https://doi.org/10.48550/arxiv.2410.18837) [↗](https://doi.org/10.48550/arXiv.2410.18837)
70. Jain, A., Montanari, A., and Sasoglu, E. Scaling laws for learning with real and surrogate data. CoRR, abs/2402.04376, 2024. doi: 10.48550/ARXIV.2402.04376. URL https://doi.org/10.48550/arXiv.2402.04376. — [doi:10.48550/arxiv.2402.04376](https://doi.org/10.48550/arxiv.2402.04376) [↗](https://doi.org/10.48550/arXiv.2402.04376)
71. Jelassi, S., Mohri, C., Brandfonbrener, D., Gu, A., Vyas, N., Anand, N., Alvarez-Melis, D., Li, Y., Kakade, S. M., and Malach, E. Mixture of parrots: Experts improve memorization more than reasoning. CoRR, abs/2410.19034, 2024. doi: 10.48550/ARXIV.2410.19034. URL https://doi.org/10.48550/arXiv.2410.… — [doi:10.48550/arxiv.2410.19034](https://doi.org/10.48550/arxiv.2410.19034) [↗](https://doi.org/10.48550/arXiv.2410.…)
72. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de Las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b. CoRR, abs/2310.06825, 2023. doi: 10.4855… [find](https://scholar.google.com/scholar?q=Jiang%2C+A%2E+Q%2E%2C+Sablayrolles%2C+A%2E%2C+Mensch%2C+A%2E%2C+Bamford%2C+C%2E%2C+Chaplot%2C+D%2E+S%2E%2C+de+Las+Casas%2C+D%2E%2C+Bressand%2C+F%2E%2C+Lengyel%2C+G%2E%2C+Lample%2C+G%2E%2C+Saulnier%2C+L%2E%2C+Lavaud%2C+L%2E+R%2E%2C+Lachaux%2C+M%2E%2C+Stock%2C+P%2E%2C+Scao%2C+T%2E+L%2E%2C+Lavr)
73. Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Barzilay, R. and Kan, M. (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, Ju… [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
74. Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., Das-Sarma, N., Tran-Johnson, E., Johnston, S., Showk, S. E., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., Ganguli, D., Hernandez, D., Jacobson, J., Kernion, J., K… [find](https://scholar.google.com/scholar?q=Kadavath%2C+S%2E%2C+Conerly%2C+T%2E%2C+Askell%2C+A%2E%2C+Henighan%2C+T%2E%2C+Drain%2C+D%2E%2C+Perez%2C+E%2E%2C+Schiefer%2C+N%2E%2C+Hatfield%2DDodds%2C+Z%2E%2C+Das%2DSarma%2C+N%2E%2C+Tran%2DJohnson%2C+E%2E%2C+Johnston%2C+S%2E%2C+Showk%2C+S%2E+E%2E%2C+Jones%2C+A%2E%2C+Elhage%2C+N%2E%2C+Hume%2C+T)
75. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. CoRR, abs/2001.08361, 2020. URL https://arxiv.org/abs/2001.08361. [↗](https://arxiv.org/abs/2001.08361) [our card](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/)
76. Kim, Y. and Rush, A. M. Sequence-level knowledge distillation. In Su, J., Carreras, X., and Duh, K. (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 1317–1327. The Association for Computational Lin… [our card](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/)
77. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6980. [↗](http://arxiv.org/abs/1412.6980)
78. Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. Big transfer (bit): General visual representation learning. In Vedaldi, A., Bischof, H., Brox, T., and Frahm, J. (eds.), Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020,… [find](https://scholar.google.com/scholar?q=Kolesnikov%2C+A%2E%2C+Beyer%2C+L%2E%2C+Zhai%2C+X%2E%2C+Puigcerver%2C+J%2E%2C+Yung%2C+J%2E%2C+Gelly%2C+S%2E%2C+and+Houlsby%2C+N%2E+Big+transfer+%28bit%29%3A+General+visual+representation+learning%2E+In+Vedaldi%2C+A%2E%2C+Bischof%2C+H%2E%2C+Brox%2C+T%2E%2C+and+Frahm%2C+)
79. Springer, 2020. doi: 10.1007/978-3-030-58558-7\\_29. URL https://doi.org/10.1007/978-3-030-58558-7_29. — [doi:10.1007/978-3-030-58558-7\\_29](https://doi.org/10.1007/978-3-030-58558-7_29) [↗](https://doi.org/10.1007/978-3-030-58558-7_29)
80. Kraft, D. A Software Package for Sequential Quadratic Programming. Deutsche Forschungs- und Versuch-sanstalt für Luft- und Raumfahrt Köln: Forschungs-bericht. Wiss. Berichtswesen d. DFVLR, 1988. URL [https://books.google.co.uk/books?id=4rKaGwAACAAJ](https://books.google.co.uk/books?id=4rKaGwAACAAJ). [↗](https://books.google.co.uk/books?id=4rKaGwAACAAJ)
81. Lee, D., Tian, Z., Zhao, Y., Cheung, K. C., and Zhang, N. L. Hard gate knowledge distillation - leverage calibration for robust and reliable language model. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EM… [find](https://scholar.google.com/scholar?q=Lee%2C+D%2E%2C+Tian%2C+Z%2E%2C+Zhao%2C+Y%2E%2C+Cheung%2C+K%2E+C%2E%2C+and+Zhang%2C+N%2E+L%2E+Hard+gate+knowledge+distillation+%2D+leverage+calibration+for+robust+and+reliable+language+model%2E+In+Goldberg%2C+Y%2E%2C+Kozareva%2C+Z%2E%2C+and+Zhang%2C+Y)
82. Li, Y., Bubeck, S., Eldan, R., Giorno, A. D., Gunasekar, S., and Lee, Y. T. Textbooks are all you need II: phi-1.5 technical report. CoRR, abs/2309.05463, 2023. doi: 10.48550/ARXIV.2309.05463. URL https://doi.org/10.48550/arXiv.2309.05463. [↗](https://doi.org/10.48550/arXiv.2309.05463) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-textbooks-all-need-ii-phi/)
83. Liu, Z., Zhao, C., Iandola, F. N., Lai, C., Tian, Y., Fedorov, I., Xiong, Y., Chang, E., Shi, Y., Krishnamoorthi, R., Lai, L., and Chandra, V. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. In Forty-first International Conference on Machine Learning, ICML 2024,… [find](https://scholar.google.com/scholar?q=Liu%2C+Z%2E%2C+Zhao%2C+C%2E%2C+Iandola%2C+F%2E+N%2E%2C+Lai%2C+C%2E%2C+Tian%2C+Y%2E%2C+Fedorov%2C+I%2E%2C+Xiong%2C+Y%2E%2C+Chang%2C+E%2E%2C+Shi%2C+Y%2E%2C+Krishnamoorthi%2C+R%2E%2C+Lai%2C+L%2E%2C+and+Chandra%2C+V%2E+Mobilellm%3A+Optimizing+sub%2Dbillion+parameter+language+mode)
84. OpenReview.net, 2024. URL [https://openreview.net/forum?id=EIGbXbxcUQ](https://openreview.net/forum?id=EIGbXbxcUQ). [↗](https://openreview.net/forum?id=EIGbXbxcUQ)
85. Lopez-Paz, D., Bottou, L., Schölkopf, B., and Vapnik, V. Unifying distillation and privileged information. In Bengio, Y. and LeCun, Y. (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http://arx… — [arXiv:1511.03643](https://arxiv.org/abs/1511.03643) [find](https://scholar.google.com/scholar?q=Lopez%2DPaz%2C+D%2E%2C+Bottou%2C+L%2E%2C+Sch%C3%B6lkopf%2C+B%2E%2C+and+Vapnik%2C+V%2E+Unifying+distillation+and+privileged+information%2E+In+Bengio%2C+Y%2E+and+LeCun%2C+Y%2E+%28eds%2E%29%2C+4th+International+Conference+on+Learning+Representations%2C)
86. Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. [find](https://scholar.google.com/scholar?q=Loshchilov%2C+I%2E+and+Hutter%2C+F%2E+Decoupled+weight+decay+regularization%2E+In+7th+International+Conference+on+Learning+Representations%2C+ICLR+2019%2C+New+Orleans%2C+LA%2C+USA%2C+May+6%2D9%2C+2019%2E)
87. OpenReview.net, 2019. URL [https://openreview.net/forum?id=Bkg6RiCqY7](https://openreview.net/forum?id=Bkg6RiCqY7). [↗](https://openreview.net/forum?id=Bkg6RiCqY7)
88. Ludziejewski, J., Krajewski, J., Adamczewski, K., Pióro, M., Krutul, M., Antoniak, S., Ciebiera, K., Król, K., Odrzygózdz, T., Sankowski, P., Cygan, M., and Jaszczur, S. Scaling laws for fine-grained mixture of experts. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna,… [find](https://scholar.google.com/scholar?q=Ludziejewski%2C+J%2E%2C+Krajewski%2C+J%2E%2C+Adamczewski%2C+K%2E%2C+Pi%C3%B3ro%2C+M%2E%2C+Krutul%2C+M%2E%2C+Antoniak%2C+S%2E%2C+Ciebiera%2C+K%2E%2C+Kr%C3%B3l%2C+K%2E%2C+Odrzyg%C3%B3zdz%2C+T%2E%2C+Sankowski%2C+P%2E%2C+Cygan%2C+M%2E%2C+and+Jaszczur%2C+S%2E+Scaling+laws+for+fine%2Dgrained+)
89. OpenReview.net, 2024. URL [https://openreview.net/forum?id=yoqdlynCRs](https://openreview.net/forum?id=yoqdlynCRs). [↗](https://openreview.net/forum?id=yoqdlynCRs)
90. Lukasik, M., Bhojanapalli, S., Menon, A. K., and Kumar, S. Teacher’s pet: understanding and mitigating biases in distillation. Trans. Mach. Learn. Res., 2022, 2022. URL [https://openreview.net/forum?id=ph3AYXpwEb](https://openreview.net/forum?id=ph3AYXpwEb). [↗](https://openreview.net/forum?id=ph3AYXpwEb)
91. Menon, A. K., Rawat, A. S., Reddi, S. J., Kim, S., and Kumar, S. Why distillation helps: a statistical perspective. CoRR, abs/2005.10419, 2020. URL https://arxiv.org/abs/2005.10419. — [arXiv:2005.10419](https://arxiv.org/abs/2005.10419) [↗](https://arxiv.org/abs/2005.10419)
92. Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., Tafti, P., Hussenot, L., Chowdhery, A., Roberts, A., Barua, A., Botev, A., Castro-Ros, A., Slone, A., Héliou, A., Tacchetti, A., Bulanova, A., Paterson, A., Tsai, B., Shahriari, B., Lan,… [find](https://scholar.google.com/scholar?q=Mesnard%2C+T%2E%2C+Hardin%2C+C%2E%2C+Dadashi%2C+R%2E%2C+Bhupatiraju%2C+S%2E%2C+Pathak%2C+S%2E%2C+Sifre%2C+L%2E%2C+Rivi%C3%A8re%2C+M%2E%2C+Kale%2C+M%2E+S%2E%2C+Love%2C+J%2E%2C+Tafti%2C+P%2E%2C+Hussenot%2C+L%2E%2C+Chowdhery%2C+A%2E%2C+Roberts%2C+A%2E%2C+Barua%2C+A%2E%2C+Botev%2C+A%2E%2C+Castro%2DRos%2C)
93. Minderer, M., Djolonga, J., Romijnders, R., Hubis, F., Zhai, X., Houlsby, N., Tran, D., and Lucic, M. Revisiting the calibration of modern neural networks. In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Ann… [find](https://scholar.google.com/scholar?q=Minderer%2C+M%2E%2C+Djolonga%2C+J%2E%2C+Romijnders%2C+R%2E%2C+Hubis%2C+F%2E%2C+Zhai%2C+X%2E%2C+Houlsby%2C+N%2E%2C+Tran%2C+D%2E%2C+and+Lucic%2C+M%2E+Revisiting+the+calibration+of+modern+neural+networks%2E+In+Ranzato%2C+M%2E%2C+Beygelzimer%2C+A%2E%2C+Dauphin%2C+Y%2E)
94. Mirzadeh, S., Farajtabar, M., Li, A., Levine, N., Matsukawa, A., and Ghasemzadeh, H. Improved knowledge distillation via teacher assistant. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAA… [our card](https://md.research.elamur.ai/free/en/papers/mirzadeh-2019-improved-knowledge-distillation-teacher-assistant/)
95. AAAI Press, 2020. doi: 10.1609/AAAI.V34I04.5963. URL https://doi.org/10.1609/aaai.v34i04.5963. — [doi:10.1609/aaai.v34i04.5963](https://doi.org/10.1609/aaai.v34i04.5963) [↗](https://doi.org/10.1609/aaai.v34i04.5963)
96. Mobahi, H., Farajtabar, M., and Bartlett, P. L. Self-distillation amplifies regularization in hilbert space. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2… [find](https://scholar.google.com/scholar?q=Mobahi%2C+H%2E%2C+Farajtabar%2C+M%2E%2C+and+Bartlett%2C+P%2E+L%2E+Self%2Ddistillation+amplifies+regularization+in+hilbert+space%2E+In+Larochelle%2C+H%2E%2C+Ranzato%2C+M%2E%2C+Hadsell%2C+R%2E%2C+Balcan%2C+M%2E%2C+and+Lin%2C+H%2E+%28eds%2E%29%2C+Advances+in+Ne)
97. Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., and Raffel, C. Scaling data-constrained language models. CoRR, abs/2305.16264, 2023a. doi: 10.48550/ARXIV.2305.16264. URL https://doi.org/10.48550/arXiv.2305.16264. [↗](https://doi.org/10.48550/arXiv.2305.16264)
98. Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, C. A. Scaling data-constrained language models. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: An… [find](https://scholar.google.com/scholar?q=Muennighoff%2C+N%2E%2C+Rush%2C+A%2E+M%2E%2C+Barak%2C+B%2E%2C+Scao%2C+T%2E+L%2E%2C+Tazi%2C+N%2E%2C+Piktus%2C+A%2E%2C+Pyysalo%2C+S%2E%2C+Wolf%2C+T%2E%2C+and+Raffel%2C+C%2E+A%2E+Scaling+data%2Dconstrained+language+models%2E+In+Oh%2C+A%2E%2C+Naumann%2C+T%2E%2C+Globerson%2C+A%2E%2C+Sa)
99. Mukhoti, J., Kulharia, V., Sanyal, A., Golodetz, S., Torr, P. H. S., and Dokania, P. K. Calibrating deep neural networks using focal loss. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural In… [find](https://scholar.google.com/scholar?q=Mukhoti%2C+J%2E%2C+Kulharia%2C+V%2E%2C+Sanyal%2C+A%2E%2C+Golodetz%2C+S%2E%2C+Torr%2C+P%2E+H%2E+S%2E%2C+and+Dokania%2C+P%2E+K%2E+Calibrating+deep+neural+networks+using+focal+loss%2E+In+Larochelle%2C+H%2E%2C+Ranzato%2C+M%2E%2C+Hadsell%2C+R%2E%2C+Balcan%2C+M%2E%2C+and+)
100. Muralidharan, S., Sreenivas, S. T., Joshi, R., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. Compact language models via pruning and knowledge distillation. CoRR, abs/2407.14679, 2024. doi: 10.48550/ARXIV.2407.14679. URL https://doi.org/10.48550/arXiv.2407.146… [↗](https://doi.org/10.48550/arXiv.2407.146…)
101. Nagarajan, V., Menon, A. K., Bhojanapalli, S., Mobahi, H., and Kumar, S. On student-teacher deviations in distillation: does it pay to disobey? In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference… [find](https://scholar.google.com/scholar?q=Nagarajan%2C+V%2E%2C+Menon%2C+A%2E+K%2E%2C+Bhojanapalli%2C+S%2E%2C+Mobahi%2C+H%2E%2C+and+Kumar%2C+S%2E+On+student%2Dteacher+deviations+in+distillation%3A+does+it+pay+to+disobey%3F+In+Oh%2C+A%2E%2C+Naumann%2C+T%2E%2C+Globerson%2C+A%2E%2C+Saenko%2C+K%2E%2C+Hardt)
102. Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., and Zaharia, M. Efficient large-scale language model training on GPU clusters using megatron-lm. In de Supin-ski, B. R., Hall, M. W., and… [find](https://scholar.google.com/scholar?q=Narayanan%2C+D%2E%2C+Shoeybi%2C+M%2E%2C+Casper%2C+J%2E%2C+LeGresley%2C+P%2E%2C+Patwary%2C+M%2E%2C+Korthikanti%2C+V%2E%2C+Vainbrand%2C+D%2E%2C+Kashinkunti%2C+P%2E%2C+Bernauer%2C+J%2E%2C+Catanzaro%2C+B%2E%2C+Phanishayee%2C+A%2E%2C+and+Zaharia%2C+M%2E+Efficient+large%2Dscale)
103. ACM, 2021. doi: 10.1145/3458817.3476209. URL https://doi.org/10.1145/3458817.3476209. — [doi:10.1145/3458817.3476209](https://doi.org/10.1145/3458817.3476209) [↗](https://doi.org/10.1145/3458817.3476209)
104. Nguyen, T. Q. and Salazar, J. Transformers without tears: Improving the normalization of self-attention. In Niehues, J., Cattoni, R., Stüker, S., Negri, M., Turchi, M., Ha, T., Salesky, E., Sanabria, R., Barrault, L., Specia, L., and Federico, M. (eds.), Proceedings of the 16th International Confere… [find](https://scholar.google.com/scholar?q=Nguyen%2C+T%2E+Q%2E+and+Salazar%2C+J%2E+Transformers+without+tears%3A+Improving+the+normalization+of+self%2Dattention%2E+In+Niehues%2C+J%2E%2C+Cattoni%2C+R%2E%2C+St%C3%BCker%2C+S%2E%2C+Negri%2C+M%2E%2C+Turchi%2C+M%2E%2C+Ha%2C+T%2E%2C+Salesky%2C+E%2E%2C+Sanabria%2C+)
105. Nilsback, M. and Zisserman, A. Automated flower classification over a large number of classes. In Sixth Indian Conference on Computer Vision, Graphics & Image Processing, ICVGIP 2008, Bhubaneswar, India, 16-19 December 2008, pp. 722–729. [find](https://scholar.google.com/scholar?q=Nilsback%2C+M%2E+and+Zisserman%2C+A%2E+Automated+flower+classification+over+a+large+number+of+classes%2E+In+Sixth+Indian+Conference+on+Computer+Vision%2C+Graphics+%26+Image+Processing%2C+ICVGIP+2008%2C+Bhubaneswar%2C+Ind)
106. IEEE Computer Society, 2008. doi: 10.1109/ICVGIP.2008.47. URL https://doi.org/10.1109/ICVGIP.2008.47. — [doi:10.1109/icvgip.2008.47](https://doi.org/10.1109/icvgip.2008.47) [↗](https://doi.org/10.1109/ICVGIP.2008.47)
107. OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/ARXIV.2303.08774. URL https://doi.org/10.48550/arXiv.2303.08774. [↗](https://doi.org/10.48550/arXiv.2303.08774) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
108. OpenAI and Pilipiszyn, A. Gpt-3 powers the next generation of apps, 2021. URL [http://website-url.com](http://website-url.com). Accessed on Jan 19, 2025. [↗](http://website-url.com)
109. Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 201… [our card](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/)
110. Paquette, E., Paquette, C., Xiao, L., and Pennington, J. 4+3 phases of compute-optimal neural scaling laws. CoRR, abs/2405.15074, 2024. doi: 10.48550/ARXIV.2405.15074. URL https://doi.org/10.48550/arXiv.2405.15074. — [doi:10.48550/arxiv.2405.15074](https://doi.org/10.48550/arxiv.2405.15074) [↗](https://doi.org/10.48550/arXiv.2405.15074)
111. Pareek, D., Du, S. S., and Oh, S. Understanding the gains from repeated self-distillation. CoRR, abs/2407.04600, 2024. doi: 10.48550/ARXIV.2407.04600. URL https://doi.org/10.48550/arXiv.2407.04600. — [doi:10.48550/arxiv.2407.04600](https://doi.org/10.48550/arxiv.2407.04600) [↗](https://doi.org/10.48550/arXiv.2407.04600)
112. Pearce, T. and Song, J. Reconciling kaplan and chinchilla scaling laws. CoRR, abs/2406.12907, 2024. doi: 10.48550/ARXIV.2406.12907. URL https://doi.org/10.48550/arXiv.2406.12907. — [doi:10.48550/arxiv.2406.12907](https://doi.org/10.48550/arxiv.2406.12907) [↗](https://doi.org/10.48550/arXiv.2406.12907)
113. Peng, H., Lv, X., Bai, Y., Yao, Z., Zhang, J., Hou, L., and Li, J. Pre-training distillation for large language models: A design space exploration. CoRR, abs/2410.16215, 2024. doi: 10.48550/ARXIV.2410.16215. URL https://doi.org/10.48550/arXiv.2410.16215. [↗](https://doi.org/10.48550/arXiv.2410.16215)
114. Porian, T., Wortsman, M., Jitsev, J., Schmidt, L., and Carmon, Y. Resolving discrepancies in compute-optimal scaling of language models. CoRR, abs/2406.19146, 2024. doi: 10.48550/ARXIV.2406.19146. URL https://doi.org/10.48550/arXiv.2406.19146. [↗](https://doi.org/10.48550/arXiv.2406.19146) [our card](https://md.research.elamur.ai/free/en/papers/porian-2024-resolving-discrepancies-compute-optimal-scaling/)
115. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020. URL [https://jmlr.org/papers/v21/20-074.html](https://jmlr.org/papers/v21/20-074.html). [↗](https://jmlr.org/papers/v21/20-074.html) [our card](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/)
116. Rawat, A. S., Sadhanala, V., Rostamizadeh, A., Chakrabarti, A., Jitkrittum, W., Feinberg, V., Kim, S., Harutyunyan, H., Saunshi, N., Nado, Z., Shivanna, R., Reddi, S. J., Menon, A. K., Anil, R., and Kumar, S. A little help goes a long way: Efficient LLM training by leveraging small lms. CoRR, abs/24… — [doi:10.48550/arxiv.2410.18779](https://doi.org/10.48550/arxiv.2410.18779) [find](https://scholar.google.com/scholar?q=Rawat%2C+A%2E+S%2E%2C+Sadhanala%2C+V%2E%2C+Rostamizadeh%2C+A%2E%2C+Chakrabarti%2C+A%2E%2C+Jitkrittum%2C+W%2E%2C+Feinberg%2C+V%2E%2C+Kim%2C+S%2E%2C+Harutyunyan%2C+H%2E%2C+Saunshi%2C+N%2E%2C+Nado%2C+Z%2E%2C+Shivanna%2C+R%2E%2C+Reddi%2C+S%2E+J%2E%2C+Menon%2C+A%2E+K%2E%2C+Anil%2C+R%2E%2C+and+K)
117. Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillicrap, T. P., Alayrac, J., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., Antonoglou, I., Anil, R., Borgeaud, S., Dai, A. M., Millican, K., Dyer, E., Glaese, M., Sottiaux, T., Lee, B., Viola, F., Reynolds, M., Xu, Y., Molloy, J., Ch… [our card](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/)
118. Rivière, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C. L., Jerome, S., Tsitsulin, A., Vieillard, N., Stanczyk, P., Girgin, S., Momchev, N., Hoffman,… [find](https://scholar.google.com/scholar?q=Rivi%C3%A8re%2C+M%2E%2C+Pathak%2C+S%2E%2C+Sessa%2C+P%2E+G%2E%2C+Hardin%2C+C%2E%2C+Bhupatiraju%2C+S%2E%2C+Hussenot%2C+L%2E%2C+Mesnard%2C+T%2E%2C+Shahriari%2C+B%2E%2C+Ram%C3%A9%2C+A%2E%2C+Ferret%2C+J%2E%2C+Liu%2C+P%2E%2C+Tafti%2C+P%2E%2C+Friesen%2C+A%2E%2C+Casbon%2C+M%2E%2C+Ramos%2C+S%2E%2C+Kumar%2C+R%2E%2C+L)
119. Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., and Bengio, Y. Fitnets: Hints for thin deep nets. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://… [find](https://scholar.google.com/scholar?q=Romero%2C+A%2E%2C+Ballas%2C+N%2E%2C+Kahou%2C+S%2E+E%2E%2C+Chassang%2C+A%2E%2C+Gatta%2C+C%2E%2C+and+Bengio%2C+Y%2E+Fitnets%3A+Hints+for+thin+deep+nets%2E+In+Bengio%2C+Y%2E+and+LeCun%2C+Y%2E+%28eds%2E%29%2C+3rd+International+Conference+on+Learning+Representa)
120. Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y., and Shavit, N. A constructive prediction of the generalization error across scales. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. [find](https://scholar.google.com/scholar?q=Rosenfeld%2C+J%2E+S%2E%2C+Rosenfeld%2C+A%2E%2C+Belinkov%2C+Y%2E%2C+and+Shavit%2C+N%2E+A+constructive+prediction+of+the+generalization+error+across+scales%2E+In+8th+International+Conference+on+Learning+Representations%2C+ICLR+202)
121. OpenReview.net, 2020. URL [https://openreview.net/forum?id=ryenvpEKDr](https://openreview.net/forum?id=ryenvpEKDr). [↗](https://openreview.net/forum?id=ryenvpEKDr)
122. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99–106, 2021. doi: 10.1145/3474381. URL https://doi.org/10.1145/3474381. [↗](https://doi.org/10.1145/3474381)
123. Sardana, N., Portes, J. P., Doubov, S., and Frankle, J. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. [our card](https://md.research.elamur.ai/free/en/papers/sardana-2024-chinchilla-optimal-accounting-inference-language/)
124. OpenReview.net, 2024. URL [https://openreview.net/forum?id=0bmXrtTDUu](https://openreview.net/forum?id=0bmXrtTDUu). [↗](https://openreview.net/forum?id=0bmXrtTDUu)
125. Shazeer, N. GLU variants improve transformer. CoRR, abs/2002.05202, 2020. URL https://arxiv.org/abs/2002.05202. [↗](https://arxiv.org/abs/2002.05202)
126. Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. V., Hinton, G. E., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. CoRR, abs/1701.06538, 2017. URL http://arxiv.org/abs/1701.06538. [↗](http://arxiv.org/abs/1701.06538)
127. Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. CoRR, abs/2408.03314, 2024. doi: 10.48550/ARXIV.2408.03314. URL https://doi.org/10.48550/arXiv.2408.03314. [↗](https://doi.org/10.48550/arXiv.2408.03314) [our card](https://md.research.elamur.ai/free/en/papers/snell-2024-scaling-llm-test-time-compute/)
128. Sreenivas, S. T., Muralidharan, S., Joshi, R., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. LLM pruning and distillation in practice: The minitron approach. CoRR, abs/2408.11796, 2024. doi: 10.48550/ARXIV.2408.11796. URL https://doi.org/10.48550/arXiv.2408.11… [↗](https://doi.org/10.48550/arXiv.2408.11…)
129. Stanton, S., Izmailov, P., Kirichenko, P., Alemi, A. A., and Wilson, A. G. Does knowledge distillation really work? In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Pro… [find](https://scholar.google.com/scholar?q=Stanton%2C+S%2E%2C+Izmailov%2C+P%2E%2C+Kirichenko%2C+P%2E%2C+Alemi%2C+A%2E+A%2E%2C+and+Wilson%2C+A%2E+G%2E+Does+knowledge+distillation+really+work%3F+In+Ranzato%2C+M%2E%2C+Beygelzimer%2C+A%2E%2C+Dauphin%2C+Y%2E+N%2E%2C+Liang%2C+P%2E%2C+and+Vaughan%2C+J%2E+W%2E+%28eds%2E)
130. Stein, C. Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, 1:197–206, 1956. [find](https://scholar.google.com/scholar?q=Stein%2C+C%2E+Inadmissibility+of+the+usual+estimator+for+the+mean+of+a+multivariate+normal+distribution%2E+Proceedings+of+the+Third+Berkeley+Symposium+on+Mathematical+Statistics+and+Probability%2C+1%3A197%E2%80%93206%2C+)
131. Su, J., Ahmed, M. H. M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. doi: 10.1016/J.NEUCOM.2023.127063. URL https://doi.org/10.1016/j.neucom.2023.127063. [↗](https://doi.org/10.1016/j.neucom.2023.127063) [our card](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/)
132. Tian, Y., Krishnan, D., and Isola, P. Contrastive representation distillation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. [find](https://scholar.google.com/scholar?q=Tian%2C+Y%2E%2C+Krishnan%2C+D%2E%2C+and+Isola%2C+P%2E+Contrastive+representation+distillation%2E+In+8th+International+Conference+on+Learning+Representations%2C+ICLR+2020%2C+Addis+Ababa%2C+Ethiopia%2C+April+26%2D30%2C+2020%2E)
133. OpenReview.net, 2020. URL [https://openreview.net/forum?id=SkgpBJrtvS](https://openreview.net/forum?id=SkgpBJrtvS). [↗](https://openreview.net/forum?id=SkgpBJrtvS)
134. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.48550/ARXIV.2302.13971. URL htt… [our card](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/)
135. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Canton-Ferrer, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Ho… [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%2C+Martin%2C+L%2E%2C+Stone%2C+K%2E%2C+Albert%2C+P%2E%2C+Almahairi%2C+A%2E%2C+Babaei%2C+Y%2E%2C+Bashlykov%2C+N%2E%2C+Batra%2C+S%2E%2C+Bhargava%2C+P%2E%2C+Bhosale%2C+S%2E%2C+Bikel%2C+D%2E%2C+Blecher%2C+L%2E%2C+Canton%2DFerrer%2C+C%2E%2C+Chen%2C+M%2E%2C+Cucurull%2C+G%2E%2C+Esiob)
136. Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S., Brett, M., Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E., Carey, C., Polat, I., Feng, Y., Moore,… [our card](https://md.research.elamur.ai/free/en/papers/virtanen-2019-scipy-1-0-fundamental-algorithms/)
137. Welbl, J., Liu, N. F., and Gardner, M. Crowdsourcing multiple choice science questions. In Derczynski, L., Xu, W., Ritter, A., and Baldwin, T. (eds.), Proceedings of the 3rd Workshop on Noisy User-generated Text, NUT@EMNLP 2017, Copenhagen, Denmark, September 7, 2017, pp. 94–106. Association for Com… [our card](https://md.research.elamur.ai/free/en/papers/welbl-2017-crowdsourcing-multiple-choice-science-questions/)
138. Wortsman, M., Liu, P. J., Xiao, L., Everett, K., Alemi, A., Adlam, B., Co-Reyes, J. D., Gur, I., Kumar, A., Novak, R., Pennington, J., Sohl-Dickstein, J., Xu, K., Lee, J., Gilmer, J., and Kornblith, S. Small-scale proxies for large-scale transformer training instabilities. CoRR, abs/2309.14322, 2023… [find](https://scholar.google.com/scholar?q=Wortsman%2C+M%2E%2C+Liu%2C+P%2E+J%2E%2C+Xiao%2C+L%2E%2C+Everett%2C+K%2E%2C+Alemi%2C+A%2E%2C+Adlam%2C+B%2E%2C+Co%2DReyes%2C+J%2E+D%2E%2C+Gur%2C+I%2E%2C+Kumar%2C+A%2E%2C+Novak%2C+R%2E%2C+Pennington%2C+J%2E%2C+Sohl%2DDickstein%2C+J%2E%2C+Xu%2C+K%2E%2C+Lee%2C+J%2E%2C+Gilmer%2C+J%2E%2C+and+Kornblith%2C+S)
139. Wortsman, M., Liu, P. J., Xiao, L., Everett, K. E., Alemi, A. A., Adlam, B., Co-Reyes, J. D., Gur, I., Kumar, A., Novak, R., Pennington, J., Sohl-Dickstein, J., Xu, K., Lee, J., Gilmer, J., and Kornblith, S. Small-scale proxies for large-scale transformer training instabilities. In The Twelfth Inter… [find](https://scholar.google.com/scholar?q=Wortsman%2C+M%2E%2C+Liu%2C+P%2E+J%2E%2C+Xiao%2C+L%2E%2C+Everett%2C+K%2E+E%2E%2C+Alemi%2C+A%2E+A%2E%2C+Adlam%2C+B%2E%2C+Co%2DReyes%2C+J%2E+D%2E%2C+Gur%2C+I%2E%2C+Kumar%2C+A%2E%2C+Novak%2C+R%2E%2C+Pennington%2C+J%2E%2C+Sohl%2DDickstein%2C+J%2E%2C+Xu%2C+K%2E%2C+Lee%2C+J%2E%2C+Gilmer%2C+J%2E%2C+and+Kornbl)
140. OpenReview.net, 2024. URL [https://openreview.net/forum?id=d8w0pmvXbZ](https://openreview.net/forum?id=d8w0pmvXbZ). [↗](https://openreview.net/forum?id=d8w0pmvXbZ)
141. Wu, C., Acun, B., Raghavendra, R., and Hazelwood, K. M. Beyond efficiency: Scaling AI sustainably. IEEE Micro, 44(5):37–46, 2024a. doi: 10.1109/MM.2024.3409275. URL https://doi.org/10.1109/MM.2024.3409275. — [doi:10.1109/mm.2024.3409275](https://doi.org/10.1109/mm.2024.3409275) [↗](https://doi.org/10.1109/MM.2024.3409275)
142. Wu, Y., Sun, Z., Li, S., Welleck, S., and Yang, Y. An empirical analysis of compute-optimal inference for problem-solving with language models. CoRR, abs/2408.00724, 2024b. doi: 10.48550/ARXIV.2408.00724. URL https://doi.org/10.48550/arXiv.2408.00724. [↗](https://doi.org/10.48550/arXiv.2408.00724) [our card](https://md.research.elamur.ai/free/en/papers/wu-2024-inference-scaling-laws-empirical-analysis/)
143. Yang, G. and Hu, E. J. Tensor programs IV: feature learning in infinite-width neural networks. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, p… [find](https://scholar.google.com/scholar?q=Yang%2C+G%2E+and+Hu%2C+E%2E+J%2E+Tensor+programs+IV%3A+feature+learning+in+infinite%2Dwidth+neural+networks%2E+In+Meila%2C+M%2E+and+Zhang%2C+T%2E+%28eds%2E%29%2C+Proceedings+of+the+38th+International+Conference+on+Machine+Learning%2C+)
144. PMLR, 2021. URL [http://proceedings.mlr.press/v139/yang21c.html](http://proceedings.mlr.press/v139/yang21c.html). [↗](http://proceedings.mlr.press/v139/yang21c.html)
145. Yang, G. and Littwin, E. Tensor programs ivb: Adaptive optimization in the infinite-width limit. CoRR, abs/2308.01814, 2023. doi: 10.48550/ARXIV.2308.01814. URL https://doi.org/10.48550/arXiv.2308.01814. — [doi:10.48550/arxiv.2308.01814](https://doi.org/10.48550/arxiv.2308.01814) [↗](https://doi.org/10.48550/arXiv.2308.01814)
146. Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tensor programs V: tuning large neural networks via zero-shot hyperparameter transfer. CoRR, abs/2203.03466, 2022. doi: 10.48550/ARXIV.2203.03466. URL https://doi.org/10.48550/arXiv.220… [↗](https://doi.org/10.48550/arXiv.220…)
147. Yang, G., Simon, J. B., and Bernstein, J. A spectral condition for feature learning. CoRR, abs/2310.17813, 2023. doi: 10.48550/ARXIV.2310.17813. URL https://doi.org/10.48550/arXiv.2310.17813. [↗](https://doi.org/10.48550/arXiv.2310.17813)
148. Yang, G., Yu, D., Zhu, C., and Hayou, S. Tensor programs VI: feature learning in infinite depth neural networks. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. [find](https://scholar.google.com/scholar?q=Yang%2C+G%2E%2C+Yu%2C+D%2E%2C+Zhu%2C+C%2E%2C+and+Hayou%2C+S%2E+Tensor+programs+VI%3A+feature+learning+in+infinite+depth+neural+networks%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+ICLR+2024%2C+Vienna%2C)
149. OpenReview.net, 2024. URL [https://openreview.net/forum?id=17pVDnpwwl](https://openreview.net/forum?id=17pVDnpwwl). [↗](https://openreview.net/forum?id=17pVDnpwwl)
150. Yuan, M., Lang, B., and Quan, F. Student-friendly knowledge distillation. Knowl. Based Syst., 296: 111915, 2024. doi: 10.1016/J.KNOSYS.2024.111915. URL https://doi.org/10.1016/j.knosys.2024.111915. — [doi:10.1016/j.knosys.2024.111915](https://doi.org/10.1016/j.knosys.2024.111915) [↗](https://doi.org/10.1016/j.knosys.2024.111915)
151. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In Korhonen, A., Traum, D. R., and Màrquez, L. (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August… [find](https://scholar.google.com/scholar?q=Zellers%2C+R%2E%2C+Holtzman%2C+A%2E%2C+Bisk%2C+Y%2E%2C+Farhadi%2C+A%2E%2C+and+Choi%2C+Y%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Korhonen%2C+A%2E%2C+Traum%2C+D%2E+R%2E%2C+and+M%C3%A0rquez%2C+L%2E+%28eds%2E%29%2C+Proceedings+of+the+57th+Conf)
152. Zhang, B. and Sennrich, R. Root mean square layer normalization. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 201… [find](https://scholar.google.com/scholar?q=Zhang%2C+B%2E+and+Sennrich%2C+R%2E+Root+mean+square+layer+normalization%2E+In+Wallach%2C+H%2E+M%2E%2C+Larochelle%2C+H%2E%2C+Beygelzimer%2C+A%2E%2C+d%E2%80%99Alch%C3%A9%2DBuc%2C+F%2E%2C+Fox%2C+E%2E+B%2E%2C+and+Garnett%2C+R%2E+%28eds%2E%29%2C+Advances+in+Neural+Information)
153. Zhang, C., Raghu, M., Kleinberg, J. M., and Bengio, S. Pointer value retrieval: A new benchmark for understanding the limits of neural network generalization. CoRR, abs/2107.12580, 2021. URL https://arxiv.org/abs/2107.12580. [↗](https://arxiv.org/abs/2107.12580)
154. Zhang, C., Song, D., Ye, Z., and Gao, Y. Towards the law of capacity gap in distilling language models. CoRR, abs/2311.07052, 2023a. doi: 10.48550/ARXIV.2311.07052. URL https://doi.org/10.48550/arXiv.2311.07052. [↗](https://doi.org/10.48550/arXiv.2311.07052)
155. Zhang, C., Yang, Y., Liu, J., Wang, J., Xian, Y., Wang, B., and Song, D. Lifting the curse of capacity gap in distilling language models. In Rogers, A., Boyd-Graber, J. L., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… [find](https://scholar.google.com/scholar?q=Zhang%2C+C%2E%2C+Yang%2C+Y%2E%2C+Liu%2C+J%2E%2C+Wang%2C+J%2E%2C+Xian%2C+Y%2E%2C+Wang%2C+B%2E%2C+and+Song%2C+D%2E+Lifting+the+curse+of+capacity+gap+in+distilling+language+models%2E+In+Rogers%2C+A%2E%2C+Boyd%2DGraber%2C+J%2E+L%2E%2C+and+Okazaki%2C+N%2E+%28eds%2E%29%2C+Pro)
156. Zhu, C., Xu, B., Wang, Q., Zhang, Y., and Mao, Z. On the calibration of large language models and alignment. CoRR, abs/2311.13240, 2023. doi: 10.48550/ARXIV.2311.13240. URL https://doi.org/10.48550/arXiv.2311.13240. [↗](https://doi.org/10.48550/arXiv.2311.13240)