---
title: "How to Set the Learning Rate for Large-Scale Pre-training? (2026)"
description: "The authors compare two strategies under the Warmup-Stable-Decay (WSD) schedule on Mixture of… 37 references, cited by 1 paper here."
authors: ["Yunhua Zhou", "Shuhao Xing", "Junhao Huang", "Xipeng Qiu", "Qipeng Guo"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1861/"
doi: "10.18653/v1/2026.findings-acl.1861"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhou-2026-how-set-learning-rate-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhou-2026-how-set-learning-rate-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/13/) / How to Set the Learning Rate for Large-Scale Pre-training?

# How to Set the Learning Rate for Large-Scale Pre-training?

**Authors:** [Yunhua Zhou](https://md.research.elamur.ai/free/en/authors/yunhua-zhou/), Shuhao Xing, Junhao Huang, [Xipeng Qiu](https://md.research.elamur.ai/free/en/authors/xipeng-qiu/) and [Qipeng Guo](https://md.research.elamur.ai/free/en/authors/qipeng-guo/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1861/) · [PDF](https://aclanthology.org/2026.findings-acl.1861.pdf) · [DOI 10.18653/v1/2026.findings-acl.1861](https://doi.org/10.18653/v1/2026.findings-acl.1861) · [OpenAlex](https://openalex.org/W7166877317)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Choosing the learning rate (LR) for large-scale pre-training is expensive to get wrong and too costly to tune by trial at full scale. The question is whether small, cheap runs can reliably predict the best LR for a much larger target. The authors compare two strategies under the Warmup-Stable-Decay (WSD) schedule on Mixture of Experts (MoE) models.

- **Fitting Paradigm:** they fit loss against log-LR with a quadratic curve. They also model loss as a power law in training tokens, which lets them extrapolate rather than search every configuration. From these fits they derive a formula that expresses the optimal LR in terms of N (how large the model is) and D (how much data it is trained on).
- **Transfer Paradigm:** they extend µTransfer to MoE, covering depth, weight decay and token horizon.

Both approaches are tested at a target far beyond earlier hyperparameter studies. The globally fitted LR beats µTransfer, which calls the scalability of µTransfer into question. Analyses of training stability and feature learning help explain why module-wise tuning falls short at scale. For practitioners, this is concrete guidance on setting the LR.

## Key points
- Search cost in the Fitting Paradigm drops from O(n³) to O(n·C_D·C_η) by combining quadratic LR fits with a loss-vs-data power law.
- The fitted rule is Lr(N, D) = 38.4588 · N^−0.2219 · D^−0.3509, with R² ≈ 0.9622. Optimal LR depends more strongly on data volume than on model size.
- The target setting is an MoE with 12B total and 1.3B active parameters, trained on 500B tokens. This is more than ten times larger than prior LR prediction work.
- For the 4B model, after a decay phase (LR annealed to 10%, plus 100B high-quality tokens), the fitted LR outperformed µTransfer:
  - MMLU: 42.23% → 43.51%
  - CMMLU: 40.58% → 42.81%
- The study has three stated limitations:
  - Only the WSD schedule is studied.
  - Only MoE models are tested; dense models are not.
  - The maximum scale at which these extrapolations stay accurate was not tested.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 202 works · 22,509 citations
- [Shanghai Jiao Tong University](https://md.research.elamur.ai/free/en/universities/shanghai-jiao-tong-university/) — University, Shanghai, China · 212 works · 19,544 citations
- [Shanghai Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/shanghai-artificial-intelligence-laboratory/) — Research institute, Shanghai, China · 152 works · 13,166 citations

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Yunhua Zhou, Shuhao Xing, Junhao Huang, Xipeng Qiu and Qipeng Guo. 2026. How to Set the Learning Rate for Large-Scale Pre-training?. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhou2026how,
  title = {{How to Set the Learning Rate for Large-Scale Pre-training?}},
  author = {Yunhua Zhou, Shuhao Xing, Junhao Huang, Xipeng Qiu and Qipeng Guo},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1861},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1861}
}
```

- [← previous How Robust Are Large Language Models for Clinical Numeracy? An Empirical Study on Numerical Reasoning Abilities in Clinical Contexts](https://md.research.elamur.ai/free/en/papers/nguyen-2026-how-robust-are-large-language/)
- [next → Howard University-AI4PC at SemEval-2026 Task 8: Query Reformulation and Dense-Lexical Retrieval Fusion for Multi-Turn Retrieval-Augmented Generation](https://md.research.elamur.ai/free/en/papers/shrestha-2026-howard-university-ai4pc-semeval-2026/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/13/)

## References

37 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Lei Bai, Zhongrui Cai, Yuhang Cao, Maosong Cao, Weihan Cao, Chiyu Chen, Haojiong Chen, Kai Chen, Pengcheng Chen, Ying Chen, Yongkang Chen, Yu Cheng, Pei Chu, Tao Chu, Erfei Cui, Ganqu Cui, Long Cui, Ziyun Cui, Nianchen Deng, and 158 others. 2025. [Intern-s1: A scientific multimodal foundation model](https://md.research.elamur.ai/free/en/papers/bai-2025-intern-s1-scientific-multimodal/).…
2. Johan Bjorck, Alon Benhaim, Vishrav Chaudhary, Furu Wei, and Xia Song. 2025. Scaling optimal lr across token horizons. Preprint, arXiv:2409.19913. — [arXiv:2409.19913](https://arxiv.org/abs/2409.19913) [find](https://scholar.google.com/scholar?q=Johan+Bjorck%2C+Alon+Benhaim%2C+Vishrav+Chaudhary%2C+Furu+Wei%2C+and+Xia+Song%2E+2025%2E+Scaling+optimal+lr+across+token+horizons%2E+Preprint%2C+arXiv%3A2409%2E19913%2E+%E2%80%94+%5BarXiv%3A2409%2E19913%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2409%2E19913)
3. Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, and 81 others. 2024. [Internlm2 technical report](https://md.research.elamur.ai/free/en/papers/cai-2024-internlm2-technical-report/). Preprint, arXiv:2403.17297.
4. LMDeploy Contributors. 2023a. Lmdeploy: A toolkit for compressing, deploying, and serving llm. [https://github.com/InternLM/lmdeploy](https://github.com/InternLM/lmdeploy). [↗](https://github.com/InternLM/lmdeploy)
5. OpenCompass Contributors. 2023b. Opencompass: A universal evaluation platform for foundation models. [https://github.com/open-compass/](https://github.com/open-compass/) opencompass. [↗](https://github.com/open-compass/)
6. DeepSeek-AI, :, Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, and 69 others. 2024a. Deepseek llm: Scaling open-source language models with longter… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+%3A%2C+Xiao+Bi%2C+Deli+Chen%2C+Guanting+Chen%2C+Shanhuang+Chen%2C+Damai+Dai%2C+Chengqi+Deng%2C+Honghui+Ding%2C+Kai+Dong%2C+Qiushi+Du%2C+Zhe+Fu%2C+Huazuo+Gao%2C+Kaige+Gao%2C+Wenjun+Gao%2C+Ruiqi+Ge%2C+Kang+Guan%2C+Daya+Guo%2C)
7. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025a. [Deepseek-r1: Incentivizing reasoning capability in l…](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/)
8. DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, and 138 others. 2024b. Deepseek-v2: A strong, economical, and efficient… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Aixin+Liu%2C+Bei+Feng%2C+Bin+Wang%2C+Bingxuan+Wang%2C+Bo+Liu%2C+Chenggang+Zhao%2C+Chengqi+Dengr%2C+Chong+Ruan%2C+Damai+Dai%2C+Daya+Guo%2C+Dejian+Yang%2C+Deli+Chen%2C+Dongjie+Ji%2C+Erhang+Li%2C+Fangyun+Lin%2C+Fuli+Luo%2C)
9. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025b. Deepseek-v3 technical report. Preprint, arXiv:… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+Damai+Dai%2C+Daya+Guo%2C+Dejian+Yang%2C+Deli+Chen%2C+Dongjie+Ji%2C+Erhan)
10. DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenhao Xu, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, and 245 others. 2025c. [Deepseek-v3.2: Pushing the frontier of ope…](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-v3-2-pushing/)
11. Nolan Dey, Bin Claire Zhang, Lorenzo Noci, Mufan Li, Blake Bordelon, Shane Bergsma, Cengiz Pehlevan, Boris Hanin, and Joel Hestness. 2025. Don’t be lazy: Completep enables compute-efficient deep transformers. Preprint, arXiv:2505.01618. — [arXiv:2505.01618](https://arxiv.org/abs/2505.01618) [find](https://scholar.google.com/scholar?q=Nolan+Dey%2C+Bin+Claire+Zhang%2C+Lorenzo+Noci%2C+Mufan+Li%2C+Blake+Bordelon%2C+Shane+Bergsma%2C+Cengiz+Pehlevan%2C+Boris+Hanin%2C+and+Joel+Hestness%2E+2025%2E+Don%E2%80%99t+be+lazy%3A+Completep+enables+compute%2Defficient+deep+trans)
12. Zhiyuan Fan, Yifeng Liu, Qingyue Zhao, Angela Yuan, and Quanquan Gu. 2025. Robust layerwise scaling rules by proper weight decay tuning. Preprint, arXiv:2510.15262. — [arXiv:2510.15262](https://arxiv.org/abs/2510.15262) [find](https://scholar.google.com/scholar?q=Zhiyuan+Fan%2C+Yifeng+Liu%2C+Qingyue+Zhao%2C+Angela+Yuan%2C+and+Quanquan+Gu%2E+2025%2E+Robust+layerwise+scaling+rules+by+proper+weight+decay+tuning%2E+Preprint%2C+arXiv%3A2510%2E15262%2E+%E2%80%94+%5BarXiv%3A2510%2E15262%5D%28https%3A%2F%2Farxiv%2E)
13. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). Preprint, arXiv:2009.03300.
14. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — [Training Compute-Optimal Large Language Models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
15. Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, and 6 others. 2024. [Minicpm: Unveiling the potential of small lan…](https://md.research.elamur.ai/free/en/papers/hu-2024-minicpm-unveiling-potential-small-language/)
16. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). Preprint, arXiv:2001.08361.
17. Atli Kosson, Bettina Messmer, and Martin Jaggi. 2024. Rotational equilibrium: how weight decay balances learning across neural networks. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. [find](https://scholar.google.com/scholar?q=Atli+Kosson%2C+Bettina+Messmer%2C+and+Martin+Jaggi%2E+2024%2E+Rotational+equilibrium%3A+how+weight+decay+balances+learning+across+neural+networks%2E+In+Proceedings+of+the+41st+International+Conference+on+Machine+)
18. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2024. [CMMLU: Measuring massive multitask language understanding in Chinese](https://md.research.elamur.ai/free/en/papers/li-2023-cmmlu-measuring-massive-multitask-language/). In Findings of the Association for Computational Linguistics: ACL 2024, pages 11260–11285, Bangkok, Thailand. Association…
19. Houyi Li, Wenzhen Zheng, Qiufeng Wang, Hanshan Zhang, Zili Wang, Shijie Xuyang, Yuantao Fan, Zhenyu Ding, Haoying Wang, Ning Ding, Shuigeng Zhou, Xiangyu Zhang, and Daxin Jiang. 2025. Predictable scale: Part i, step law – optimal hyperparameter scaling law in large language model pretraining. Prepri… [find](https://scholar.google.com/scholar?q=Houyi+Li%2C+Wenzhen+Zheng%2C+Qiufeng+Wang%2C+Hanshan+Zhang%2C+Zili+Wang%2C+Shijie+Xuyang%2C+Yuantao+Fan%2C+Zhenyu+Ding%2C+Haoying+Wang%2C+Ning+Ding%2C+Shuigeng+Zhou%2C+Xiangyu+Zhang%2C+and+Daxin+Jiang%2E+2025%2E+Predictable+scal)
20. Lucas Lingle. 2025. An empirical study of µp learning rate transfer. Preprint, arXiv:2404.05728. — [arXiv:2404.05728](https://arxiv.org/abs/2404.05728) [find](https://scholar.google.com/scholar?q=Lucas+Lingle%2E+2025%2E+An+empirical+study+of+%C2%B5p+learning+rate+transfer%2E+Preprint%2C+arXiv%3A2404%2E05728%2E+%E2%80%94+%5BarXiv%3A2404%2E05728%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2404%2E05728%29)
21. Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, and 9 others. 2025. Muon is scalable for llm training. Preprint, arXiv:… [find](https://scholar.google.com/scholar?q=Jingyuan+Liu%2C+Jianlin+Su%2C+Xingcheng+Yao%2C+Zhejun+Jiang%2C+Guokun+Lai%2C+Yulun+Du%2C+Yidao+Qin%2C+Weixin+Xu%2C+Enzhe+Lu%2C+Junjie+Yan%2C+Yanru+Chen%2C+Huabin+Zheng%2C+Yibo+Liu%2C+Shaowei+Liu%2C+Bohong+Yin%2C+Weiran+He%2C+Han+Zhu)
22. Kamil Adamczewski, and Jakub Krajewski. 2025. µ-parametrization for mixture of experts. Preprint, arXiv:2508.09752. — [arXiv:2508.09752](https://arxiv.org/abs/2508.09752) [find](https://scholar.google.com/scholar?q=Kamil+Adamczewski%2C+and+Jakub+Krajewski%2E+2025%2E+%C2%B5%2Dparametrization+for+mixture+of+experts%2E+Preprint%2C+arXiv%3A2508%2E09752%2E+%E2%80%94+%5BarXiv%3A2508%2E09752%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2508%2E09752%29)
23. Bruno Mlodozeniec, Pierre Ablin, Louis Béthune, Dan Busbridge, Michal Klein, Jason Ramapuram, and Marco Cuturi. 2025. Completed hyperparameter transfer across modules, width, depth, batch and duration. Preprint, arXiv:2512.22382. — [arXiv:2512.22382](https://arxiv.org/abs/2512.22382) [find](https://scholar.google.com/scholar?q=Bruno+Mlodozeniec%2C+Pierre+Ablin%2C+Louis+Be%CC%81thune%2C+Dan+Busbridge%2C+Michal+Klein%2C+Jason+Ramapuram%2C+and+Marco+Cuturi%2E+2025%2E+Completed+hyperparameter+transfer+across+modules%2C+width%2C+depth%2C+batch+and+duratio)
24. OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, and 108 others. 2025. [gpt-oss-120b & gpt-oss-20b model…](https://md.research.elamur.ai/free/en/papers/agarwal-2025-gpt-oss-120b-gpt-oss/)
25. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024a. [Gpt-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). P…
26. OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, and 244 others. 2024b. [Openai o1 s…](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/)
27. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others.… — [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
28. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. [find](https://scholar.google.com/scholar?q=Qwen%2C+%3A%2C+An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+)
29. Ilya Loshchilov and Frank Hutter. 2017. Sgdr: Stochastic gradient descent with warm restarts. Preprint, arXiv:1608.03983. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2017%2E+Sgdr%3A+Stochastic+gradient+descent+with+warm+restarts%2E+Preprint%2C+arXiv%3A1608%2E03983%2E)
30. Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, and 150 others. 2025. [Kimi k2: Open agentic intelligence](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-k2-open-agentic/). Preprint, a…
31. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. Preprint, arXiv:1711.05101. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+Preprint%2C+arXiv%3A1711%2E05101%2E)
32. Howe Tissue, Venus Wang, and Lu Wang. 2024. Scaling law with learning rate annealing. Preprint, arXiv:2408.11029. — [arXiv:2408.11029](https://arxiv.org/abs/2408.11029) [find](https://scholar.google.com/scholar?q=Howe+Tissue%2C+Venus+Wang%2C+and+Lu+Wang%2E+2024%2E+Scaling+law+with+learning+rate+annealing%2E+Preprint%2C+arXiv%3A2408%2E11029%2E+%E2%80%94+%5BarXiv%3A2408%2E11029%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2408%2E11029%29)
33. Jan Małaśnicki, Kamil Ciebiera, Mateusz Boruń, Maciej Pióro, Jan Ludziejewski, Maciej Stefaniak, Michał Krutul, Sebastian Jaszczur, Marek Cygan, Xi Wang and Laurence Aitchison. 2025. [How to set adamw’s weight decay as you scale model and dataset size](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). Preprint, arXiv:2405.13698. *Figure 12: The p…
34. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 43 others. 2024. Qwen2 technical report. Preprint, arXiv:240… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chang+Zhou%2C+Chengpeng+Li%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Guanting+Dong%2C+Haoran+Wei%2C+Huan+Lin%2C+Jialong+Tang%2C+Jialin+Wang%2C+Jian+Yang%2C+Jian)
35. Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. 2022. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. Preprint, arXiv:2203.03466. [find](https://scholar.google.com/scholar?q=Greg+Yang%2C+Edward+J%2E+Hu%2C+Igor+Babuschkin%2C+Szymon+Sidor%2C+Xiaodong+Liu%2C+David+Farhi%2C+Nick+Ryder%2C+Jakub+Pachocki%2C+Weizhu+Chen%2C+and+Jianfeng+Gao%2E+2022%2E+Tensor+programs+v%3A+Tuning+large+neural+networks+via+)
36. Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. 2023. Tensor programs vi: Feature learning in infinite-depth neural networks. Preprint, arXiv:2310.02244. [find](https://scholar.google.com/scholar?q=Greg+Yang%2C+Dingli+Yu%2C+Chen+Zhu%2C+and+Soufiane+Hayou%2E+2023%2E+Tensor+programs+vi%3A+Feature+learning+in+infinite%2Ddepth+neural+networks%2E+Preprint%2C+arXiv%3A2310%2E02244%2E)
37. Li Zhang, Youhe Jiang, Guoliang He, Xin Chen, Han Lv, Qian Yao, Fangcheng Fu, and Kai Chen. 2025. Efficient mixed-precision large language model inference with turbomind. arXiv preprint arXiv:2508.15601. [find](https://scholar.google.com/scholar?q=Li+Zhang%2C+Youhe+Jiang%2C+Guoliang+He%2C+Xin+Chen%2C+Han+Lv%2C+Qian+Yao%2C+Fangcheng+Fu%2C+and+Kai+Chen%2E+2025%2E+Efficient+mixed%2Dprecision+large+language+model+inference+with+turbomind%2E+arXiv+preprint+arXiv%3A2508%2E156)

*Updated 31 October 2026*