---
title: "HTMuon: Improving Muon via Heavy-Tailed Spectral Correction"
description: "This matters because it offers a simple, theory-backed upgrade to a widely adopted optimizer. 72 references, cited by 2 papers here."
authors: ["Tianyu Pang", "Yujie Fang", "Zihang Liu", "Shenyang Deng", "Lei Hsiung", "Shuhua Yu", "Yaoqing Yang"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1819/"
doi: "10.18653/v1/2026.findings-acl.1819"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pang-2026-htmuon-improving-muon-heavy-tailed/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pang-2026-htmuon-improving-muon-heavy-tailed/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/7/) / HTMuon: Improving Muon via Heavy-Tailed Spectral Correction

# HTMuon: Improving Muon via Heavy-Tailed Spectral Correction

**Authors:** [Tianyu Pang](https://md.research.elamur.ai/free/en/authors/tianyu-pang/), Yujie Fang, [Zihang Liu](https://md.research.elamur.ai/free/en/authors/zihang-liu/), Shenyang Deng, [Lei Hsiung](https://md.research.elamur.ai/free/en/authors/lei-hsiung/), Shuhua Yu and [Yaoqing Yang](https://md.research.elamur.ai/free/en/authors/yaoqing-yang/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1819/) · [PDF](https://aclanthology.org/2026.findings-acl.1819.pdf) · [DOI 10.18653/v1/2026.findings-acl.1819](https://doi.org/10.18653/v1/2026.findings-acl.1819) · [OpenAlex](https://openalex.org/W7166821461)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/pang-2026-htmuon-improving-muon-heavy-tailed/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Muon is a matrix-based optimizer now used in large-scale LLM training. It orthogonalizes the momentum matrix, which forces every singular value to one. The authors argue this has two costs. It gives as much weight to noisy, small-singular-value directions as to informative ones. It also keeps the weight spectra light-tailed, while Heavy-Tailed Self-Regularization (HT-SR) theory links heavier-tailed empirical spectral densities with better-trained models.

Their fix, HTMuon, raises the momentum's singular values to a power p between 0 and 1. Setting p = 0 recovers Muon and p = 1 gives SGDM. This keeps Muon's ability to model parameter coupling while producing heavier-tailed updates and weights.

They evaluate HTMuon on:
- LLaMA pretraining on C4
- GPT-2 on OpenWebText
- ResNet and ViT image classification

Across these tasks it beats strong optimizers and also improves existing Muon variants when added on top of them. On the theory side, HTMuon is shown to be steepest descent under a Schatten-q norm constraint, with a convergence guarantee matching Muon's. This matters because it offers a simple, theory-backed upgrade to a widely adopted optimizer.

## Key points
- **Default setting:** p = 0.125, chosen by grid search. Compared with Muon, HTMuon lowers perplexity by 0.92 on LLaMA-60M and by 0.98 on LLaMA-135M trained on C4.
- **Baselines beaten:** Adam, Muon, Cautious, GaLore, Sophia, Mars, SOAP and COSMOS. It also works as a plug-in for NorMuon and AdaMuon.
- **Motivating observation:** the approximate Muon_NS (Newton–Schulz) outperforms the exact Muon_SVD, which suggests that uniform unit singular values are suboptimal.
- **Theory:** HTMuon generalizes Muon's Schatten-∞ steepest-descent view to Schatten-q. In smooth non-convex settings its sample complexity is O(ε⁻⁴), the same as Muon and SGDM.
- **Scaling and limitations:** two accelerated variants let HTMuon outperform Muon on LLaMA-1B. Per-step runtime remains slightly higher, and nothing beyond 1B parameters was tested. Code is at github.com/TDCSZ327/HTmuon.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 325 works · 142,931 citations
- [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 175 works · 227,440 citations
- [International Computer Science Institute](https://md.research.elamur.ai/free/en/labs/international-computer-science-institute/) — Research institute, Berkeley, United States · 15 works · 11,471 citations
- [Dartmouth College](https://md.research.elamur.ai/free/en/universities/dartmouth-college/) — University, Hanover, United States · 37 works · 42,625 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

## Cite

Cite as: Tianyu Pang, Yujie Fang, Zihang Liu, Shenyang Deng, Lei Hsiung, Shuhua Yu and Yaoqing Yang. 2026. HTMuon: Improving Muon via Heavy-Tailed Spectral Correction. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{pang2026htmuon,
  title = {{HTMuon: Improving Muon via Heavy-Tailed Spectral Correction}},
  author = {Tianyu Pang, Yujie Fang, Zihang Liu, Shenyang Deng, Lei Hsiung, Shuhua Yu and Yaoqing Yang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1819},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1819}
}
```

- [← previous HistoriQA-ThirdRepublic: Multi-Hop Question Answering Corpus for Historical Research, Parliamentary Debates from the French Third Republic (1870-1940)](https://md.research.elamur.ai/free/en/papers/pellet-2026-historiqa-thirdrepublic-multi-hop-question/)
- [next → Improving Romanian LLM Pretraining Data using Diversity and Quality Filtering](https://md.research.elamur.ai/free/en/papers/negoita-2026-improving-romanian-llm-pretraining-data/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/7/)

## References

72 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Woodworth. 2023. Lower bounds for non-convex stochastic optimization. Mathematical Programming, 199(1):165–214. [find](https://scholar.google.com/scholar?q=Yossi+Arjevani%2C+Yair+Carmon%2C+John+C+Duchi%2C+Dylan+J+Foster%2C+Nathan+Srebro%2C+and+Blake+Woodworth%2E+2023%2E+Lower+bounds+for+non%2Dconvex+stochastic+optimization%2E+Mathematical+Programming%2C+199%281%29%3A165%E2%80%93214%2E)
2. Jeremy Bernstein and Laker Newhouse. 2024. Modular duality in deep learning. arXiv preprint arXiv:2410.21265. [find](https://scholar.google.com/scholar?q=Jeremy+Bernstein+and+Laker+Newhouse%2E+2024%2E+Modular+duality+in+deep+learning%2E+arXiv+preprint+arXiv%3A2410%2E21265%2E)
3. Da Chang, Yongxiang Liu, and Ganzhao Yuan. 2025. On the convergence of muon and beyond. arXiv preprint arXiv:2509.15816. — [arXiv:2509.15816](https://arxiv.org/abs/2509.15816) [find](https://scholar.google.com/scholar?q=Da+Chang%2C+Yongxiang+Liu%2C+and+Ganzhao+Yuan%2E+2025%2E+On+the+convergence+of+muon+and+beyond%2E+arXiv+preprint+arXiv%3A2509%2E15816%2E+%E2%80%94+%5BarXiv%3A2509%2E15816%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2509%2E15816%29)
4. Lei Chen, Joan Bruna, and Alberto Bietti. 2024. Distributional associations vs in-context reasoning: A study of feed-forward and attention layers. arXiv preprint arXiv:2406.03068. — [arXiv:2406.03068](https://arxiv.org/abs/2406.03068) [find](https://scholar.google.com/scholar?q=Lei+Chen%2C+Joan+Bruna%2C+and+Alberto+Bietti%2E+2024%2E+Distributional+associations+vs+in%2Dcontext+reasoning%3A+A+study+of+feed%2Dforward+and+attention+layers%2E+arXiv+preprint+arXiv%3A2406%2E03068%2E+%E2%80%94+%5BarXiv%3A2406%2E03068%5D)
5. Lizhang Chen, Jonathan Li, and Qiang Liu. 2025. Muon optimizes under spectral norm constraints. arXiv preprint arXiv:2506.15054. — [arXiv:2506.15054](https://arxiv.org/abs/2506.15054) [find](https://scholar.google.com/scholar?q=Lizhang+Chen%2C+Jonathan+Li%2C+and+Qiang+Liu%2E+2025%2E+Muon+optimizes+under+spectral+norm+constraints%2E+arXiv+preprint+arXiv%3A2506%2E15054%2E+%E2%80%94+%5BarXiv%3A2506%2E15054%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2506%2E15054%29)
6. Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc V Le. 2023. Symbolic discovery of optimization algorithms. In Thirty-seventh Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Xiangning+Chen%2C+Chen+Liang%2C+Da+Huang%2C+Esteban+Real%2C+Kaiyuan+Wang%2C+Hieu+Pham%2C+Xuanyi+Dong%2C+Thang+Luong%2C+Cho%2DJui+Hsieh%2C+Yifeng+Lu%2C+and+Quoc+V+Le%2E+2023%2E+Symbolic+discovery+of+optimization+algorithms%2E+In+)
7. Yatin Dandi, Luca Pesce, Hugo Cui, Florent Krzakala, Yue M Lu, and Bruno Loureiro. 2024. A random matrix theory perspective on the spectrum of learned features and asymptotic generalization capabilities. arXiv preprint arXiv:2410.18938. — [arXiv:2410.18938](https://arxiv.org/abs/2410.18938) [find](https://scholar.google.com/scholar?q=Yatin+Dandi%2C+Luca+Pesce%2C+Hugo+Cui%2C+Florent+Krzakala%2C+Yue+M+Lu%2C+and+Bruno+Loureiro%2E+2024%2E+A+random+matrix+theory+perspective+on+the+spectrum+of+learned+features+and+asymptotic+generalization+capabiliti)
8. Damek Davis and Dmitriy Drusvyatskiy. 2025. When do spectral gradient updates help in deep learning? arXiv preprint arXiv:2512.04299. — [arXiv:2512.04299](https://arxiv.org/abs/2512.04299) [find](https://scholar.google.com/scholar?q=Damek+Davis+and+Dmitriy+Drusvyatskiy%2E+2025%2E+When+do+spectral+gradient+updates+help+in+deep+learning%3F+arXiv+preprint+arXiv%3A2512%2E04299%2E+%E2%80%94+%5BarXiv%3A2512%2E04299%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2512%2E04299%29)
9. Leonardo Defilippis, Yizhou Xu, Julius Girardin, Emanuele Troiani, Vittorio Erba, Lenka Zdeborová, Bruno Loureiro, and Florent Krzakala. 2025. Scaling laws and spectra of shallow neural networks in the feature learning regime. arXiv preprint arXiv:2509.24882. — [arXiv:2509.24882](https://arxiv.org/abs/2509.24882) [find](https://scholar.google.com/scholar?q=Leonardo+Defilippis%2C+Yizhou+Xu%2C+Julius+Girardin%2C+Emanuele+Troiani%2C+Vittorio+Erba%2C+Lenka+Zdeborov%C3%A1%2C+Bruno+Loureiro%2C+and+Florent+Krzakala%2E+2025%2E+Scaling+laws+and+spectra+of+shallow+neural+networks+in+th)
10. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee. [find](https://scholar.google.com/scholar?q=Jia+Deng%2C+Wei+Dong%2C+Richard+Socher%2C+Li%2DJia+Li%2C+Kai+Li%2C+and+Li+Fei%2DFei%2E+2009%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+2009+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pa)
11. Shenyang Deng, Boyao Liao, Zhuoli Ouyang, Tianyu Pang, Minhak Song, and Yaoqing Yang. 2026. Suspicious alignment of sgd: A fine-grained step size condition analysis. arXiv preprint arXiv:2601.11789. — [arXiv:2601.11789](https://arxiv.org/abs/2601.11789) [find](https://scholar.google.com/scholar?q=Shenyang+Deng%2C+Boyao+Liao%2C+Zhuoli+Ouyang%2C+Tianyu+Pang%2C+Minhak+Song%2C+and+Yaoqing+Yang%2E+2026%2E+Suspicious+alignment+of+sgd%3A+A+fine%2Dgrained+step+size+condition+analysis%2E+arXiv+preprint+arXiv%3A2601%2E11789%2E+%E2%80%94)
12. Alexey Dosovitskiy. 2020. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). arXiv preprint arXiv:2010.11929.
13. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7). [find](https://scholar.google.com/scholar?q=John+Duchi%2C+Elad+Hazan%2C+and+Yoram+Singer%2E+2011%2E+Adaptive+subgradient+methods+for+online+learning+and+stochastic+optimization%2E+Journal+of+machine+learning+research%2C+12%287%29%2E)
14. Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. 2020. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412. — [arXiv:2010.01412](https://arxiv.org/abs/2010.01412) [find](https://scholar.google.com/scholar?q=Pierre+Foret%2C+Ariel+Kleiner%2C+Hossein+Mobahi%2C+and+Behnam+Neyshabur%2E+2020%2E+Sharpness%2Daware+minimization+for+efficiently+improving+generalization%2E+arXiv+preprint+arXiv%3A2010%2E01412%2E+%E2%80%94+%5BarXiv%3A2010%2E01412%5D%28ht)
15. Guillaume Garrigos and Robert M Gower. 2023. Handbook of convergence theorems for (stochastic) gradient methods. arXiv preprint arXiv:2301.11235. — [arXiv:2301.11235](https://arxiv.org/abs/2301.11235) [find](https://scholar.google.com/scholar?q=Guillaume+Garrigos+and+Robert+M+Gower%2E+2023%2E+Handbook+of+convergence+theorems+for+%28stochastic%29+gradient+methods%2E+arXiv+preprint+arXiv%3A2301%2E11235%2E+%E2%80%94+%5BarXiv%3A2301%2E11235%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2301%2E11235%29)
16. Aaron Gokaslan and Vanya Cohen. 2019. Openwebtext corpus. [http://Skylion007.github.io/](http://Skylion007.github.io/) OpenWebTextCorpus. [↗](http://Skylion007.github.io/)
17. Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, and 24 others. 2024. Olmo: A… [find](https://scholar.google.com/scholar?q=Dirk+Groeneveld%2C+Iz+Beltagy%2C+Evan+Walsh%2C+Akshita+Bhagia%2C+Rodney+Kinney%2C+Oyvind+Tafjord%2C+Ananya+Jha%2C+Hamish+Ivison%2C+Ian+Magnusson%2C+Yizhong+Wang%2C+Shane+Arora%2C+David+Atkinson%2C+Russell+Authur%2C+Khyathi+Cha)
18. Vineet Gupta, Tomer Koren, and Yoram Singer. 2018. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pages 1842–1850. PMLR. [find](https://scholar.google.com/scholar?q=Vineet+Gupta%2C+Tomer+Koren%2C+and+Yoram+Singer%2E+2018%2E+Shampoo%3A+Preconditioned+stochastic+tensor+optimization%2E+In+International+Conference+on+Machine+Learning%2C+pages+1842%E2%80%931850%2E+PMLR%2E)
19. Di He, Songjun Tu, Ajay Jaiswal, Li Shen, Ganzhao Yuan, Shiwei Liu, and Lu Yin. 2025. Alphadecay: Module-wise weight decay for heavy-tailed balancing in llms. arXiv preprint arXiv:2506.14562. — [arXiv:2506.14562](https://arxiv.org/abs/2506.14562) [find](https://scholar.google.com/scholar?q=Di+He%2C+Songjun+Tu%2C+Ajay+Jaiswal%2C+Li+Shen%2C+Ganzhao+Yuan%2C+Shiwei+Liu%2C+and+Lu+Yin%2E+2025%2E+Alphadecay%3A+Module%2Dwise+weight+decay+for+heavy%2Dtailed+balancing+in+llms%2E+arXiv+preprint+arXiv%3A2506%2E14562%2E+%E2%80%94+%5BarXiv)
20. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778.
21. Liming Liu, Zhenghao Xu, Zixuan Zhang, Hao Kang, Zichong Li, Chen Liang, Weizhu Chen, and Tuo Zhao. 2025b. Cosmos: A hybrid adaptive optimizer for memory-efficient training of llms. arXiv preprint arXiv:2502.17410. — [arXiv:2502.17410](https://arxiv.org/abs/2502.17410) [find](https://scholar.google.com/scholar?q=Liming+Liu%2C+Zhenghao+Xu%2C+Zixuan+Zhang%2C+Hao+Kang%2C+Zichong+Li%2C+Chen+Liang%2C+Weizhu+Chen%2C+and+Tuo+Zhao%2E+2025b%2E+Cosmos%3A+A+hybrid+adaptive+optimizer+for+memory%2Defficient+training+of+llms%2E+arXiv+preprint+arX)
22. Nicholas J Higham and Robert S Schreiber. 1990. Fast polar decomposition of an arbitrary matrix. SIAM Journal on Scientific and Statistical Computing, 11(4):648–655. [find](https://scholar.google.com/scholar?q=Nicholas+J+Higham+and+Robert+S+Schreiber%2E+1990%2E+Fast+polar+decomposition+of+an+arbitrary+matrix%2E+SIAM+Journal+on+Scientific+and+Statistical+Computing%2C+11%284%29%3A648%E2%80%93655%2E)
23. Xuyuan Liu, Lei Hsiung, Yaoqing Yang, and Yujun Yan. 2025c. Spectral insights into data-oblivious critical layers in large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 4860–4877, Vienna, Austria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xuyuan+Liu%2C+Lei+Hsiung%2C+Yaoqing+Yang%2C+and+Yujun+Yan%2E+2025c%2E+Spectral+insights+into+data%2Doblivious+critical+layers+in+large+language+models%2E+In+Findings+of+the+Association+for+Computational+Linguistics)
24. Liam Hodgkinson, Umut Simsekli, Rajiv Khanna, and Michael Mahoney. 2022. Generalization bounds using lower tail exponents in stochastic optimizers. In International Conference on Machine Learning, pages 8774–8795. PMLR. [find](https://scholar.google.com/scholar?q=Liam+Hodgkinson%2C+Umut+Simsekli%2C+Rajiv+Khanna%2C+and+Michael+Mahoney%2E+2022%2E+Generalization+bounds+using+lower+tail+exponents+in+stochastic+optimizers%2E+In+International+Conference+on+Machine+Learning%2C+pag)
25. Liam Hodgkinson, Zhichao Wang, and Michael W Mahoney. 2025. Models of heavy-tailed mechanistic universality. arXiv preprint arXiv:2506.03470. — [arXiv:2506.03470](https://arxiv.org/abs/2506.03470) [find](https://scholar.google.com/scholar?q=Liam+Hodgkinson%2C+Zhichao+Wang%2C+and+Michael+W+Mahoney%2E+2025%2E+Models+of+heavy%2Dtailed+mechanistic+universality%2E+arXiv+preprint+arXiv%3A2506%2E03470%2E+%E2%80%94+%5BarXiv%3A2506%2E03470%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2506%2E03470%29)
26. Yuanzhe Hu, Kinshuk Goel, Vlad Killiakov, and Yaoqing Yang. 2025. Eigenspectrum analysis of neural networks without aspect ratio bias. In Forty-second International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Yuanzhe+Hu%2C+Kinshuk+Goel%2C+Vlad+Killiakov%2C+and+Yaoqing+Yang%2E+2025%2E+Eigenspectrum+analysis+of+neural+networks+without+aspect+ratio+bias%2E+In+Forty%2Dsecond+International+Conference+on+Machine+Learning%2E)
27. Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. 2024. Muon: An optimizer for hidden layers in neural networks. [find](https://scholar.google.com/scholar?q=Keller+Jordan%2C+Yuchen+Jin%2C+Vlado+Boza%2C+Jiacheng+You%2C+Franz+Cesista%2C+Laker+Newhouse%2C+and+Jeremy+Bernstein%2E+2024%2E+Muon%3A+An+optimizer+for+hidden+layers+in+neural+networks%2E)
28. Jean Kaddour, Linqing Liu, Ricardo Silva, and Matt J Kusner. 2022. When do flat minima optimizers work? Advances in Neural Information Processing Systems, 35:16577–16595. [find](https://scholar.google.com/scholar?q=Jean+Kaddour%2C+Linqing+Liu%2C+Ricardo+Silva%2C+and+Matt+J+Kusner%2E+2022%2E+When+do+flat+minima+optimizers+work%3F+Advances+in+Neural+Information+Processing+Systems%2C+35%3A16577%E2%80%9316595%2E)
29. Diederik P Kingma. 2014. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980.
30. Vignesh Kothapalli, Tianyu Pang, Shenyang Deng, Zongmin Liu, and Yaoqing Yang. 2025. From spikes to heavy tails: Unveiling the spectral evolution of neural networks. Transactions on Machine Learning Research. [find](https://scholar.google.com/scholar?q=Vignesh+Kothapalli%2C+Tianyu+Pang%2C+Shenyang+Deng%2C+Zongmin+Liu%2C+and+Yaoqing+Yang%2E+2025%2E+From+spikes+to+heavy+tails%3A+Unveiling+the+spectral+evolution+of+neural+networks%2E+Transactions+on+Machine+Learning+R)
31. Alex Krizhevsky and Geoffrey Hinton. 2009. Learning multiple layers of features from tiny images. [find](https://scholar.google.com/scholar?q=Alex+Krizhevsky+and+Geoffrey+Hinton%2E+2009%2E+Learning+multiple+layers+of+features+from+tiny+images%2E)
32. Zichong Li, Liming Liu, Chen Liang, Weizhu Chen, and Tuo Zhao. 2025. Normuon: Making muon more efficient and scalable. arXiv preprint arXiv:2510.05491. — [arXiv:2510.05491](https://arxiv.org/abs/2510.05491) [find](https://scholar.google.com/scholar?q=Zichong+Li%2C+Liming+Liu%2C+Chen+Liang%2C+Weizhu+Chen%2C+and+Tuo+Zhao%2E+2025%2E+Normuon%3A+Making+muon+more+efficient+and+scalable%2E+arXiv+preprint+arXiv%3A2510%2E05491%2E+%E2%80%94+%5BarXiv%3A2510%2E05491%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2510%2E)
33. Kaizhao Liang, Lizhang Chen, Bo Liu, and Qiang Liu. 2024. Cautious optimizers: Improving training with one line of code. arXiv preprint arXiv:2411.16085. — [arXiv:2411.16085](https://arxiv.org/abs/2411.16085) [find](https://scholar.google.com/scholar?q=Kaizhao+Liang%2C+Lizhang+Chen%2C+Bo+Liu%2C+and+Qiang+Liu%2E+2024%2E+Cautious+optimizers%3A+Improving+training+with+one+line+of+code%2E+arXiv+preprint+arXiv%3A2411%2E16085%2E+%E2%80%94+%5BarXiv%3A2411%2E16085%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F241)
34. Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. 2024a. Sophia: A scalable stochastic second-order optimizer for language model pre-training. Preprint, arXiv:2305.14342. [find](https://scholar.google.com/scholar?q=Hong+Liu%2C+Zhiyuan+Li%2C+David+Hall%2C+Percy+Liang%2C+and+Tengyu+Ma%2E+2024a%2E+Sophia%3A+A+scalable+stochastic+second%2Dorder+optimizer+for+language+model+pre%2Dtraining%2E+Preprint%2C+arXiv%3A2305%2E14342%2E)
35. Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, and 9 others. 2025a. Muon is scalable for llm training. arXiv preprint… [find](https://scholar.google.com/scholar?q=Jingyuan+Liu%2C+Jianlin+Su%2C+Xingcheng+Yao%2C+Zhejun+Jiang%2C+Guokun+Lai%2C+Yulun+Du%2C+Yidao+Qin%2C+Weixin+Xu%2C+Enzhe+Lu%2C+Junjie+Yan%2C+Yanru+Chen%2C+Huabin+Zheng%2C+Yibo+Liu%2C+Shaowei+Liu%2C+Bohong+Yin%2C+Weiran+He%2C+Han+Zhu)
36. Zihang Liu, Yuanzhe Hu, Tianyu Pang, Yefan Zhou, Pu Ren, and Yaoqing Yang. 2024b. Model balancing helps low-data training and fine-tuning. arXiv preprint arXiv:2410.12178. [find](https://scholar.google.com/scholar?q=Zihang+Liu%2C+Yuanzhe+Hu%2C+Tianyu+Pang%2C+Yefan+Zhou%2C+Pu+Ren%2C+and+Yaoqing+Yang%2E+2024b%2E+Model+balancing+helps+low%2Ddata+training+and+fine%2Dtuning%2E+arXiv+preprint+arXiv%3A2410%2E12178%2E)
37. Zihang Liu, Tianyu Pang, Oleg Balabanov, Chaoqun Yang, Tianjin Huang, Lu Yin, Yaoqing Yang, and Shiwei Liu. 2025d. Lift the veil for the truth: Principal weights emerge after rank reduction for reasoning-focused supervised fine-tuning. arXiv preprint arXiv:2506.00772. — [arXiv:2506.00772](https://arxiv.org/abs/2506.00772) [find](https://scholar.google.com/scholar?q=Zihang+Liu%2C+Tianyu+Pang%2C+Oleg+Balabanov%2C+Chaoqun+Yang%2C+Tianjin+Huang%2C+Lu+Yin%2C+Yaoqing+Yang%2C+and+Shiwei+Liu%2E+2025d%2E+Lift+the+veil+for+the+truth%3A+Principal+weights+emerge+after+rank+reduction+for+reason)
38. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2017%2E+Decoupled+weight+decay+regularization%2E+arXiv+preprint+arXiv%3A1711%2E05101%2E)
39. Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W Mahoney, and Yaoqing Yang. 2024. Al-phapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models. Advances in neural information processing systems, 37:9117–9152. [find](https://scholar.google.com/scholar?q=Haiquan+Lu%2C+Yefan+Zhou%2C+Shiwei+Liu%2C+Zhangyang+Wang%2C+Michael+W+Mahoney%2C+and+Yaoqing+Yang%2E+2024%2E+Al%2Dphapruning%3A+Using+heavy%2Dtailed+self+regularization+theory+for+improved+layer%2Dwise+pruning+of+large+lan)
40. Jianhao Ma, Yu Huang, Yuejie Chi, and Yuxin Chen. 2026. Preconditioning benefits of spectral orthogonalization in muon. Preprint, arXiv:2601.13474. — [arXiv:2601.13474](https://arxiv.org/abs/2601.13474) [find](https://scholar.google.com/scholar?q=Jianhao+Ma%2C+Yu+Huang%2C+Yuejie+Chi%2C+and+Yuxin+Chen%2E+2026%2E+Preconditioning+benefits+of+spectral+orthogonalization+in+muon%2E+Preprint%2C+arXiv%3A2601%2E13474%2E+%E2%80%94+%5BarXiv%3A2601%2E13474%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2601%2E1347)
41. Charles H Martin and Christopher Hinrichs. 2025. Setol: A semi-empirical theory of (deep) learning. arXiv preprint arXiv:2507.17912. — [arXiv:2507.17912](https://arxiv.org/abs/2507.17912) [find](https://scholar.google.com/scholar?q=Charles+H+Martin+and+Christopher+Hinrichs%2E+2025%2E+Setol%3A+A+semi%2Dempirical+theory+of+%28deep%29+learning%2E+arXiv+preprint+arXiv%3A2507%2E17912%2E+%E2%80%94+%5BarXiv%3A2507%2E17912%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2507%2E17912%29)
42. Charles H Martin and Michael W Mahoney. 2021. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165):1–73. [find](https://scholar.google.com/scholar?q=Charles+H+Martin+and+Michael+W+Mahoney%2E+2021%2E+Implicit+self%2Dregularization+in+deep+neural+networks%3A+Evidence+from+random+matrix+theory+and+implications+for+learning%2E+Journal+of+Machine+Learning+Resear)
43. Charles H Martin, Tongsu Peng, and Michael W Mahoney. 2021. Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data. Nature Communications, 12(1):4122. [find](https://scholar.google.com/scholar?q=Charles+H+Martin%2C+Tongsu+Peng%2C+and+Michael+W+Mahoney%2E+2021%2E+Predicting+trends+in+the+quality+of+state%2Dof%2Dthe%2Dart+neural+networks+without+access+to+training+or+testing+data%2E+Nature+Communications%2C+12%281)
44. Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. 2018. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957. — [arXiv:1802.05957](https://arxiv.org/abs/1802.05957) [find](https://scholar.google.com/scholar?q=Takeru+Miyato%2C+Toshiki+Kataoka%2C+Masanori+Koyama%2C+and+Yuichi+Yoshida%2E+2018%2E+Spectral+normalization+for+generative+adversarial+networks%2E+arXiv+preprint+arXiv%3A1802%2E05957%2E+%E2%80%94+%5BarXiv%3A1802%2E05957%5D%28https%3A%2F%2Farx)
45. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, and 1 others. 2024. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656. [find](https://scholar.google.com/scholar?q=Team+OLMo%2C+Pete+Walsh%2C+Luca+Soldaini%2C+Dirk+Groeneveld%2C+Kyle+Lo%2C+Shane+Arora%2C+Akshita+Bhagia%2C+Yuling+Gu%2C+Shengyi+Huang%2C+Matt+Jordan%2C+and+1+others%2E+2024%2E+2+olmo+2+furious%2E+arXiv+preprint+arXiv%3A2501%2E0065)
46. Thomas Pethick, Wanyun Xie, Kimon Antonakopou-los, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. 2025. Training deep learning models with norm-constrained lmos. arXiv preprint arXiv:2502.07529. — [arXiv:2502.07529](https://arxiv.org/abs/2502.07529) [find](https://scholar.google.com/scholar?q=Thomas+Pethick%2C+Wanyun+Xie%2C+Kimon+Antonakopou%2Dlos%2C+Zhenyu+Zhu%2C+Antonio+Silveti%2DFalls%2C+and+Volkan+Cevher%2E+2025%2E+Training+deep+learning+models+with+norm%2Dconstrained+lmos%2E+arXiv+preprint+arXiv%3A2502%2E07529)
47. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+and+1+others%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
48. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of machine learning research, 21(140):1–67.
49. Andrei Semenov, Matteo Pagliardini, and Martin Jaggi. 2025. Benchmarking optimizers for large language model pretraining. arXiv preprint arXiv:2509.01440. — [arXiv:2509.01440](https://arxiv.org/abs/2509.01440) [find](https://scholar.google.com/scholar?q=Andrei+Semenov%2C+Matteo+Pagliardini%2C+and+Martin+Jaggi%2E+2025%2E+Benchmarking+optimizers+for+large+language+model+pretraining%2E+arXiv+preprint+arXiv%3A2509%2E01440%2E+%E2%80%94+%5BarXiv%3A2509%2E01440%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F25)
50. Ishaan Shah, Anthony M. Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Ashish Tanwer, Darsh J Shah, Khoi Nguyen, Kurt Smith, Michael Callahan, Michael Pust, Mohit Parmar, Peter Rushton, Platon Mazarakis, Ritvik Kapila, Saurabh Srivastava, and 4 others… [find](https://scholar.google.com/scholar?q=Ishaan+Shah%2C+Anthony+M%2E+Polloreno%2C+Karl+Stratos%2C+Philip+Monk%2C+Adarsh+Chaluvaraju%2C+Andrew+Hojel%2C+Andrew+Ma%2C+Anil+Thomas%2C+Ashish+Tanwer%2C+Darsh+J+Shah%2C+Khoi+Nguyen%2C+Kurt+Smith%2C+Michael+Callahan%2C+Michael+)
51. Pratyusha Sharma, Jordan T Ash, and Dipendra Misra. 2023. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. arXiv preprint arXiv:2312.13558. — [arXiv:2312.13558](https://arxiv.org/abs/2312.13558) [find](https://scholar.google.com/scholar?q=Pratyusha+Sharma%2C+Jordan+T+Ash%2C+and+Dipendra+Misra%2E+2023%2E+The+truth+is+in+there%3A+Improving+reasoning+in+language+models+with+layer%2Dselective+rank+reduction%2E+arXiv+preprint+arXiv%3A2312%2E13558%2E+%E2%80%94+%5BarXiv%3A2)
52. Wei Shen, Ruichuan Huang, Minhui Huang, Cong Shen, and Jiawei Zhang. 2025. On the convergence analysis of muon. arXiv preprint arXiv:2505.23737. — [arXiv:2505.23737](https://arxiv.org/abs/2505.23737) [find](https://scholar.google.com/scholar?q=Wei+Shen%2C+Ruichuan+Huang%2C+Minhui+Huang%2C+Cong+Shen%2C+and+Jiawei+Zhang%2E+2025%2E+On+the+convergence+analysis+of+muon%2E+arXiv+preprint+arXiv%3A2505%2E23737%2E+%E2%80%94+%5BarXiv%3A2505%2E23737%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2505%2E23737%29)
53. Chongjie Si, Debing Zhang, and Wei Shen. 2025. Adamuon: Adaptive muon optimizer. arXiv preprint arXiv:2507.11005. — [arXiv:2507.11005](https://arxiv.org/abs/2507.11005) [find](https://scholar.google.com/scholar?q=Chongjie+Si%2C+Debing+Zhang%2C+and+Wei+Shen%2E+2025%2E+Adamuon%3A+Adaptive+muon+optimizer%2E+arXiv+preprint+arXiv%3A2507%2E11005%2E+%E2%80%94+%5BarXiv%3A2507%2E11005%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2507%2E11005%29)
54. James B Simon, Madeline Dickens, Dhruva Karkada, and Michael R DeWeese. 2023. The eigenlearning framework: A conservation law perspective on kernel ridge regression and wide neural networks. Transactions on Machine Learning Research. [find](https://scholar.google.com/scholar?q=James+B+Simon%2C+Madeline+Dickens%2C+Dhruva+Karkada%2C+and+Michael+R+DeWeese%2E+2023%2E+The+eigenlearning+framework%3A+A+conservation+law+perspective+on+kernel+ridge+regression+and+wide+neural+networks%2E+Transacti)
55. Umut Simsekli, Ozan Sener, George Deligiannidis, and Murat A Erdogdu. 2020. Hausdorff dimension, heavy tails, and generalization in neural networks. Advances in Neural Information Processing Systems, 33:5138–5151. [find](https://scholar.google.com/scholar?q=Umut+Simsekli%2C+Ozan+Sener%2C+George+Deligiannidis%2C+and+Murat+A+Erdogdu%2E+2020%2E+Hausdorff+dimension%2C+heavy+tails%2C+and+generalization+in+neural+networks%2E+Advances+in+Neural+Information+Processing+Systems%2C+)
56. Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. 2013. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139– 1147. pmlr. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+James+Martens%2C+George+Dahl%2C+and+Geoffrey+Hinton%2E+2013%2E+On+the+importance+of+initialization+and+momentum+in+deep+learning%2E+In+International+conference+on+machine+learning%2C+pages+1139%E2%80%93+1)
57. Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, and 1 others. 2025. [Kimi k2: Open agentic intelligence](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-k2-open-agentic/). arXiv preprint arXiv:2507.20534.
58. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
59. Nikhil Vyas, Depen Morwani, Rosie Zhao, Mujin Kwun, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. 2024. Soap: Improving and stabilizing shampoo using adam. arXiv preprint arXiv:2409.11321. [find](https://scholar.google.com/scholar?q=Nikhil+Vyas%2C+Depen+Morwani%2C+Rosie+Zhao%2C+Mujin+Kwun%2C+Itai+Shapira%2C+David+Brandfonbrener%2C+Lucas+Janson%2C+and+Sham+Kakade%2E+2024%2E+Soap%3A+Improving+and+stabilizing+shampoo+using+adam%2E+arXiv+preprint+arXiv%3A24)
60. Jinbo Wang, Mingze Wang, Zhanpeng Zhou, Junchi Yan, and Lei Wu. 2025a. The sharpness disparity principle in transformers for accelerating language model pre-training. arXiv preprint arXiv:2502.19002. — [arXiv:2502.19002](https://arxiv.org/abs/2502.19002) [find](https://scholar.google.com/scholar?q=Jinbo+Wang%2C+Mingze+Wang%2C+Zhanpeng+Zhou%2C+Junchi+Yan%2C+and+Lei+Wu%2E+2025a%2E+The+sharpness+disparity+principle+in+transformers+for+accelerating+language+model+pre%2Dtraining%2E+arXiv+preprint+arXiv%3A2502%2E19002%2E+)
61. Shuche Wang, Fengzhuo Zhang, Jiaxiang Li, Cunxiao Du, Chao Du, Tianyu Pang, Zhuoran Yang, Mingyi Hong, and Vincent YF Tan. 2025b. Muon outperforms adam in tail-end associative memory learning. arXiv preprint arXiv:2509.26030. — [arXiv:2509.26030](https://arxiv.org/abs/2509.26030) [find](https://scholar.google.com/scholar?q=Shuche+Wang%2C+Fengzhuo+Zhang%2C+Jiaxiang+Li%2C+Cunxiao+Du%2C+Chao+Du%2C+Tianyu+Pang%2C+Zhuoran+Yang%2C+Mingyi+Hong%2C+and+Vincent+YF+Tan%2E+2025b%2E+Muon+outperforms+adam+in+tail%2Dend+associative+memory+learning%2E+arXiv+p)
62. Zhichao Wang, Andrew Engel, Anand D Sarwate, Ioana Dumitriu, and Tony Chiang. 2023. Spectral evolution and invariance in linear-width neural networks. Advances in neural information processing systems, 36:20695–20728. [find](https://scholar.google.com/scholar?q=Zhichao+Wang%2C+Andrew+Engel%2C+Anand+D+Sarwate%2C+Ioana+Dumitriu%2C+and+Tony+Chiang%2E+2023%2E+Spectral+evolution+and+invariance+in+linear%2Dwidth+neural+networks%2E+Advances+in+neural+information+processing+systems)
63. Kaiyue Wen, David Hall, Tengyu Ma, and Percy Liang. 2025. Fantastic pretraining optimizers and where to find them. arXiv preprint arXiv:2509.02046. [find](https://scholar.google.com/scholar?q=Kaiyue+Wen%2C+David+Hall%2C+Tengyu+Ma%2C+and+Percy+Liang%2E+2025%2E+Fantastic+pretraining+optimizers+and+where+to+find+them%2E+arXiv+preprint+arXiv%3A2509%2E02046%2E)
64. Yaoqing Yang, Ryan Theisen, Liam Hodgkinson, Joseph E Gonzalez, Kannan Ramchandran, Charles H Martin, and Michael W Mahoney. 2023. Test accuracy vs. generalization gap: Model selection in nlp without accessing training or testing data. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Di… [find](https://scholar.google.com/scholar?q=Yaoqing+Yang%2C+Ryan+Theisen%2C+Liam+Hodgkinson%2C+Joseph+E+Gonzalez%2C+Kannan+Ramchandran%2C+Charles+H+Martin%2C+and+Michael+W+Mahoney%2E+2023%2E+Test+accuracy+vs%2E+generalization+gap%3A+Model+selection+in+nlp+without+)
65. Huizhuo Yuan, Yifeng Liu, Shuang Wu, Xun Zhou, and Quanquan Gu. 2024. Mars: Unleashing the power of variance reduction for training large models. arXiv preprint arXiv:2411.10438. — [arXiv:2411.10438](https://arxiv.org/abs/2411.10438) [find](https://scholar.google.com/scholar?q=Huizhuo+Yuan%2C+Yifeng+Liu%2C+Shuang+Wu%2C+Xun+Zhou%2C+and+Quanquan+Gu%2E+2024%2E+Mars%3A+Unleashing+the+power+of+variance+reduction+for+training+large+models%2E+arXiv+preprint+arXiv%3A2411%2E10438%2E+%E2%80%94+%5BarXiv%3A2411%2E10438%5D%28)
66. Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, and 1 others. 2025. [Glm-4.5: Agentic, reasoning, and coding (arc) foundation models](https://md.research.elamur.ai/free/en/papers/team-2025-4-glm-4-5-agentic/). arXiv preprint arXiv:2508.06471.
67. Chunhui Zhang, Yiren Jian, Zhongyu Ouyang, and Soroush Vosoughi. 2024a. Working memory identifies reasoning limits in language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16896–16922. [find](https://scholar.google.com/scholar?q=Chunhui+Zhang%2C+Yiren+Jian%2C+Zhongyu+Ouyang%2C+and+Soroush+Vosoughi%2E+2024a%2E+Working+memory+identifies+reasoning+limits+in+language+models%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Nat)
68. Chunhui Zhang, Yiren Jian, Zhongyu Ouyang, and Soroush Vosoughi. 2025. Pretrained image-text models are secretly video captioners. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies. [find](https://scholar.google.com/scholar?q=Chunhui+Zhang%2C+Yiren+Jian%2C+Zhongyu+Ouyang%2C+and+Soroush+Vosoughi%2E+2025%2E+Pretrained+image%2Dtext+models+are+secretly+video+captioners%2E+In+Proceedings+of+the+2025+Conference+of+the+Nations+of+the+Americas+)
69. Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Diederik P Kingma, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. 2024b. Adam-mini: Use fewer learning rates to gain more. arXiv preprint arXiv:2406.16793. [find](https://scholar.google.com/scholar?q=Yushun+Zhang%2C+Congliang+Chen%2C+Ziniu+Li%2C+Tian+Ding%2C+Chenwei+Wu%2C+Diederik+P+Kingma%2C+Yinyu+Ye%2C+Zhi%2DQuan+Luo%2C+and+Ruoyu+Sun%2E+2024b%2E+Adam%2Dmini%3A+Use+fewer+learning+rates+to+gain+more%2E+arXiv+preprint+arXiv%3A2)
70. Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. 2024. Galore: Memory-efficient llm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507. [find](https://scholar.google.com/scholar?q=Jiawei+Zhao%2C+Zhenyu+Zhang%2C+Beidi+Chen%2C+Zhangyang+Wang%2C+Anima+Anandkumar%2C+and+Yuandong+Tian%2E+2024%2E+Galore%3A+Memory%2Defficient+llm+training+by+gradient+low%2Drank+projection%2E+arXiv+preprint+arXiv%3A2403%2E03507)
71. Beitong Zhou, Jun Liu, Weigao Sun, Ruijuan Chen, Claire J Tomlin, and Ye Yuan. 2020. pbsgd: Powered stochastic gradient descent methods for accelerated non-convex optimization. In IJCAI, pages 3258–3266. [find](https://scholar.google.com/scholar?q=Beitong+Zhou%2C+Jun+Liu%2C+Weigao+Sun%2C+Ruijuan+Chen%2C+Claire+J+Tomlin%2C+and+Ye+Yuan%2E+2020%2E+pbsgd%3A+Powered+stochastic+gradient+descent+methods+for+accelerated+non%2Dconvex+optimization%2E+In+IJCAI%2C+pages+3258%E2%80%9332)
72. Yefan Zhou, Tianyu Pang, Keqin Liu, Michael W Mahoney, and Yaoqing Yang. 2023. Temperature balancing, layer-wise weight analysis, and neural network training. Advances in Neural Information Processing Systems, 36:63542–63572. [find](https://scholar.google.com/scholar?q=Yefan+Zhou%2C+Tianyu+Pang%2C+Keqin+Liu%2C+Michael+W+Mahoney%2C+and+Yaoqing+Yang%2E+2023%2E+Temperature+balancing%2C+layer%2Dwise+weight+analysis%2C+and+neural+network+training%2E+Advances+in+Neural+Information+Processing)

*Updated 31 October 2026*