---
title: "ReLoRA: High-Rank Training Through Low-Rank Updates (2023)"
description: "Pre-training large language models requires a great deal of memory and compute. 52 references, cited by 64 papers here."
authors: ["Vladislav Lialin", "Namrata Shivagunde", "Sherin Muckatira", "Anna Rumshisky"]
year: 2023
venue: "arXiv:2307.05695"
source: "https://arxiv.org/abs/2307.05695"
doi: "10.48550/arxiv.2307.05695"
arxiv: "2307.05695"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lialin-2023-relora-high-rank-training-through/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lialin-2023-relora-high-rank-training-through/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / ReLoRA: High-Rank Training Through Low-Rank Updates

# ReLoRA: High-Rank Training Through Low-Rank Updates

**Authors:** [Vladislav Lialin](https://md.research.elamur.ai/free/en/authors/vladislav-lialin/), Namrata Shivagunde, Sherin Muckatira and [Anna Rumshisky](https://md.research.elamur.ai/free/en/authors/anna-rumshisky/) · **Year:** 2023 · **Venue:** arXiv:2307.05695 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2307.05695) · [PDF](https://arxiv.org/pdf/2307.05695) · [DOI 10.48550/arxiv.2307.05695](https://doi.org/10.48550/arxiv.2307.05695) · [arXiv 2307.05695](https://arxiv.org/abs/2307.05695) · [OpenAlex](https://openalex.org/W4384263459)

**Citations:** 10 citations (OpenAlex, 2026-10-14) · [64](https://md.research.elamur.ai/free/en/papers/lialin-2023-relora-high-rank-training-through/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Pre-training large language models requires a great deal of memory and compute. Parameter-efficient fine-tuning methods such as LoRA cut these costs, but on their own they cannot pre-train a model well, because each LoRA adapter can only make a low-rank change. ReLoRA addresses this by repeatedly merging the LoRA factors into the frozen weights and then starting new factors. A sum of many low-rank updates can reach high rank. To make this restart scheme work, the authors add three things: a brief full-rank warm start, partial pruning of the Adam state at each merge, and a "jagged" cosine learning-rate schedule that drops to zero and quickly warms up again after every reset. They tested LLaMA-style transformers from 60M to 1.3B parameters, trained on C4 with Chinchilla-sized token budgets. ReLoRA came close to full-rank perplexity and clearly beat both plain LoRA and a small full-rank model with the same number of trainable parameters. It also reduced memory and training time. This makes pre-training more accessible on modest hardware.

## Key points
- **Setup:** Models of 60M, 130M, 250M, 350M and 1.3B parameters were trained on C4 for one epoch. The architecture follows LLaMA (RMSNorm, SwiGLU, rotary embeddings). All runs used bfloat16 and FlashAttention. LoRA/ReLoRA rank was r = 128.
- **Perplexity:** At 1.3B, ReLoRA reached 17.27 against 16.83 for full training, with 250M trainable parameters. At 250M it reached 22.48 against 22.39. The Control baseline (a smaller full-rank model) and plain LoRA did noticeably worse, and LoRA without a warm start reached 57.11 at 350M.
- **Downstream results:** On GLUE, a 350M model pre-trained with ReLoRA averaged 73.14, close to 73.54 for the full-rank model.
- **Efficiency:** Each GPU's memory footprint drops by as much as 5.5GB with ReLoRA. Training was about 9% faster on 8xA100 and 20–40% faster on cheaper hardware, and the gains grow with model size.
- **Recipe details:** The warm start covered roughly a quarter of the training updates. Resets happened every few thousand steps, removing about 99% of the optimizer state by magnitude pruning. The ablations showed that both the optimizer reset and the jagged schedule are needed to beat LoRA.

## Interesting ideas
- **Repeated merge-and-restart:** Merging an adapter and starting a fresh one turns a fixed low-rank budget into a cumulative high-rank update. This trick could extend adapter-based continued pre-training or domain adaptation, for example of a model to Ukrainian text, beyond what a single adapter can capture.
- **Clearing optimizer momentum:** Adam's stored moments would push a new adapter back into the subspace the previous adapter already explored. Pruning most of that state at each reset avoids this. The general lesson is that any method that re-initializes parameters mid-training should also think about stale optimizer statistics.
- **Jagged learning-rate schedule:** Dropping the learning rate to zero and re-warming it over a short window after each structural change kept training stable. This is a cheap safeguard for other mid-run interventions, such as pruning, growing layers or swapping modules.
- **Full-rank warm start before going low-rank:** Spending the early part of training at full rank mattered most in the ablations, which suggests early training sets up much of what later training builds on. Hybrid schedules, with expensive training early and cheap training later, may be worth trying in other resource-limited pipelines.
- **Control baseline with matched trainable parameters:** Comparing against a small full-rank model with the same trainable-parameter count isolates whether a method truly adds capacity. This is a fair evaluation habit for any parameter-efficient technique.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Massachusetts Lowell](https://md.research.elamur.ai/free/en/universities/university-of-massachusetts-lowell/) — University, Lowell, United States · 8 works · 3,758 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

## Cite

Cite as: Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira and Anna Rumshisky. 2023. ReLoRA: High-Rank Training Through Low-Rank Updates. arXiv:2307.05695.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{lialin2023relora,
  title = {{ReLoRA: High-Rank Training Through Low-Rank Updates}},
  author = {Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira and Anna Rumshisky},
  year = {2023},
  doi = {10.48550/arxiv.2307.05695},
  eprint = {2307.05695},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.05695}
}
```

- [← previous Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker](https://md.research.elamur.ai/free/en/papers/sclar-2023-minding-language-models-lack-theory/)
- [next → Wider and Deeper LLM Networks are Fairer LLM Evaluators](https://md.research.elamur.ai/free/en/papers/zhang-2023-wider-deeper-llm-networks-fairer/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

52 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Aghajanyan, S. Gupta, and L. Zettlemoyer. [Intrinsic dimensionality explains the effectiveness of language model fine-tuning](https://md.research.elamur.ai/free/en/papers/aghajanyan-2021-intrinsic-dimensionality-explains-effectiveness-language/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volum…
2. Z. Allen-Zhu, Y. Li, and Z. Song. A convergence theory for deep learning via over-parameterization. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 242–252. PMLR, 09–15 J… [find](https://scholar.google.com/scholar?q=Z%2E+Allen%2DZhu%2C+Y%2E+Li%2C+and+Z%2E+Song%2E+A+convergence+theory+for+deep+learning+via+over%2Dparameterization%2E+In+K%2E+Chaudhuri+and+R%2E+Salakhutdinov%2C+editors%2C+Proceedings+of+the+36th+International+Conference+on+M)
3. S. Arora, N. Cohen, W. Hu, and Y. Luo. Implicit regularization in deep matrix factorization, 2019. [find](https://scholar.google.com/scholar?q=S%2E+Arora%2C+N%2E+Cohen%2C+W%2E+Hu%2C+and+Y%2E+Luo%2E+Implicit+regularization+in+deep+matrix+factorization%2C+2019%2E)
4. M. Belkin, D. J. Hsu, S. Ma, and S. Mandal. Reconciling modern machine-learning practice and the classical bias–variance trade-off. Proceedings of the National Academy of Sciences, 116:15849 – 15854, 2018. [find](https://scholar.google.com/scholar?q=M%2E+Belkin%2C+D%2E+J%2E+Hsu%2C+S%2E+Ma%2C+and+S%2E+Mandal%2E+Reconciling+modern+machine%2Dlearning+practice+and+the+classical+bias%E2%80%93variance+trade%2Doff%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+116%3A15849+%E2%80%93+15854%2C+)
5. S. Bhojanapalli, C. Yun, A. S. Rawat, S. Reddi, and S. Kumar. Low-rank bottleneck in multi-head attention models. In International Conference on Machine Learning, pages 864–873. PMLR, 2020. [find](https://scholar.google.com/scholar?q=S%2E+Bhojanapalli%2C+C%2E+Yun%2C+A%2E+S%2E+Rawat%2C+S%2E+Reddi%2C+and+S%2E+Kumar%2E+Low%2Drank+bottleneck+in+multi%2Dhead+attention+models%2E+In+International+Conference+on+Machine+Learning%2C+pages+864%E2%80%93873%2E+PMLR%2C+2020%2E)
6. S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driessche, J.-B. Lespiau, B. Damoc, A. Clark, D. De Las Casas, A. Guy, J. Menick, R. Ring, T. Hennigan, S. Huang, L. Maggiore, C. Jones, A. Cassirer, A. Brock, M. Paganini, G. Irving, O. Vinyals, S. Osindero, K. S… [find](https://scholar.google.com/scholar?q=S%2E+Borgeaud%2C+A%2E+Mensch%2C+J%2E+Hoffmann%2C+T%2E+Cai%2C+E%2E+Rutherford%2C+K%2E+Millican%2C+G%2E+B%2E+Van+Den+Driessche%2C+J%2E%2DB%2E+Lespiau%2C+B%2E+Damoc%2C+A%2E+Clark%2C+D%2E+De+Las+Casas%2C+A%2E+Guy%2C+J%2E+Menick%2C+R%2E+Ring%2C+T%2E+Hennigan%2C+S%2E+Huang%2C)
7. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner… [find](https://scholar.google.com/scholar?q=T%2E+Brown%2C+B%2E+Mann%2C+N%2E+Ryder%2C+M%2E+Subbiah%2C+J%2E+D%2E+Kaplan%2C+P%2E+Dhariwal%2C+A%2E+Neelakantan%2C+P%2E+Shyam%2C+G%2E+Sastry%2C+A%2E+Askell%2C+S%2E+Agarwal%2C+A%2E+Herbert%2DVoss%2C+G%2E+Krueger%2C+T%2E+Henighan%2C+R%2E+Child%2C+A%2E+Ramesh%2C+D%2E+Ziegle)
8. Curran Associates, Inc., 2020. URL [https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf](https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf). [↗](https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf)
9. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. M. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. C. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
10. T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Re. Flashattention: Fast and memory-efficient exact attention with IO-awareness. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. URL [https://openreview.net/forum?id=H4DqfPSibmx](https://openreview.net/forum?id=H4DqfPSibmx). [↗](https://openreview.net/forum?id=H4DqfPSibmx)
11. T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer. GPT3.int8(): 8-bit matrix multiplication for transformers at scale. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. URL [https://openreview.net/forum?id=dXiGWqBoxaD](https://openreview.net/forum?id=dXiGWqBoxaD). [↗](https://openreview.net/forum?id=dXiGWqBoxaD)
12. T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. Qlora: Efficient finetuning of quantized llms. ArXiv, abs/2305.14314, 2023. URL [https://api.semanticscholar.org/CorpusID:258841328](https://api.semanticscholar.org/CorpusID:258841328). [↗](https://api.semanticscholar.org/CorpusID:258841328)
13. A. Edalati, M. S. Tahaei, I. Kobyzev, V. Nia, J. J. Clark, and M. Rezagholizadeh. Krona: Parameter efficient tuning with kronecker adapter. ArXiv, abs/2212.10650, 2022. URL [https://api.semanticscholar.org/](https://api.semanticscholar.org/) CorpusID:254926823. — [arXiv:2212.10650](https://arxiv.org/abs/2212.10650) [↗](https://api.semanticscholar.org/)
14. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23(1), jan 2022. ISSN 1532-4435. [find](https://scholar.google.com/scholar?q=W%2E+Fedus%2C+B%2E+Zoph%2C+and+N%2E+Shazeer%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+J%2E+Mach%2E+Learn%2E+Res%2E%2C+23%281%29%2C+jan+2022%2E+ISSN+1532%2D4435%2E)
15. J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=rJl-b3RcF7. [↗](https://openreview.net/forum?)
16. J. Frankle, G. Karolina Dziugaite, D. M. Roy, and M. Carbin. Stabilizing the lottery ticket hypothesis. arXiv e-prints, pages arXiv–1903, 2019. [find](https://scholar.google.com/scholar?q=J%2E+Frankle%2C+G%2E+Karolina+Dziugaite%2C+D%2E+M%2E+Roy%2C+and+M%2E+Carbin%2E+Stabilizing+the+lottery+ticket+hypothesis%2E+arXiv+e%2Dprints%2C+pages+arXiv%E2%80%931903%2C+2019%2E)
17. B. Ghorbani, O. Firat, M. Freitag, A. Bapna, M. Krikun, X. García, C. Chelba, and C. Cherry. Scaling laws for neural machine translation. ArXiv, abs/2109.07740, 2021. — [arXiv:2109.07740](https://arxiv.org/abs/2109.07740) [find](https://scholar.google.com/scholar?q=B%2E+Ghorbani%2C+O%2E+Firat%2C+M%2E+Freitag%2C+A%2E+Bapna%2C+M%2E+Krikun%2C+X%2E+Garc%C3%ADa%2C+C%2E+Chelba%2C+and+C%2E+Cherry%2E+Scaling+laws+for+neural+machine+translation%2E+ArXiv%2C+abs%2F2109%2E07740%2C+2021%2E+%E2%80%94+%5BarXiv%3A2109%2E07740%5D%28https%3A%2F%2Farxi)
18. K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. CoRR, abs/1502.01852, 2015. URL http://arxiv.org/abs/1502.01852. [↗](http://arxiv.org/abs/1502.01852)
19. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016. [find](https://scholar.google.com/scholar?q=K%2E+He%2C+X%2E+Zhang%2C+S%2E+Ren%2C+and+J%2E+Sun%2E+Deep+residual+learning+for+image+recognition%2E+2016+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+pages+770%E2%80%93778%2C+2016%2E)
20. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, O. Vinyals, J. W. Rae, and L. Sifre. An empirical analysi… [find](https://scholar.google.com/scholar?q=J%2E+Hoffmann%2C+S%2E+Borgeaud%2C+A%2E+Mensch%2C+E%2E+Buchatskaya%2C+T%2E+Cai%2C+E%2E+Rutherford%2C+D%2E+de+las+Casas%2C+L%2E+A%2E+Hendricks%2C+J%2E+Welbl%2C+A%2E+Clark%2C+T%2E+Hennigan%2C+E%2E+Noland%2C+K%2E+Millican%2C+G%2E+van+den+Driessche%2C+B%2E+Damoc%2C+A)
21. E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations, 2022. URL [https://openreview.net/forum?id=nZeVKeeFYf9](https://openreview.net/forum?id=nZeVKeeFYf9). [↗](https://openreview.net/forum?id=nZeVKeeFYf9)
22. Y. Idelbayev and M. A. Carreira-Perpinan. Low-rank compression of neural nets: Learning the rank of each layer. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8046–8056, 2020. doi: 10.1109/CVPR42600.2020.00807. — [doi:10.1109/cvpr42600.2020.00807](https://doi.org/10.1109/cvpr42600.2020.00807) [find](https://scholar.google.com/scholar?q=Y%2E+Idelbayev+and+M%2E+A%2E+Carreira%2DPerpinan%2E+Low%2Drank+compression+of+neural+nets%3A+Learning+the+rank+of+each+layer%2E+In+2020+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+pages+804)
23. A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, page 8580–8589, Red Hook, NY, USA, 2018. Curran Associates Inc. [find](https://scholar.google.com/scholar?q=A%2E+Jacot%2C+F%2E+Gabriel%2C+and+C%2E+Hongler%2E+Neural+tangent+kernel%3A+Convergence+and+generalization+in+neural+networks%2E+In+Proceedings+of+the+32nd+International+Conference+on+Neural+Information+Processing+Sys)
24. M. Jaderberg, A. Vedaldi, and A. Zisserman. Speeding up convolutional neural networks with low rank expansions. In Proceedings of the British Machine Vision Conference. BMVA Press, 2014. doi: http://dx.doi.org/10.5244/C.28.88. — [doi:10.5244/c.28.88](https://doi.org/10.5244/c.28.88) [↗](http://dx.doi.org/10.5244/C.28.88)
25. S. R. Kamalakara, A. F. Locatelli, B. Venkitesh, J. Ba, Y. Gal, and A. N. Gomez. Exploring low rank training of deep neural networks. ArXiv, abs/2209.13569, 2022. URL [https://api.semanticscholar.org/](https://api.semanticscholar.org/) CorpusID:252545358. — [arXiv:2209.13569](https://arxiv.org/abs/2209.13569) [↗](https://api.semanticscholar.org/)
26. J. Kaplan, S. McCandlish, T. J. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). ArXiv, abs/2001.08361, 2020.
27. U. Khandelwal, O. Levy, D. Jurafsky, L. Zettlemoyer, and M. Lewis. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations, 2020. URL [https://openreview.net/forum?id=HklBjCEKvH](https://openreview.net/forum?id=HklBjCEKvH). [↗](https://openreview.net/forum?id=HklBjCEKvH)
28. D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2015. [find](https://scholar.google.com/scholar?q=D%2E+P%2E+Kingma+and+J%2E+Ba%2E+Adam%3A+A+method+for+stochastic+optimization%2E+CoRR%2C+abs%2F1412%2E6980%2C+2015%2E)
29. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. Burges, L. Bottou, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL [https://proceedings.neurip…](https://proceedings.neurip…) [↗](https://proceedings.neurip…)
30. V. Lialin, V. Deshpande, and A. Rumshisky. [Scaling down to scale up: A guide to parameter-efficient fine-tuning](https://md.research.elamur.ai/free/en/papers/lialin-2023-scaling-down-scale-up-guide/), 2023.
31. R. Lin, C.-Y. Ko, Z. He, C. Chen, Y. Cheng, H. Yu, G. Chesi, and N. Wong. Hotcake: Higher order tucker articulated kernels for deeper cnn compression. In 2020 IEEE 15th International Conference on Solid-State & Integrated Circuit Technology (ICSICT), pages 1–4, 2020. doi: 10.1109/ICSICT49897.2020.92… — [doi:10.1109/icsict49897.2020.9278257](https://doi.org/10.1109/icsict49897.2020.9278257) [find](https://scholar.google.com/scholar?q=R%2E+Lin%2C+C%2E%2DY%2E+Ko%2C+Z%2E+He%2C+C%2E+Chen%2C+Y%2E+Cheng%2C+H%2E+Yu%2C+G%2E+Chesi%2C+and+N%2E+Wong%2E+Hotcake%3A+Higher+order+tucker+articulated+kernels+for+deeper+cnn+compression%2E+In+2020+IEEE+15th+International+Conference+on+Sol)
32. R. K. mahabadi, J. Henderson, and S. Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL [https://openreview.net/forum?id=bqGK5PyI6-N](https://openreview.net/forum?id=bqGK5PyI6-N). [↗](https://openreview.net/forum?id=bqGK5PyI6-N)
33. P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu. Mixed precision training. In International Conference on Learning Representations, 2018. URL [https://openreview.net/forum?id=r1gs9JgRZ](https://openreview.net/forum?id=r1gs9JgRZ). [↗](https://openreview.net/forum?id=r1gs9JgRZ)
34. P. Nakkiran, G. Kaplun, Y. Bansal, T. Yang, B. Barak, and I. Sutskever. Deep double descent: where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021, 2019. [find](https://scholar.google.com/scholar?q=P%2E+Nakkiran%2C+G%2E+Kaplun%2C+Y%2E+Bansal%2C+T%2E+Yang%2C+B%2E+Barak%2C+and+I%2E+Sutskever%2E+Deep+double+descent%3A+where+bigger+models+and+more+data+hurt%2E+Journal+of+Statistical+Mechanics%3A+Theory+and+Experiment%2C+2021%2C+2019)
35. A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training. 2018. [find](https://scholar.google.com/scholar?q=A%2E+Radford%2C+K%2E+Narasimhan%2C+T%2E+Salimans%2C+I%2E+Sutskever%2C+et+al%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+2018%2E)
36. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21 (140):1–67, 2020. URL [http://jmlr.org/papers/v21/20-074.html](http://jmlr.org/papers/v21/20-074.html). [↗](http://jmlr.org/papers/v21/20-074.html)
37. S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16, 2020. doi: 10.1109/SC41405.2020.00024. [find](https://scholar.google.com/scholar?q=S%2E+Rajbhandari%2C+J%2E+Rasley%2C+O%2E+Ruwase%2C+and+Y%2E+He%2E+Zero%3A+Memory+optimizations+toward+training+trillion+parameter+models%2E+In+SC20%3A+International+Conference+for+High+Performance+Computing%2C+Networking%2C+Sto)
38. S. Schotthöfer, E. Zangrando, J. Kusch, G. Ceruti, and F. Tudisco. Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems,… [find](https://scholar.google.com/scholar?q=S%2E+Schotth%C3%B6fer%2C+E%2E+Zangrando%2C+J%2E+Kusch%2C+G%2E+Ceruti%2C+and+F%2E+Tudisco%2E+Low%2Drank+lottery+tickets%3A+finding+efficient+low%2Drank+neural+networks+via+matrix+differential+equations%2E+In+S%2E+Koyejo%2C+S%2E+Mohamed%2C+A%2E+)
39. N. Shazeer. Glu variants improve transformer, 2020. [find](https://scholar.google.com/scholar?q=N%2E+Shazeer%2E+Glu+variants+improve+transformer%2C+2020%2E)
40. K. Simonyan and A. Zisserman. [Very deep convolutional networks for large-scale image recognition](https://md.research.elamur.ai/free/en/papers/simonyan-2014-very-deep-convolutional-networks-large/). In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1… [↗](http://arxiv.org/abs/1…)
41. S. P. Singh, G. Bachmann, and T. Hofmann. Analytic insights into structure and rank of neural network hessian maps. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL [https://openreview.net/forum?id=otDgw7LM7Nn](https://openreview.net/forum?id=otDgw7LM7Nn). [↗](https://openreview.net/forum?id=otDgw7LM7Nn)
42. J. Su, Y. Lu, S. Pan, B. Wen, and Y. Liu. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). ArXiv, abs/2104.09864, 2021.
43. Y. Sui, M. Yin, W. Yang, Y. Gong, J. Xiao, H. Phan, D. Ding, X. Xu, S. Liu, Z. Chen, and B. Yuan. ELRT: Towards efficient low-rank training for compact neural networks, 2023. URL [https://openreview.net/](https://openreview.net/) forum?id=TC39w69m8bB. [↗](https://openreview.net/)
44. Y.-L. Sung, J. Cho, and M. Bansal. Lst: Ladder side-tuning for parameter and memory efficient transfer learning. ArXiv, abs/2206.06522, 2022. URL [https://api.semanticscholar.org/CorpusID:249642544](https://api.semanticscholar.org/CorpusID:249642544). — [arXiv:2206.06522](https://arxiv.org/abs/2206.06522) [↗](https://api.semanticscholar.org/CorpusID:249642544)
45. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
46. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
47. S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020. [find](https://scholar.google.com/scholar?q=S%2E+Wang%2C+B%2E+Z%2E+Li%2C+M%2E+Khabsa%2C+H%2E+Fang%2C+and+H%2E+Ma%2E+Linformer%3A+Self%2Dattention+with+linear+complexity%2E+arXiv+preprint+arXiv%3A2006%2E04768%2C+2020%2E)
48. X. Yuan, P. H. P. Savarese, and M. Maire. Growing efficient deep networks by structured continuous sparsification. In International Conference on Learning Representations, 2021. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=wb3wxCObbRT. [↗](https://openreview.net/forum?)
49. B. Zhang and R. Sennrich. Root mean square layer normalization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL [https://proceedings.neurips.cc/paper_files/…](https://proceedings.neurips.cc/paper_files/…) [↗](https://proceedings.neurips.cc/paper_files/…)
50. C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017. URL [https://openreview.net/forum?id=Sy8gdB9xx](https://openreview.net/forum?id=Sy8gdB9xx). [↗](https://openreview.net/forum?id=Sy8gdB9xx)
51. C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64:107 – 115, 2021. [find](https://scholar.google.com/scholar?q=C%2E+Zhang%2C+S%2E+Bengio%2C+M%2E+Hardt%2C+B%2E+Recht%2C+and+O%2E+Vinyals%2E+Understanding+deep+learning+%28still%29+requires+rethinking+generalization%2E+Communications+of+the+ACM%2C+64%3A107+%E2%80%93+115%2C+2021%2E)
52. J. Zhao, Y. Zhang, B. Chen, F. Schäfer, and A. Anandkumar. Inrank: Incremental low-rank learning. arXiv preprint arXiv:2306.11250, 2023. — [arXiv:2306.11250](https://arxiv.org/abs/2306.11250) [find](https://scholar.google.com/scholar?q=J%2E+Zhao%2C+Y%2E+Zhang%2C+B%2E+Chen%2C+F%2E+Sch%C3%A4fer%2C+and+A%2E+Anandkumar%2E+Inrank%3A+Incremental+low%2Drank+learning%2E+arXiv+preprint+arXiv%3A2306%2E11250%2C+2023%2E+%E2%80%94+%5BarXiv%3A2306%2E11250%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2306%2E11250%29)

*Updated 24 October 2026*