---
title: "LoRA-FA — paper by Zhang et al. (2023) — summary and ideas"
description: "The authors argue that LoRA's two factors contribute unequally. 48 references, cited by 47 papers here."
authors: ["Longteng Zhang", "Lin Zhang", "Shaohuai Shi", "Xiaowen Chu", "Bo Li"]
year: 2023
venue: "arXiv:2308.03303"
source: "https://arxiv.org/abs/2308.03303"
doi: "10.48550/arxiv.2308.03303"
arxiv: "2308.03303"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2023-lora-fa-efficient-effective-low/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2023-lora-fa-efficient-effective-low/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/) / LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning

# LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning

**Authors:** Longteng Zhang, [Lin Zhang](https://md.research.elamur.ai/free/en/authors/lin-zhang/), Shaohuai Shi, [Xiaowen Chu](https://md.research.elamur.ai/free/en/authors/xiaowen-chu/) and [Bo Li](https://md.research.elamur.ai/free/en/authors/bo-li/) · **Year:** 2023 · **Venue:** arXiv:2308.03303 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2308.03303) · [PDF](https://arxiv.org/pdf/2308.03303) · [DOI 10.48550/arxiv.2308.03303](https://doi.org/10.48550/arxiv.2308.03303) · [arXiv 2308.03303](https://arxiv.org/abs/2308.03303) · [OpenAlex](https://openalex.org/W4385681736)

**Citations:** 11 citations (OpenAlex, 2026-10-14) · [47](https://md.research.elamur.ai/free/en/papers/zhang-2023-lora-fa-efficient-effective-low/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Full-parameter fine-tuning of large language models needs enormous GPU memory. LoRA is the common cheaper option, but it often falls short of Full-FT because its updates are restricted to a low-rank gradient. The authors argue that LoRA's two factors contribute unequally. The product of the projection-down matrix A and the projection-up matrix B reduces to what is effectively a one-layer linear regression, so fixing A at its starting value costs no expressive power (Theorem 3.1). Building on this, LoRA-FA freezes a randomly initialized A and trains only B. It also adjusts B's gradient with a closed-form correction, chosen so that the induced weight update is as close as possible to the full gradient in Frobenius norm. With A frozen, a layer only has to keep its small rank-r intermediate for backpropagation, not its full-width input. This lowers activation memory and computation. Tests on GLUE, GSM8K, MT-Bench and HumanEval show quality comparable to other PEFT methods and to Full-FT. The approach is useful when fine-tuning large models on limited hardware, especially with long sequences or large batches.

## Key points
- **Per-layer memory (Table 1, 16-bit mixed precision):** activation cost is 2bsd + 2bsr for LoRA and only 2bsr for LoRA-FA. LoRA-FA also halves the trainable parameters (dr vs. 2dr) and the optimizer state (4dr vs. 8dr).
- **Variants and baselines:** the paper also derives LoRA-FB (A trained, B frozen), which has its own closed-form gradient fix but brings no activation savings. LoRA-FA is compared against related gradient-approximation methods such as LoRA-GA and LoRA-Pro.
- **Initialization of A:** an appendix argues that Gaussian-random initialization is the safer default compared with truncated-SVD (PiSSA-style). SVD-based A can completely miss gradients orthogonal to its top-r subspace. A random A captures, on average, an r/m share of the gradient's energy.
- **Compatibility and limits:** LoRA-FA combines with QLoRA-style 8-bit/4-bit weight quantization (2× or 4× smaller weight memory), ZeRO stage-3 sharding, and selective recomputation such as FlashAttention. It cannot remove other Transformer activations (e.g., attention inputs and outputs), and it remains a low-rank method.
- **Bibliographic note:** the provided PDF (arXiv v3) is titled "LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning", which differs from the title given for this entry. The extract omitted the results section, so per-benchmark scores are not reported here.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Harbin Institute of Technology](https://md.research.elamur.ai/free/en/universities/harbin-institute-of-technology/) — University, Harbin, China · 158 works · 28,841 citations
- [Hong Kong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/hong-kong-university-of-science-and-technology/) — University, Hong Kong, China · 142 works · 25,136 citations
- [The Hong Kong University of Science and Technology (Guangzhou)](https://md.research.elamur.ai/free/en/universities/the-hong-kong-university-of-science-and-technology-guangzhou/) — University, Guangzhou, China · 53 works · 1,367 citations
- [Hong Kong Baptist University](https://md.research.elamur.ai/free/en/universities/hong-kong-baptist-university/) — University, Hong Kong, China · 7 works · 246 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 242 works

## Cite

Cite as: Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu and Bo Li. 2023. LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning. arXiv:2308.03303.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhang2023lora,
  title = {{LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning}},
  author = {Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu and Bo Li},
  year = {2023},
  doi = {10.48550/arxiv.2308.03303},
  eprint = {2308.03303},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.03303}
}
```

- [← previous EduChat: A Large-Scale Language Model-based Chatbot System for Intelligent Education](https://md.research.elamur.ai/free/en/papers/dan-2023-educhat-large-scale-language-model/)
- [next → MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning](https://md.research.elamur.ai/free/en/papers/zhao-2023-mmicl-empowering-vision-language-model/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/)

## References

48 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marah Abdin and Sam Ade Jacobs et al. [Phi-3 technical report: A highly capable language model locally on your phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/), 2024. URL https://arxiv.org/abs/2404.14219. [↗](https://arxiv.org/abs/2404.14219)
2. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, and Eric Chu et al. [Palm 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/), 2023.
3. Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, et al. [Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues](https://md.research.elamur.ai/free/en/papers/bai-2024-mt-bench-101-fine-grained/). arXiv preprint arXiv:2402.14762, 2024.
4. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+et+al%2E+Training+a+helpful+and+harmless+assistant+with+reinforcem)
5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
6. Kerim Büyükakyüz. Olora: Orthonormal low-rank adaptation of large language models, 2024. URL https://arxiv.org/abs/2406.01775. [↗](https://arxiv.org/abs/2406.01775)
7. Mark Chen and et al. Jerry Tworek. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374, 2021.
8. Aakanksha Chowdhery and Sharan Narang et al. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/), 2022. URL https://arxiv.org/abs/2204.02311. [↗](https://arxiv.org/abs/2204.02311)
9. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
10. Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359, 2022. [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Dan+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+Re%CC%81%2E+Flashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A1634)
11. DeepSeek-AI, Aixin Liu, and et al. Bei Feng. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024. URL https://arxiv.org/abs/2405.04434. [↗](https://arxiv.org/abs/2405.04434)
12. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems, 2022. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Mike+Lewis%2C+Younes+Belkada%2C+and+Luke+Zettlemoyer%2E+Llm%2E+int8+%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2E+Advances+in+neural+information+processing+systems%2C+2022%2E)
13. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. [Qlora: Efficient finetuning of quantized llms](https://md.research.elamur.ai/free/en/papers/dettmers-2023-qlora-efficient-finetuning-quantized-llms/). arXiv preprint arXiv:2305.14314, 2023.
14. Abhimanyu Dubey and Abhinav Jauhri et al. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/), 2024. URL https://arxiv.org/abs/2407.21783. [↗](https://arxiv.org/abs/2407.21783)
15. Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models, 2024. URL https://arxiv.org/abs/2402.12354. [↗](https://arxiv.org/abs/2402.12354)
16. Haonan He, Peng Ye, Yuchen Ren, Yuan Yuan, Luyang Zhou, Shucun Ju, and Lei Chen. Gora: Gradient-driven adaptive low rank adaptation, 2025. URL https://arxiv.org/abs/2502. 12171. [↗](https://arxiv.org/abs/2502)
17. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790–2799. PMLR, 2019. [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+De+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+Parameter%2Defficient+transfer+learning+for+nlp%2E+In+Intern)
18. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations, 2022. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=nZeVKeeFYf9. [↗](https://openreview.net/forum?)
19. Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, Roy Ka-Wei Lee, Lidong Bing, and Soujanya Poria. [Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models](https://md.research.elamur.ai/free/en/papers/hu-2023-llm-adapters-adapter-family-parameter/). arXiv preprint arXiv:2304.01933, 2023.
20. Dawid J. Kopiczko, Tijmen Blankevoort, and Yuki M. Asano. [Vera: Vector-based random matrix adaptation](https://md.research.elamur.ai/free/en/papers/kopiczko-2023-vera-vector-based-random-matrix/), 2024. URL https://arxiv.org/abs/2310.11454. [↗](https://arxiv.org/abs/2310.11454)
21. Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. [Reducing activation recomputation in large transformer models](https://md.research.elamur.ai/free/en/papers/korthikanti-2022-reducing-activation-recomputation-large-transformer/). Proceedings of Machine Learning and Systems, 5, 2023.
22. Brian Lester, Rami Al-Rfou, and Noah Constant. [The power of scale for parameter-efficient prompt tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045–3059, 2021.
23. Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582–4597, 20… [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11th+Inte)
24. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353, 2024. [find](https://scholar.google.com/scholar?q=Shih%2DYang+Liu%2C+Chien%2DYi+Wang%2C+Hongxu+Yin%2C+Pavlo+Molchanov%2C+Yu%2DChiang+Frank+Wang%2C+Kwang%2DTing+Cheng%2C+and+Min%2DHung+Chen%2E+Dora%3A+Weight%2Ddecomposed+low%2Drank+adaptation%2E+arXiv+preprint+arXiv%3A2402%2E09353%2C+2024)
25. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/), 2019.
26. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Conference on Learning Representations, 2017. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+Decoupled+weight+decay+regularization%2E+International+Conference+on+Learning+Representations%2C+2017%2E)
27. Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, and Sayak Paul. Peft: State-of-the-art parameter-efficient fine-tuning methods. [https://github.com/huggingface/](https://github.com/huggingface/) peft, 2022. [↗](https://github.com/huggingface/)
28. Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models, 2024. URL https://arxiv.org/abs/2404.02948. [↗](https://arxiv.org/abs/2404.02948)
29. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023.
30. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35: 27730–27744, 2022.
31. Kaustubh Ponkshe, Raghav Singhal, Eduard Gorbunov, Alexey Tumanov, Samuel Horvath, and Praneeth Vepakomma. Initialization using update approximation is a silver bullet for extremely efficient low-rank fine-tuning, 2025. URL https://arxiv.org/abs/2411.19557. — [arXiv:2411.19557](https://arxiv.org/abs/2411.19557) [↗](https://arxiv.org/abs/2411.19557)
32. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–16. IEEE, 2020. [find](https://scholar.google.com/scholar?q=Samyam+Rajbhandari%2C+Jeff+Rasley%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+Zero%3A+Memory+optimizations+toward+training+trillion+parameter+models%2E+In+SC20%3A+International+Conference+for+High+Performance+Computing)
33. Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning, 2024. URL https://arxiv.org/abs/2404.19245. — [arXiv:2404.19245](https://arxiv.org/abs/2404.19245) [↗](https://arxiv.org/abs/2404.19245)
34. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/), 2023a.
35. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, and Shruti Bhosale et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/), 2023b.
36. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In International Conference on Learning Representations, 2019.
37. Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approximation, 2024a. URL https://arxiv.org/abs/2407.05000. — [arXiv:2407.05000](https://arxiv.org/abs/2407.05000) [↗](https://arxiv.org/abs/2407.05000)
38. Xindi Wang, Mahsa Salmani, Parsa Omidi, Xiangyu Ren, Mehdi Rezagholizadeh, and Armaghan Eshaghi. [Beyond the limits: A survey of techniques to extend the context length in large language models](https://md.research.elamur.ai/free/en/papers/wang-2024-limits-survey-techniques-extend-context/), 2024b. URL https://arxiv.org/abs/2402.02244. [↗](https://arxiv.org/abs/2402.02244)
39. Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. Lora-pro: Are low-rank adapters properly optimized?, 2024c. URL https://arxiv.org/abs/2407.18242. — [arXiv:2407.18242](https://arxiv.org/abs/2407.18242) [↗](https://arxiv.org/abs/2407.18242)
40. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). In International Conference on Learning Representations, 2021.
41. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://md.research.elamur.ai/free/en/papers/xu-2023-wizardlm-empowering-large-pre-trained/). In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.ne…](https://openreview.ne…) [↗](https://openreview.ne…)
42. An Yang and et al. Anfeng Li. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/), 2025. URL https://arxiv.org/abs/ 2505.09388. [↗](https://arxiv.org/abs/)
43. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). arXiv preprint arXiv:2309.12284, 2023.
44. Juzheng Zhang, Jiacheng You, Ashwinee Panda, and Tom Goldstein. Lori: Reducing cross-task interference in multi-task low-rank adaptation, 2025. URL https://arxiv.org/abs/2504. 07448. [↗](https://arxiv.org/abs/2504)
45. Longteng Zhang, Xiang Liu, Zeyu Li, Xinglin Pan, Peijie Dong, Ruibo Fan, Rui Guo, Xin Wang, Qiong Luo, Shaohuai Shi, and Xiaowen Chu. Dissecting the runtime performance of the training, fine-tuning, and inference of large language models, 2023a. URL https://arxiv.org/abs/ 2311.03687. [↗](https://arxiv.org/abs/)
46. Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. [Adalora: Adaptive budget allocation for parameter-efficient fine-tuning](https://md.research.elamur.ai/free/en/papers/zhang-2023-adalora-adaptive-budget-allocation-parameter/), 2023b. URL https://arxiv.org/abs/2303.10512. [↗](https://arxiv.org/abs/2303.10512)
47. Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinterpreter: Integrating code generation with execution and refinement, 2025. URL https://arxiv.org/abs/2402.14658. [↗](https://arxiv.org/abs/2402.14658)
48. Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz Sáez de Ocáriz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. Asymmetry in low-rank adapters of foundation models, 2024. URL [https://arxiv.org/](https://arxiv.org/) abs/2402.16842. — [arXiv:2402.16842](https://arxiv.org/abs/2402.16842) [↗](https://arxiv.org/)

*Updated 1 November 2026*