---
title: "WeightLoRA: Keep Only Necessary Adapters (2026)"
description: "LoRA fine-tuning still uses a lot of memory, and practitioners must guess which layers should receive adapters. 54 references."
authors: ["Andrey Veprikov", "Vladimir Solodkin", "Zyl Alexander", "Andrey Savchenko", "Aleksandr Beznosikov"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.566/"
doi: "10.18653/v1/2026.acl-long.566"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/veprikov-2026-weightlora-keep-only-necessary-adapters/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/veprikov-2026-weightlora-keep-only-necessary-adapters/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/47/) / WeightLoRA: Keep Only Necessary Adapters

# WeightLoRA: Keep Only Necessary Adapters

**Authors:** Andrey Veprikov, Vladimir Solodkin, Zyl Alexander, Andrey Savchenko and Aleksandr Beznosikov · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.566/) · [PDF](https://aclanthology.org/2026.acl-long.566.pdf) · [DOI 10.18653/v1/2026.acl-long.566](https://doi.org/10.18653/v1/2026.acl-long.566) · [OpenAlex](https://openalex.org/W4415134425)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
LoRA fine-tuning still uses a lot of memory, and practitioners must guess which layers should receive adapters. WeightLoRA addresses this by learning which adapters matter. Every adapter, meaning the low-rank update on a single projection such as a query, key or value matrix, gets one trainable scalar weight. During a brief warm-up, the adapters and these weights are trained together. A sparsity constraint limits how many weights can be nonzero, and it is enforced with the StoIHT hard-thresholding algorithm. After warm-up, adapters whose weight is zero are dropped. The K survivors are reinitialized, and training continues as ordinary LoRA. A variant called WeightLoRA+ spends the freed parameter budget on higher ranks for the retained adapters. The methods were evaluated with DeBERTaV3-base, BART-large, Llama3-7B and Qwen3-8B on GLUE, LLM reasoning benchmarks, question answering and generation tasks. WeightLoRA performs on par with or close to standard LoRA while training about 2–3× fewer parameters. WeightLoRA+ beats LoRA at an equal parameter budget in almost every case. This makes it a cheap, LoRA-compatible way to choose where adapters go.

## Key points
- **Default settings:** warm-up lasts at most about 0.5% of training steps, and K = n/2 adapters are kept. Ablations indicate that these defaults work well enough in practice.
- **DeBERTaV3-base on GLUE:** LoRA trains 442K parameters. WLoRA with K=10 trains only 123K (0.07%) and still scores competitively, for example 0.9473 vs 0.9450 on SST-2 and 0.8706 vs 0.8797 on MNLI.
- **Baselines compared:** AdaLoRA, IncreLoRA, EVA, MELoRA, Dynamic LoRA, ALoRA, FlexLoRA, LoRA-drop, PrunedLoRA and full fine-tuning.
- **Selection check:** a first-order Taylor analysis on DeBERTa suggests that adapters differ widely in how much they affect the loss. The adapters WeightLoRA picks largely match those with large gradient–adapter dot products.
- **Code:** released at github.com/brain-lab-research/WLoRA.
- **Stated limitations:** the warm-up adds hyperparameters, selection works only on whole adapters, and adapter placement must be fixed before training.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [National Research University Higher School of Economics](https://md.research.elamur.ai/free/en/universities/national-research-university-higher-school-of-economics/) — University, Moscow, Russia · 23 works · 39,585 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

## Cite

Cite as: Andrey Veprikov, Vladimir Solodkin, Zyl Alexander, Andrey Savchenko and Aleksandr Beznosikov. 2026. WeightLoRA: Keep Only Necessary Adapters. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{veprikov2026weightlora,
  title = {{WeightLoRA: Keep Only Necessary Adapters}},
  author = {Andrey Veprikov, Vladimir Solodkin, Zyl Alexander, Andrey Savchenko and Aleksandr Beznosikov},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.566},
  url = {https://doi.org/10.18653/v1/2026.acl-long.566}
}
```

- [← previous Weight Tying Biases Token Embeddings Towards the Output Space](https://md.research.elamur.ai/free/en/papers/lopardo-2026-weight-tying-biases-token-embeddings/)
- [next → wenbin@EEUCA 2026: MoEs-VaxAgent, A Two-Stage Framework for Multimodal Vaccine Critical Meme Detection](https://md.research.elamur.ai/free/en/papers/shen-2026-wenbin-eeuca-2026-moes-vaxagent/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/47/)

## References

54 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. AI@Meta. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). Preprint, arXiv:2407.21783.
2. Vladimir Bogachev, Vladimir Aletov, Alexander Molozhavenko, Denis Bobkov, Vera Soboleva, Aibek Alanov, and Maxim Rakhuba. 2025. Lora meets riemannion: Muon optimizer for parametrization-independent low-rank adapters. arXiv preprint arXiv:2507.12142. — [arXiv:2507.12142](https://arxiv.org/abs/2507.12142) [find](https://scholar.google.com/scholar?q=Vladimir+Bogachev%2C+Vladimir+Aletov%2C+Alexander+Molozhavenko%2C+Denis+Bobkov%2C+Vera+Soboleva%2C+Aibek+Alanov%2C+and+Maxim+Rakhuba%2E+2025%2E+Lora+meets+riemannion%3A+Muon+optimizer+for+parametrization%2Dindependent+lo)
3. Peter Bühlmann and Sara Van De Geer. 2011. Statistics for high-dimensional data: methods, theory and applications. Springer Science & Business Media. [find](https://scholar.google.com/scholar?q=Peter+B%C3%BChlmann+and+Sara+Van+De+Geer%2E+2011%2E+Statistics+for+high%2Ddimensional+data%3A+methods%2C+theory+and+applications%2E+Springer+Science+%26+Business+Media%2E)
4. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). arXiv preprint arXiv:1905.10044.
5. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457.
6. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
7. William de Vazelhes, Hualin Zhang, Huimin Wu, Xiaotong Yuan, and Bin Gu. 2022. Zeroth-order hard-thresholding: gradient error vs. expansivity. Advances in Neural Information Processing Systems, 35:22589–22601. [find](https://scholar.google.com/scholar?q=William+de+Vazelhes%2C+Hualin+Zhang%2C+Huimin+Wu%2C+Xiaotong+Yuan%2C+and+Bin+Gu%2E+2022%2E+Zeroth%2Dorder+hard%2Dthresholding%3A+gradient+error+vs%2E+expansivity%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A225)
8. Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, and 1 others. 2023. [Parameter-efficient fine-tuning of large-scale pre-trained language models](https://md.research.elamur.ai/free/en/papers/ding-2023-parameter-efficient-fine-tuning-large/). Nature Machine Intelligence, 5(3):220–235.
9. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. [The Llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
10. Chongyang Gao, Kezhen Chen, Jinmeng Rao, Ruibo Liu, Baochen Sun, Yawen Zhang, Daiyi Peng, Xiaoyuan Guo, and VS Subrahmanian. 2025. Mola: Moe lora with layer-wise expert allocation. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 5097–5112. [find](https://scholar.google.com/scholar?q=Chongyang+Gao%2C+Kezhen+Chen%2C+Jinmeng+Rao%2C+Ruibo+Liu%2C+Baochen+Sun%2C+Yawen+Zhang%2C+Daiyi+Peng%2C+Xiaoyuan+Guo%2C+and+VS+Subrahmanian%2E+2025%2E+Mola%3A+Moe+lora+with+layer%2Dwise+expert+allocation%2E+In+Findings+of+the+)
11. Chendi Ge, Xin Wang, Zeyang Zhang, Hong Chen, Jiapei Fan, Longtao Huang, Hui Xue, and Wenwu Zhu. 2025. Dynamic mixture of curriculum lora experts for continual multimodal instruction tuning. arXiv preprint arXiv:2506.11672. — [arXiv:2506.11672](https://arxiv.org/abs/2506.11672) [find](https://scholar.google.com/scholar?q=Chendi+Ge%2C+Xin+Wang%2C+Zeyang+Zhang%2C+Hong+Chen%2C+Jiapei+Fan%2C+Longtao+Huang%2C+Hui+Xue%2C+and+Wenwu+Zhu%2E+2025%2E+Dynamic+mixture+of+curriculum+lora+experts+for+continual+multimodal+instruction+tuning%2E+arXiv+pre)
12. Ming Gong, Yingnan Deng, Nia Qi, Yujun Zou, Zhihao Xue, and Yun Zi. 2025. Structure-learnable adapter fine-tuning for parameter-efficient large language models. In IET Conference Proceedings CP944, volume 2025, pages 225–230. IET. [find](https://scholar.google.com/scholar?q=Ming+Gong%2C+Yingnan+Deng%2C+Nia+Qi%2C+Yujun+Zou%2C+Zhihao+Xue%2C+and+Yun+Zi%2E+2025%2E+Structure%2Dlearnable+adapter+fine%2Dtuning+for+parameter%2Defficient+large+language+models%2E+In+IET+Conference+Proceedings+CP944%2C+vo)
13. Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. 2024. [Parameter-efficient fine-tuning for large models: A comprehensive survey](https://md.research.elamur.ai/free/en/papers/han-2024-parameter-efficient-fine-tuning-large/). arXiv preprint arXiv:2403.14608.
14. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. [Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing](https://md.research.elamur.ai/free/en/papers/he-2021-debertav3-improving-deberta-using-electra/). arXiv preprint arXiv:2111.09543.
15. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. [DeBERTa: Decoding-enhanced bert with disentangled attention](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/). arXiv preprint arXiv:2006.03654.
16. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. [Teaching machines to read and comprehend](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/). Advances in neural information processing systems, 28.
17. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). arXiv preprint arXiv:2106.09685.
18. Anna R Karlin. 2018. Alternating minimization, scaling algorithms, and the null-cone problem from invariant theory. In Proceedings of 9th Innovations in Theoretical Computer Science Conference (ITCS 2018), volume 94, page 24. Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik. [find](https://scholar.google.com/scholar?q=Anna+R+Karlin%2E+2018%2E+Alternating+minimization%2C+scaling+algorithms%2C+and+the+null%2Dcone+problem+from+invariant+theory%2E+In+Proceedings+of+9th+Innovations+in+Theoretical+Computer+Science+Conference+%28ITCS+2)
19. Dawid J Kopiczko, Tijmen Blankevoort, and Yuki M Asano. 2023. [VeRA: Vector-based random matrix adaptation](https://md.research.elamur.ai/free/en/papers/kopiczko-2023-vera-vector-based-random-matrix/). arXiv preprint arXiv:2310.11454.
20. Mike Lewis. 2019. [Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). arXiv preprint arXiv:1910.13461.
21. Xingguo Li, Raman Arora, Han Liu, Jarvis Haupt, and Tuo Zhao. 2016. Nonconvex sparse learning via stochastic optimization with progressive variance reduction. arXiv preprint arXiv:1605.02711. — [arXiv:1605.02711](https://arxiv.org/abs/1605.02711) [find](https://scholar.google.com/scholar?q=Xingguo+Li%2C+Raman+Arora%2C+Han+Liu%2C+Jarvis+Haupt%2C+and+Tuo+Zhao%2E+2016%2E+Nonconvex+sparse+learning+via+stochastic+optimization+with+progressive+variance+reduction%2E+arXiv+preprint+arXiv%3A1605%2E02711%2E+%E2%80%94+%5BarXiv)
22. Yang Li, Shaobo Han, and Shihao Ji. 2024. VB-LoRA: Extreme parameter efficient fine-tuning with vector banks. arXiv preprint arXiv:2405.15179. — [arXiv:2405.15179](https://arxiv.org/abs/2405.15179) [find](https://scholar.google.com/scholar?q=Yang+Li%2C+Shaobo+Han%2C+and+Shihao+Ji%2E+2024%2E+VB%2DLoRA%3A+Extreme+parameter+efficient+fine%2Dtuning+with+vector+banks%2E+arXiv+preprint+arXiv%3A2405%2E15179%2E+%E2%80%94+%5BarXiv%3A2405%2E15179%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2405%2E15179%29)
23. Zongqian Li, Yixuan Su, Han Zhou, Zihao Fu, and Nigel Collier. 2025. Flexi-lora: Efficient lora finetuning with input-adaptive dynamic ranks. In ES-FoMo III: 3rd Workshop on Efficient Systems for Foundation Models. [find](https://scholar.google.com/scholar?q=Zongqian+Li%2C+Yixuan+Su%2C+Han+Zhou%2C+Zihao+Fu%2C+and+Nigel+Collier%2E+2025%2E+Flexi%2Dlora%3A+Efficient+lora+finetuning+with+input%2Dadaptive+dynamic+ranks%2E+In+ES%2DFoMo+III%3A+3rd+Workshop+on+Efficient+Systems+for+Foun)
24. Cheng Lin, Lujun Li, Dezhi Li, Jie Zou, Wei Xue, and Yike Guo. 2024. NoRA: Nested low-rank adaptation for efficient fine-tuning large models. arXiv preprint arXiv:2408.10280. — [arXiv:2408.10280](https://arxiv.org/abs/2408.10280) [find](https://scholar.google.com/scholar?q=Cheng+Lin%2C+Lujun+Li%2C+Dezhi+Li%2C+Jie+Zou%2C+Wei+Xue%2C+and+Yike+Guo%2E+2024%2E+NoRA%3A+Nested+low%2Drank+adaptation+for+efficient+fine%2Dtuning+large+models%2E+arXiv+preprint+arXiv%3A2408%2E10280%2E+%E2%80%94+%5BarXiv%3A2408%2E10280%5D%28http)
25. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024a. DoRA: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353. [find](https://scholar.google.com/scholar?q=Shih%2DYang+Liu%2C+Chien%2DYi+Wang%2C+Hongxu+Yin%2C+Pavlo+Molchanov%2C+Yu%2DChiang+Frank+Wang%2C+Kwang%2DTing+Cheng%2C+and+Min%2DHung+Chen%2E+2024a%2E+DoRA%3A+Weight%2Ddecomposed+low%2Drank+adaptation%2E+arXiv+preprint+arXiv%3A2402%2E0935)
26. Zequan Liu, Jiawen Lyn, Wei Zhu, and Xing Tian. 2024b. ALoRA: Allocating low-rank adaptation for fine-tuning large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers… [find](https://scholar.google.com/scholar?q=Zequan+Liu%2C+Jiawen+Lyn%2C+Wei+Zhu%2C+and+Xing+Tian%2E+2024b%2E+ALoRA%3A+Allocating+low%2Drank+adaptation+for+fine%2Dtuning+large+language+models%2E+In+Proceedings+of+the+2024+Conference+of+the+North+American+Chapter+)
27. Yulong Mao, Kaiyu Huang, Changhao Guan, Ganglin Bao, Fengran Mo, and Jinan Xu. 2024. DoRA: Enhancing parameter-efficient fine-tuning with dynamic rank distribution. arXiv preprint arXiv:2405.17357. [find](https://scholar.google.com/scholar?q=Yulong+Mao%2C+Kaiyu+Huang%2C+Changhao+Guan%2C+Ganglin+Bao%2C+Fengran+Mo%2C+and+Jinan+Xu%2E+2024%2E+DoRA%3A+Enhancing+parameter%2Defficient+fine%2Dtuning+with+dynamic+rank+distribution%2E+arXiv+preprint+arXiv%3A2405%2E17357%2E)
28. Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. [Don’t give me the details, just the summary!](https://md.research.elamur.ai/free/en/papers/narayan-2018-don-t-give-me-details/) topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745.
29. Nam Nguyen, Deanna Needell, and Tina Woolf. 2014. Linear convergence of stochastic iterative greedy algorithms with sparse constraints. Preprint, arXiv:1407.0088. — [arXiv:1407.0088](https://arxiv.org/abs/1407.0088) [find](https://scholar.google.com/scholar?q=Nam+Nguyen%2C+Deanna+Needell%2C+and+Tina+Woolf%2E+2014%2E+Linear+convergence+of+stochastic+iterative+greedy+algorithms+with+sparse+constraints%2E+Preprint%2C+arXiv%3A1407%2E0088%2E+%E2%80%94+%5BarXiv%3A1407%2E0088%5D%28https%3A%2F%2Farxiv%2Eorg)
30. Fabian Paischer, Lukas Hauzenberger, Thomas Schmied, Benedikt Alkin, Marc Peter Deisenroth, and Sepp Hochreiter. 2024. Parameter efficient fine-tuning via explained variance adaptation. arXiv preprint arXiv:2410.07170. — [arXiv:2410.07170](https://arxiv.org/abs/2410.07170) [find](https://scholar.google.com/scholar?q=Fabian+Paischer%2C+Lukas+Hauzenberger%2C+Thomas+Schmied%2C+Benedikt+Alkin%2C+Marc+Peter+Deisenroth%2C+and+Sepp+Hochreiter%2E+2024%2E+Parameter+efficient+fine%2Dtuning+via+explained+variance+adaptation%2E+arXiv+preprint)
31. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, and 2 others. 2019.… — [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
32. P Rajpurkar. 2016. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). arXiv preprint arXiv:1606.05250.
33. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. [Know what you don’t know: Unanswerable questions for squad](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/). arXiv preprint arXiv:1806.03822.
34. Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten Rijke, Zhumin Chen, and Jiahuan Pei. 2024. MELoRA: mini-ensemble low-rank adapters for parameter-efficient fine-tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: L… [find](https://scholar.google.com/scholar?q=Pengjie+Ren%2C+Chengshun+Shi%2C+Shiguang+Wu%2C+Mengqi+Zhang%2C+Zhaochun+Ren%2C+Maarten+Rijke%2C+Zhumin+Chen%2C+and+Jiahuan+Pei%2E+2024%2E+MELoRA%3A+mini%2Densemble+low%2Drank+adapters+for+parameter%2Defficient+fine%2Dtuning%2E+In+)
35. Adithya Renduchintala, Tugrul Konuk, and Oleksii Kuchaiev. 2023. Tied-LoRA: Enhacing parameter efficiency of lora with weight tying. arXiv preprint arXiv:2311.09578. — [arXiv:2311.09578](https://arxiv.org/abs/2311.09578) [find](https://scholar.google.com/scholar?q=Adithya+Renduchintala%2C+Tugrul+Konuk%2C+and+Oleksii+Kuchaiev%2E+2023%2E+Tied%2DLoRA%3A+Enhacing+parameter+efficiency+of+lora+with+weight+tying%2E+arXiv+preprint+arXiv%3A2311%2E09578%2E+%E2%80%94+%5BarXiv%3A2311%2E09578%5D%28https%3A%2F%2Farxiv)
36. Adithya Renduchintala, Tugrul Konuk, and Oleksii Kuchaiev. 2024. Tied-lora: Enhancing parameter efficiency of lora with weight tying. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pape… [find](https://scholar.google.com/scholar?q=Adithya+Renduchintala%2C+Tugrul+Konuk%2C+and+Oleksii+Kuchaiev%2E+2024%2E+Tied%2Dlora%3A+Enhancing+parameter+efficiency+of+lora+with+weight+tying%2E+In+Proceedings+of+the+2024+Conference+of+the+North+American+Chapte)
37. Chenxing Wei, Yao Shu, Ying Tiffany He, and Fei Yu. 2025. Flexora: Flexible low-rank adaptation for large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14643– 14682. [find](https://scholar.google.com/scholar?q=Chenxing+Wei%2C+Yao+Shu%2C+Ying+Tiffany+He%2C+and+Fei+Yu%2E+2025%2E+Flexora%3A+Flexible+low%2Drank+adaptation+for+large+language+models%2E+In+Proceedings+of+the+63rd+Annual+Meeting+of+the+Association+for+Computationa)
38. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. [Huggin…](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
39. Diyuan Wu, Ionut-Vlad Modoranu, Mher Safaryan, Denis Kuznedelev, and Dan Alistarh. 2024. The iterative optimal brain surgeon: Faster sparse recovery by leveraging second-order information. arXiv preprint arXiv:2408.17163. — [arXiv:2408.17163](https://arxiv.org/abs/2408.17163) [find](https://scholar.google.com/scholar?q=Diyuan+Wu%2C+Ionut%2DVlad+Modoranu%2C+Mher+Safaryan%2C+Denis+Kuznedelev%2C+and+Dan+Alistarh%2E+2024%2E+The+iterative+optimal+brain+surgeon%3A+Faster+sparse+recovery+by+leveraging+second%2Dorder+information%2E+arXiv+prepr)
40. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
41. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). arXiv preprint arXiv:2309.12284.
42. Xin Yu, Cong Xie, Ziyu Zhao, Tiantian Fan, Lingzhou Xue, and Zhi Zhang. 2025. Prunedlora: Robust gradient-based structured pruning for low-rank adaptation in fine-tuning. arXiv preprint arXiv:2510.00192. — [arXiv:2510.00192](https://arxiv.org/abs/2510.00192) [find](https://scholar.google.com/scholar?q=Xin+Yu%2C+Cong+Xie%2C+Ziyu+Zhao%2C+Tiantian+Fan%2C+Lingzhou+Xue%2C+and+Zhi+Zhang%2E+2025%2E+Prunedlora%3A+Robust+gradient%2Dbased+structured+pruning+for+low%2Drank+adaptation+in+fine%2Dtuning%2E+arXiv+preprint+arXiv%3A2510%2E001)
43. Xiaotong Yuan and Ping Li. 2021. Stability and risk bounds of iterative hard thresholding. In International conference on artificial intelligence and statistics, pages 1702–1710. PMLR. [find](https://scholar.google.com/scholar?q=Xiaotong+Yuan+and+Ping+Li%2E+2021%2E+Stability+and+risk+bounds+of+iterative+hard+thresholding%2E+In+International+conference+on+artificial+intelligence+and+statistics%2C+pages+1702%E2%80%931710%2E+PMLR%2E)
44. Jie Shen and Ping Li. 2018. A tight bound of hard thresholding. Journal of Machine Learning Research, 18(208):1–42. [find](https://scholar.google.com/scholar?q=Jie+Shen+and+Ping+Li%2E+2018%2E+A+tight+bound+of+hard+thresholding%2E+Journal+of+Machine+Learning+Research%2C+18%28208%29%3A1%E2%80%9342%2E)
45. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+arXiv+preprint+arXiv%3A1905%2E07830%2E)
46. Haseeb Ullah Khan Shinwari and Muhammad Usama. 2025. Ard-lora: Dynamic rank allocation for parameter-efficient fine-tuning of foundation models with heterogeneous adaptation needs. IEEE Transactions on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Haseeb+Ullah+Khan+Shinwari+and+Muhammad+Usama%2E+2025%2E+Ard%2Dlora%3A+Dynamic+rank+allocation+for+parameter%2Defficient+fine%2Dtuning+of+foundation+models+with+heterogeneous+adaptation+needs%2E+IEEE+Transactions+o)
47. Dacao Zhang, Kun Zhang, Shimao Chu, Le Wu, Xin Li, and Si Wei. 2025. More: A mixture of low-rank experts for adaptive multi-task learning. arXiv preprint arXiv:2505.22694. [find](https://scholar.google.com/scholar?q=Dacao+Zhang%2C+Kun+Zhang%2C+Shimao+Chu%2C+Le+Wu%2C+Xin+Li%2C+and+Si+Wei%2E+2025%2E+More%3A+A+mixture+of+low%2Drank+experts+for+adaptive+multi%2Dtask+learning%2E+arXiv+preprint+arXiv%3A2505%2E22694%2E)
48. Alex Wang. 2018. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). arXiv preprint arXiv:1804.07461.
49. Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. 2024. MiLoRA: Harnessing minor singular components for parameter-efficient llm fine-tuning. arXiv preprint arXiv:2406.09044. — [arXiv:2406.09044](https://arxiv.org/abs/2406.09044) [find](https://scholar.google.com/scholar?q=Hanqing+Wang%2C+Yixia+Li%2C+Shuo+Wang%2C+Guanhua+Chen%2C+and+Yun+Chen%2E+2024%2E+MiLoRA%3A+Harnessing+minor+singular+components+for+parameter%2Defficient+llm+fine%2Dtuning%2E+arXiv+preprint+arXiv%3A2406%2E09044%2E+%E2%80%94+%5BarXiv%3A240)
50. Fangzhao Zhang and Mert Pilanci. 2024. Riemannian preconditioned lora for fine-tuning foundation models. arXiv preprint arXiv:2402.02347. — [arXiv:2402.02347](https://arxiv.org/abs/2402.02347) [find](https://scholar.google.com/scholar?q=Fangzhao+Zhang+and+Mert+Pilanci%2E+2024%2E+Riemannian+preconditioned+lora+for+fine%2Dtuning+foundation+models%2E+arXiv+preprint+arXiv%3A2402%2E02347%2E+%E2%80%94+%5BarXiv%3A2402%2E02347%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2402%2E02347%29)
51. Feiyu Zhang, Liangzhi Li, Junhao Chen, Zhouqiang Jiang, Bowen Wang, and Yiming Qian. 2023a. Increlora: Incremental parameter allocation method for parameter-efficient fine-tuning. arXiv preprint arXiv:2308.12043. — [arXiv:2308.12043](https://arxiv.org/abs/2308.12043) [find](https://scholar.google.com/scholar?q=Feiyu+Zhang%2C+Liangzhi+Li%2C+Junhao+Chen%2C+Zhouqiang+Jiang%2C+Bowen+Wang%2C+and+Yiming+Qian%2E+2023a%2E+Increlora%3A+Incremental+parameter+allocation+method+for+parameter%2Defficient+fine%2Dtuning%2E+arXiv+preprint+arXiv)
52. Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023b. [AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning](https://md.research.elamur.ai/free/en/papers/zhang-2023-adalora-adaptive-budget-allocation-parameter/). arXiv preprint arXiv:2303.10512. simplifies hyperparameter tuning across different values of r. In…
53. Hongyun Zhou, Xiangyu Lu, Wang Xu, Conghui Zhu, Tiejun Zhao, and Muyun Yang. 2025. [Lora-drop: Efficient lora parameter pruning based on output evaluation](https://md.research.elamur.ai/free/en/papers/zhou-2025-lora-drop-efficient-lora-parameter/). In Proceedings of the 31st International Conference on Computational Linguistics, pages 5530– 5543.
54. LoRA dropout. To regularize the low-rank path and mitigate overfitting, LoRA typically applies dropout to the intermediate low-rank representation. Concretely, one first computes Pan Zhou, Xiaotong Yuan, and Jiashi Feng. 2018. Efficient stochastic gradient hard thresholding. Advances in Neural Infor… [find](https://scholar.google.com/scholar?q=LoRA+dropout%2E+To+regularize+the+low%2Drank+path+and+mitigate+overfitting%2C+LoRA+typically+applies+dropout+to+the+intermediate+low%2Drank+representation%2E+Concretely%2C+one+first+computes+Pan+Zhou%2C+Xiaotong+Yu)

*Updated 1 November 2026*