---
title: "G-LoRA: Global-Local Decoupled Low-Rank Adaptation (2026)"
description: "LoRA and adaptive variants such as AdaLoRA give a single rank to an entire weight update matrix ∆W. 2.7 accuracy. 54 references."
authors: ["Jiahao Xiong", "Yihong Huang", "Yihe Liu", "Xianming Hu", "Hongbo Zhao", "Kai Zhang"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1005/"
doi: "10.18653/v1/2026.findings-acl.1005"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xiong-2026-g-lora-global-local-decoupled/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xiong-2026-g-lora-global-local-decoupled/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/34/) / G-LoRA: Global-Local Decoupled Low-Rank Adaptation

# G-LoRA: Global-Local Decoupled Low-Rank Adaptation

**Authors:** [Jiahao Xiong](https://md.research.elamur.ai/free/en/authors/jiahao-xiong/), [Yihong Huang](https://md.research.elamur.ai/free/en/authors/yihong-huang/), [Yihe Liu](https://md.research.elamur.ai/free/en/authors/yihe-liu/), [Xianming Hu](https://md.research.elamur.ai/free/en/authors/xianming-hu/), [Hongbo Zhao](https://md.research.elamur.ai/free/en/authors/hongbo-zhao/) and [Kai Zhang](https://md.research.elamur.ai/free/en/authors/kai-zhang/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1005/) · [PDF](https://aclanthology.org/2026.findings-acl.1005.pdf) · [DOI 10.18653/v1/2026.findings-acl.1005](https://doi.org/10.18653/v1/2026.findings-acl.1005) · [OpenAlex](https://openalex.org/W7166886330)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
LoRA and adaptive variants such as AdaLoRA give a single rank to an entire weight update matrix ∆W. This ignores the fact that parameter importance varies inside one matrix. Sparse fine-tuning does capture that variation, but it needs a memory-heavy gradient warm-up. G-LoRA addresses the gap by splitting each update into two parts. A low-rank G-Adapter spans the whole matrix. An L-Adapter gets a larger share of the rank and acts on a sub-block where the most influential parameters have been grouped. The grouping comes from permuting rows and columns by a first-order Taylor importance estimate. During training, this local region is repeatedly shrunk and re-initialized with randomized SVD, which concentrates capacity further. The authors derive upper and lower bounds on the attainable rank, showing that it exceeds LoRA's under an equal parameter budget. Across benchmarks testing commonsense and mathematical reasoning as well as code writing, G-LoRA beats LoRA, DoRA, PiSSA, RaSA, RoSA and GraLoRA by as much as 2.7 accuracy points. This makes it a practical way to extract more from a fixed fine-tuning budget at small extra cost.

## Key points
- With the L-Adapter restricted to 1/16 of the matrix, theory gives roughly 1.5–2.5× higher effective rank than LoRA at the same budget. The gain grows as the local region gets smaller.
- On LLaMA2-7B at rank 8, the Entropy-Based Effective Rank (EBER) of the updates is 137.23 for G-LoRA. LoRA reaches 5.57 and GraLoRA 13.14.
- Training time rises only about 8% over LoRA, mostly from the reordering steps.
- Even with re-initialization switched off, G-LoRA scores 70.4% versus LoRA's 70.0%. Re-initializing every 2–4 epochs works best.
- Initializing the G-Adapter with small singular values and the L-Adapter with large ones gave the top average of 71.8%. Limitations: experiments cover English text only, and the global/local rank ratio is set by hand.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [East China Normal University](https://md.research.elamur.ai/free/en/universities/east-china-normal-university/) — University, Shanghai, China · 69 works · 2,490 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 297 works

## Cite

Cite as: Jiahao Xiong, Yihong Huang, Yihe Liu, Xianming Hu, Hongbo Zhao and Kai Zhang. 2026. G-LoRA: Global-Local Decoupled Low-Rank Adaptation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xiong2026g,
  title = {{G-LoRA: Global-Local Decoupled Low-Rank Adaptation}},
  author = {Jiahao Xiong, Yihong Huang, Yihe Liu, Xianming Hu, Hongbo Zhao and Kai Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1005},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1005}
}
```

- [← previous G-HiRel: Enhancing the Adaption to Knowledge Updating for Large Language Model Reasoning](https://md.research.elamur.ai/free/en/papers/pan-2026-g-hirel-enhancing-adaption-knowledge/)
- [next → GALA: Geometric Data Selection with Strategic Prospecting for Large Language Model Self-training](https://md.research.elamur.ai/free/en/papers/xie-2026-gala-geometric-data-selection-strategic/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/34/)

## References

54 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). Preprint, arXiv:2303.08774.
2. Alan Ansell, Ivan Vulić, Hannah Sterz, Anna Korhonen, and Edoardo M Ponti. 2024. Scaling sparse fine-tuning to large language models. arXiv preprint arXiv:2401.16405. — [arXiv:2401.16405](https://arxiv.org/abs/2401.16405) [find](https://scholar.google.com/scholar?q=Alan+Ansell%2C+Ivan+Vulic%CC%81%2C+Hannah+Sterz%2C+Anna+Korhonen%2C+and+Edoardo+M+Ponti%2E+2024%2E+Scaling+sparse+fine%2Dtuning+to+large+language+models%2E+arXiv+preprint+arXiv%3A2401%2E16405%2E+%E2%80%94+%5BarXiv%3A2401%2E16405%5D%28https%3A%2F%2Farx)
3. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). Preprint, arXiv:2108.07732.
4. Elad Ben-Zaken, Shauli Ravfogel, and Yoav Goldberg. 2022. [Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models](https://md.research.elamur.ai/free/en/papers/ben-zaken-2021-bitfit-simple-parameter-efficient-fine/). In 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022, pages 1–9. Association for Computational Linguistics (ACL).
5. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Thirty-Fourth AAAI Conference on Artificial Intelligence.
6. Sahil Chaudhary. 2023. Code alpaca: An instruction-following llama model for code generation. [https://github.com/sahil280114/codealpaca](https://github.com/sahil280114/codealpaca). [↗](https://github.com/sahil280114/codealpaca)
7. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/).
8. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [BoolQ: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:…
9. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv:1803.05457v1.
10. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
11. Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. 2023. Sparse low-rank adaptation of pre-trained language models. In The 2023 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Ning+Ding%2C+Xingtai+Lv%2C+Qiaosen+Wang%2C+Yulin+Chen%2C+Bowen+Zhou%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2023%2E+Sparse+low%2Drank+adaptation+of+pre%2Dtrained+language+models%2E+In+The+2023+Conference+on+Empirical+Methods+)
12. Haonan Dong, Wenhao Zhu, Guojie Song, and Liang Wang. 2025. Aurora: Breaking low-rank bottleneck of lora with nonlinear mapping. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Haonan+Dong%2C+Wenhao+Zhu%2C+Guojie+Song%2C+and+Liang+Wang%2E+2025%2E+Aurora%3A+Breaking+low%2Drank+bottleneck+of+lora+with+nonlinear+mapping%2E+In+Advances+in+Neural+Information+Processing+Systems%2E)
13. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). Preprint, arXiv:2407.21783.
14. Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. 2011. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 53(2):217– 288. [find](https://scholar.google.com/scholar?q=Nathan+Halko%2C+Per%2DGunnar+Martinsson%2C+and+Joel+A+Tropp%2E+2011%2E+Finding+structure+with+randomness%3A+Probabilistic+algorithms+for+constructing+approximate+matrix+decompositions%2E+SIAM+review%2C+53%282%29%3A217%E2%80%93+288)
15. Marawan Gamal Abdel Hameed, Aristides Milios, Siva Reddy, and Guillaume Rabusseau. 2024. Rosa: Random subspace adaptation for efficient fine-tuning. Preprint, arXiv:2407.07802. — [arXiv:2407.07802](https://arxiv.org/abs/2407.07802) [find](https://scholar.google.com/scholar?q=Marawan+Gamal+Abdel+Hameed%2C+Aristides+Milios%2C+Siva+Reddy%2C+and+Guillaume+Rabusseau%2E+2024%2E+Rosa%3A+Random+subspace+adaptation+for+efficient+fine%2Dtuning%2E+Preprint%2C+arXiv%3A2407%2E07802%2E+%E2%80%94+%5BarXiv%3A2407%2E07802%5D%28ht)
16. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1026–1034. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+2015%2E+Delving+deep+into+rectifiers%3A+Surpassing+human%2Dlevel+performance+on+imagenet+classification%2E+In+Proceedings+of+the+IEEE+International+Confe)
17. Zhiwei He, Zhaopeng Tu, Xing Wang, Xingyu Chen, Zhijie Wang, Jiahao Xu, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, and Rui Wang. 2025. RaSA: Rank-sharing low-rank adaptation. In Proceedings of the 13th International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Zhiwei+He%2C+Zhaopeng+Tu%2C+Xing+Wang%2C+Xingyu+Chen%2C+Zhijie+Wang%2C+Jiahao+Xu%2C+Tian+Liang%2C+Wenxiang+Jiao%2C+Zhuosheng+Zhang%2C+and+Rui+Wang%2E+2025%2E+RaSA%3A+Rank%2Dsharing+low%2Drank+adaptation%2E+In+Proceedings+of+the+13)
18. Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. [Learning to solve arithmetic word problems with verb categorization](https://md.research.elamur.ai/free/en/papers/hosseini-2014-learning-solve-arithmetic-word-problems/). In EMNLP, pages 523–533.
19. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In Proceedings of the 36th International Conference on Machine Learning (ICML), pages 2790–2799. PMLR. [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+De+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+nlp%2E+In+)
20. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In Proceedings of the International Conference on Learning Representations (ICLR).
21. Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. 2023. [Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models](https://md.research.elamur.ai/free/en/papers/hu-2023-llm-adapters-adapter-family-parameter/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (…
22. Qiushi Huang, Tom Ko, Zhan Zhuang, Lilian Tang, and Yu Zhang. 2025a. Hira: Parameter-efficient hadamard high-rank adaptation for large language models. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Qiushi+Huang%2C+Tom+Ko%2C+Zhan+Zhuang%2C+Lilian+Tang%2C+and+Yu+Zhang%2E+2025a%2E+Hira%3A+Parameter%2Defficient+hadamard+high%2Drank+adaptation+for+large+language+models%2E+In+The+Thirteenth+International+Conference+on+Le)
23. Weizhong Huang, Yuxin Zhang, Xiawu Zheng, Liuyang, Jing Lin, Yiwu Yao, and Rongrong Ji. 2025b. Dynamic low-rank sparse adaptation for large language models. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Weizhong+Huang%2C+Yuxin+Zhang%2C+Xiawu+Zheng%2C+Liuyang%2C+Jing+Lin%2C+Yiwu+Yao%2C+and+Rongrong+Ji%2E+2025b%2E+Dynamic+low%2Drank+sparse+adaptation+for+large+language+models%2E+In+The+Thirteenth+International+Conference+)
24. Yeonjoon Jung, Daehyun Ahn, Hyungjun Kim, Taesu Kim, and Eunhyeok Park. 2025. Gralora: Granular low-rank adaptation for parameter-efficient fine-tuning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Yeonjoon+Jung%2C+Daehyun+Ahn%2C+Hyungjun+Kim%2C+Taesu+Kim%2C+and+Eunhyeok+Park%2E+2025%2E+Gralora%3A+Granular+low%2Drank+adaptation+for+parameter%2Defficient+fine%2Dtuning%2E+In+The+Thirty%2Dninth+Annual+Conference+on+Neural)
25. Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3:585–597. [find](https://scholar.google.com/scholar?q=Rik+Koncel%2DKedziorski%2C+Hannaneh+Hajishirzi%2C+Ashish+Sabharwal%2C+Oren+Etzioni%2C+and+Siena+Dumas+Ang%2E+2015%2E+Parsing+algebraic+word+problems+into+equations%2E+Transactions+of+the+Association+for+Computational)
26. Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. MAWPS: A math word problem repository. In Proceedings of NAACL, pages 1152–1157. [find](https://scholar.google.com/scholar?q=Rik+Koncel%2DKedziorski%2C+Subhro+Roy%2C+Aida+Amini%2C+Nate+Kushman%2C+and+Hannaneh+Hajishirzi%2E+2016%2E+MAWPS%3A+A+math+word+problem+repository%2E+In+Proceedings+of+NAACL%2C+pages+1152%E2%80%931157%2E)
27. Wen Lai, Alexander Fraser, and Ivan Titov. 2025. Joint localization and activation editing for low-resource fine-tuning. In Forty-second International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Wen+Lai%2C+Alexander+Fraser%2C+and+Ivan+Titov%2E+2025%2E+Joint+localization+and+activation+editing+for+low%2Dresource+fine%2Dtuning%2E+In+Forty%2Dsecond+International+Conference+on+Machine+Learning%2E)
28. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. [The power of scale for parameter-efficient prompt tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3045–3059.
29. Shiwei Li, Xiandi Luo, Haozhao Wang, Xing Tang, Ziqiang Cui, Dugang Liu, Yuhua Li, Xiuqiang He, and Ruixuan Li. 2025a. Beyond higher rank: Token-wise input-output projections for efficient low-rank adaptation. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Shiwei+Li%2C+Xiandi+Luo%2C+Haozhao+Wang%2C+Xing+Tang%2C+Ziqiang+Cui%2C+Dugang+Liu%2C+Yuhua+Li%2C+Xiuqiang+He%2C+and+Ruixuan+Li%2E+2025a%2E+Beyond+higher+rank%3A+Token%2Dwise+input%2Doutput+projections+for+efficient+low%2Drank+ad)
30. Shiwei Li, Xiandi Luo, Haozhao Wang, Xing Tang, Ziqiang Cui, Dugang Liu, Yuhua Li, Xiuqiang He, and Ruixuan Li. 2025b. Bora: Towards more expressive low-rank adaptation with block diversity. Preprint, arXiv:2508.06953. — [arXiv:2508.06953](https://arxiv.org/abs/2508.06953) [find](https://scholar.google.com/scholar?q=Shiwei+Li%2C+Xiandi+Luo%2C+Haozhao+Wang%2C+Xing+Tang%2C+Ziqiang+Cui%2C+Dugang+Liu%2C+Yuhua+Li%2C+Xiuqiang+He%2C+and+Ruixuan+Li%2E+2025b%2E+Bora%3A+Towards+more+expressive+low%2Drank+adaptation+with+block+diversity%2E+Preprint%2C)
31. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL), pages 4582–4597. [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL%29%2C+page)
32. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158–167. [find](https://scholar.google.com/scholar?q=Wang+Ling%2C+Dani+Yogatama%2C+Chris+Dyer%2C+and+Phil+Blunsom%2E+2017%2E+Program+induction+by+rationale+generation%3A+Learning+to+solve+and+explain+algebraic+word+problems%2E+In+Proceedings+of+the+55th+Annual+Meetin)
33. Vijay Chandra Lingam, Atula Neerkaje, Aditya Vavre, Aneesh Shetty, Gautham Krishna Gudur, Joydeep Ghosh, Eunsol Choi, Alex Dimakis, Aleksandar Bojchevski, and Sujay Sanghavi. 2024. Svft: Parameter-efficient fine-tuning with singular vectors. volume 37, pages 41425–41446. [find](https://scholar.google.com/scholar?q=Vijay+Chandra+Lingam%2C+Atula+Neerkaje%2C+Aditya+Vavre%2C+Aneesh+Shetty%2C+Gautham+Krishna+Gudur%2C+Joydeep+Ghosh%2C+Eunsol+Choi%2C+Alex+Dimakis%2C+Aleksandar+Bojchevski%2C+and+Sujay+Sanghavi%2E+2024%2E+Svft%3A+Parameter%2Deff)
34. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024a. Dora: Weight-decomposed low-rank adaptation. In Proceedings of the 41st International Conference on Machine Learning (ICML). [find](https://scholar.google.com/scholar?q=Shih%2DYang+Liu%2C+Chien%2DYi+Wang%2C+Hongxu+Yin%2C+Pavlo+Molchanov%2C+Yu%2DChiang+Frank+Wang%2C+Kwang%2DTing+Cheng%2C+and+Min%2DHung+Chen%2E+2024a%2E+Dora%3A+Weight%2Ddecomposed+low%2Drank+adaptation%2E+In+Proceedings+of+the+41st+Int)
35. Zequan Liu, Yi Zhao, Ming Tan, Wei Zhu, and Aaron Xuxiang Tian. 2024b. Para: Parameter-efficient fine-tuning with prompt-aware representation adjustment. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 728–737. [find](https://scholar.google.com/scholar?q=Zequan+Liu%2C+Yi+Zhao%2C+Ming+Tan%2C+Wei+Zhu%2C+and+Aaron+Xuxiang+Tian%2E+2024b%2E+Para%3A+Parameter%2Defficient+fine%2Dtuning+with+prompt%2Daware+representation+adjustment%2E+In+Proceedings+of+the+2024+Conference+on+Empir)
36. Zihang Liu, Tianyu Pang, Oleg Balabanov, Chaoqun Yang, Tianjin Huang, Lu Yin, Yaoqing Yang, and Shiwei Liu. 2025. LIFT the veil for the truth: Principal weights emerge after rank reduction for reasoning-focused supervised fine-tuning. In Forty-second International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Zihang+Liu%2C+Tianyu+Pang%2C+Oleg+Balabanov%2C+Chaoqun+Yang%2C+Tianjin+Huang%2C+Lu+Yin%2C+Yaoqing+Yang%2C+and+Shiwei+Liu%2E+2025%2E+LIFT+the+veil+for+the+truth%3A+Principal+weights+emerge+after+rank+reduction+for+reasoni)
37. Fanxu Meng, Zhaohui Wang, and Muhan Zhang. 2024. [Pissa: Principal singular values and singular vectors adaptation of large language models](https://md.research.elamur.ai/free/en/papers/meng-2024-pissa-principal-singular-values-singular/). In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pages 121038–121072.
38. Daiye Miao, Yufang Liu, Jie Wang, Changzhi Sun, Yunke Zhang, Demei Yan, Shaokang Dong, Qi Zhang, and Yuanbin Wu. 2025. TASO: Task-aligned sparse optimization for parameter-efficient model adaptation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 227… [find](https://scholar.google.com/scholar?q=Daiye+Miao%2C+Yufang+Liu%2C+Jie+Wang%2C+Changzhi+Sun%2C+Yunke+Zhang%2C+Demei+Yan%2C+Shaokang+Dong%2C+Qi+Zhang%2C+and+Yuanbin+Wu%2E+2025%2E+TASO%3A+Task%2Daligned+sparse+optimization+for+parameter%2Defficient+model+adaptation%2E+)
39. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2381–2391. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2018%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+In+Proceedings+of+the+2018+conference+on+)
40. P Molchanov, S Tyree, T Karras, T Aila, and J Kautz. 2019. Pruning convolutional neural networks for resource efficient inference. In 5th International Conference on Learning Representations, ICLR 2017-Conference Track Proceedings. [find](https://scholar.google.com/scholar?q=P+Molchanov%2C+S+Tyree%2C+T+Karras%2C+T+Aila%2C+and+J+Kautz%2E+2019%2E+Pruning+convolutional+neural+networks+for+resource+efficient+inference%2E+In+5th+International+Conference+on+Learning+Representations%2C+ICLR+201)
41. Lin Mu, Xiaoyu Wang, Li Ni, Yang Li, Zhize Wu, Peiquan Jin, and Yiwen Zhang. 2025. Denselora: Dense low-rank adaptation of large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pages 10198– 10211. [find](https://scholar.google.com/scholar?q=Lin+Mu%2C+Xiaoyu+Wang%2C+Li+Ni%2C+Yang+Li%2C+Zhize+Wu%2C+Peiquan+Jin%2C+and+Yiwen+Zhang%2E+2025%2E+Denselora%3A+Dense+low%2Drank+adaptation+of+large+language+models%2E+In+Proceedings+of+the+63rd+Annual+Meeting+of+the+Assoc)
42. Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. 2024. Lisa: Layer-wise importance sampling for memory-efficient large language model fine-tuning. Advances in Neural Information Processing Systems, 37:57018–57049. [find](https://scholar.google.com/scholar?q=Rui+Pan%2C+Xiang+Liu%2C+Shizhe+Diao%2C+Renjie+Pi%2C+Jipeng+Zhang%2C+Chi+Han%2C+and+Tong+Zhang%2E+2024%2E+Lisa%3A+Layer%2Dwise+importance+sampling+for+memory%2Defficient+large+language+model+fine%2Dtuning%2E+Advances+in+Neural+)
43. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. [Are NLP models really able to solve simple math word problems?](https://md.research.elamur.ai/free/en/papers/patel-2021-are-nlp-models-really-able/) In Proceedings of NAACL, pages 2080–2094.
44. Olivier Roy and Martin Vetterli. 2007. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pages 606–610. IEEE. [find](https://scholar.google.com/scholar?q=Olivier+Roy+and+Martin+Vetterli%2E+2007%2E+The+effective+rank%3A+A+measure+of+effective+dimensionality%2E+In+2007+15th+European+signal+processing+conference%2C+pages+606%E2%80%93610%2E+IEEE%2E)
45. Subhro Roy and Dan Roth. 2016. [Solving general arithmetic word problems](https://md.research.elamur.ai/free/en/papers/roy-2015-solving-general-arithmetic-word-problems/). arXiv preprint arXiv:1608.01413.
46. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2021%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Communications+of+the+ACM%2C+64%289%29%3A99%E2%80%93106%2E)
47. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. [Socialiqa: Commonsense reasoning about social interactions](https://md.research.elamur.ai/free/en/papers/sap-2019-socialiqa-commonsense-reasoning-about-social/). arXiv preprint arXiv:1904.09728.
48. Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. 2025. Milora: Harnessing minor singular components for parameter-efficient llm fine-tuning. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Tec… [find](https://scholar.google.com/scholar?q=Hanqing+Wang%2C+Yixia+Li%2C+Shuo+Wang%2C+Guanhua+Chen%2C+and+Yun+Chen%2E+2025%2E+Milora%3A+Harnessing+minor+singular+components+for+parameter%2Defficient+llm+fine%2Dtuning%2E+In+Proceedings+of+the+2025+Conference+of+the+)
49. Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. 2023. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. Preprint, arXiv:2312.12148. [find](https://scholar.google.com/scholar?q=Lingling+Xu%2C+Haoran+Xie%2C+Si%2DZhao+Joe+Qin%2C+Xiaohui+Tao%2C+and+Fu+Lee+Wang%2E+2023%2E+Parameter%2Defficient+fine%2Dtuning+methods+for+pretrained+language+models%3A+A+critical+review+and+assessment%2E+Preprint%2C+arXiv%3A)
50. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
51. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Com)
52. Jinyang Zhang, Yue Fang, Hongxin Ding, Weibin Liao, Muyang Ye, Xu Chu, Junfeng Zhao, and Yasha Wang. 2025. Adept: Continual pretraining via adaptive expansion and dynamic decoupled tuning. Preprint, arXiv:2510.10071. — [arXiv:2510.10071](https://arxiv.org/abs/2510.10071) [find](https://scholar.google.com/scholar?q=Jinyang+Zhang%2C+Yue+Fang%2C+Hongxin+Ding%2C+Weibin+Liao%2C+Muyang+Ye%2C+Xu+Chu%2C+Junfeng+Zhao%2C+and+Yasha+Wang%2E+2025%2E+Adept%3A+Continual+pretraining+via+adaptive+expansion+and+dynamic+decoupled+tuning%2E+Preprint%2C+a)
53. Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. Adaptive budget allocation for parameter-efficient fine-tuning. In Proceedings of the 11th International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Qingru+Zhang%2C+Minshuo+Chen%2C+Alexander+Bukharin%2C+Pengcheng+He%2C+Yu+Cheng%2C+Weizhu+Chen%2C+and+Tuo+Zhao%2E+2023%2E+Adaptive+budget+allocation+for+parameter%2Defficient+fine%2Dtuning%2E+In+Proceedings+of+the+11th+Inte)
54. Heming Zou, Yunliang Zang, Wutong Xu, Yao Zhu, and Xiangyang Ji. 2025. FlyloRA: Boosting task decoupling and parameter efficiency via implicit rank-wise mixture-of-experts. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Heming+Zou%2C+Yunliang+Zang%2C+Wutong+Xu%2C+Yao+Zhu%2C+and+Xiangyang+Ji%2E+2025%2E+FlyloRA%3A+Boosting+task+decoupling+and+parameter+efficiency+via+implicit+rank%2Dwise+mixture%2Dof%2Dexperts%2E+In+The+Thirty%2Dninth+Annual+)

*Updated 2 November 2026*