---
title: "Localized Low-Rank Adaptation within Clustered Parameter"
description: "They propose CPS-LoRA. 49 references. Most LoRA-style methods learn a single pair of low-rank factors for the whole weight update ∆W."
authors: ["Jiahao Xiong", "Yihe Liu", "Xianming Hu", "Hongbo Zhao", "Nuoyi Chen", "Jie Zhang", "Kai Zhang"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1223/"
doi: "10.18653/v1/2026.acl-long.1223"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xiong-2026-localized-low-rank-adaptation-within/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xiong-2026-localized-low-rank-adaptation-within/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/40/) / Localized Low-Rank Adaptation within Clustered Parameter Subspaces

# Localized Low-Rank Adaptation within Clustered Parameter Subspaces

**Authors:** [Jiahao Xiong](https://md.research.elamur.ai/free/en/authors/jiahao-xiong/), [Yihe Liu](https://md.research.elamur.ai/free/en/authors/yihe-liu/), [Xianming Hu](https://md.research.elamur.ai/free/en/authors/xianming-hu/), [Hongbo Zhao](https://md.research.elamur.ai/free/en/authors/hongbo-zhao/), [Nuoyi Chen](https://md.research.elamur.ai/free/en/authors/nuoyi-chen/), [Jie Zhang](https://md.research.elamur.ai/free/en/authors/jie-zhang/) and [Kai Zhang](https://md.research.elamur.ai/free/en/authors/kai-zhang/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1223/) · [PDF](https://aclanthology.org/2026.acl-long.1223.pdf) · [DOI 10.18653/v1/2026.acl-long.1223](https://doi.org/10.18653/v1/2026.acl-long.1223) · [OpenAlex](https://openalex.org/W7166796126)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Most LoRA-style methods learn a single pair of low-rank factors for the whole weight update ∆W. The authors examined updates in LLaMA2-7B and Qwen2.5-1.5B, from both full fine-tuning and LoRA. After bi-clustering rows and columns, these updates often break into distinct blocks. That suggests one global factorization fits local structure poorly.

They propose CPS-LoRA. It sorts the rows and columns of ∆W into internally consistent groups that differ strongly from one another, then fits a separate low-rank update in each block. The grouping is refreshed between epochs, and the learned updates are merged into the base weights.

A theoretical argument shows the benefit when blocks have orthogonal bases: at an equal parameter budget, the local scheme reaches a higher rank than the global one. Experiments cover 17 subtasks in commonsense reasoning, math reasoning and code generation, on models from 1.5B to 13B parameters. CPS-LoRA beats LoRA and its variants by as much as 3.0% absolute accuracy. Training cost is close to LoRA's, and inference cost is unchanged.

## Key points
- **Theory:** with four equal orthogonal blocks and a budget of 4nr parameters, local adaptation reaches rank 4r, while global LoRA reaches only 2r.
- **Rank efficiency:** on Math10K with LLaMA2-7B, CPS-LoRA at rank 8 outperforms both LoRA and GraLoRA (random block partitioning) at rank 16.
- **Partitioning:** partitioning by cosine similarity scores 70.3%, matching balanced spectral bi-clustering, so it is used throughout. GraLoRA with per-epoch merging still trails by about 1.7%, which points to structured partitioning as the source of the gains.
- **Block count:** 2×2 blocks work best (70.3% vs 69.8% for 4×4 and 69.7% for 8×8). The average angle between subspaces falls from 84.9° to 78.3° as blocks get finer.
- **Cost and budget:** training on Math10K takes 77 min, against 71 min for LoRA and 74 min for GraLoRA. With a single training epoch, CPS-LoRA averages 67.1%, compared with 65.7% for LoRA and 64.9% for GraLoRA.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 202 works · 22,509 citations
- [East China Normal University](https://md.research.elamur.ai/free/en/universities/east-china-normal-university/) — University, Shanghai, China · 69 works · 2,490 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 297 works

## Cite

Cite as: Jiahao Xiong, Yihe Liu, Xianming Hu, Hongbo Zhao, Nuoyi Chen, Jie Zhang and Kai Zhang. 2026. Localized Low-Rank Adaptation within Clustered Parameter Subspaces. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xiong2026localized,
  title = {{Localized Low-Rank Adaptation within Clustered Parameter Subspaces}},
  author = {Jiahao Xiong, Yihe Liu, Xianming Hu, Hongbo Zhao, Nuoyi Chen, Jie Zhang and Kai Zhang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1223},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1223}
}
```

- [← previous LLMSurgeon: Diagnosing Data Mixture of Large Language Models](https://md.research.elamur.ai/free/en/papers/luo-2026-llmsurgeon-diagnosing-data-mixture-large/)
- [next → Locate and Explain: Joint Multimodal Emotion Cause Extraction and Summarization in Conversation](https://md.research.elamur.ai/free/en/papers/wan-2026-locate-explain-joint-multimodal-emotion/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/40/)

## References

49 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). Preprint, arXiv:2303.08774.
2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). Preprint, arXiv:2108.07732.
3. Elad Ben-Zaken, Shauli Ravfogel, and Yoav Goldberg. 2022. [Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models](https://md.research.elamur.ai/free/en/papers/ben-zaken-2021-bitfit-simple-parameter-efficient-fine/). In 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022, pages 1–9. Association for Computational Linguistics (ACL).
4. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Thirty-Fourth AAAI Conference on Artificial Intelligence.
5. Sahil Chaudhary. 2023. Code alpaca: An instruction-following llama model for code generation. [https://github.com/sahil280114/codealpaca](https://github.com/sahil280114/codealpaca). [↗](https://github.com/sahil280114/codealpaca)
6. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/).
7. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [BoolQ: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:…
8. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv:1803.05457v1.
9. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
10. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). Preprint, arXiv:2407.21783.
11. Marawan Gamal Abdel Hameed, Aristides Milios, Siva Reddy, and Guillaume Rabusseau. 2024. Rosa: Random subspace adaptation for efficient fine-tuning. Preprint, arXiv:2407.07802. — [arXiv:2407.07802](https://arxiv.org/abs/2407.07802) [find](https://scholar.google.com/scholar?q=Marawan+Gamal+Abdel+Hameed%2C+Aristides+Milios%2C+Siva+Reddy%2C+and+Guillaume+Rabusseau%2E+2024%2E+Rosa%3A+Random+subspace+adaptation+for+efficient+fine%2Dtuning%2E+Preprint%2C+arXiv%3A2407%2E07802%2E+%E2%80%94+%5BarXiv%3A2407%2E07802%5D%28ht)
12. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1026–1034. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+2015%2E+Delving+deep+into+rectifiers%3A+Surpassing+human%2Dlevel+performance+on+imagenet+classification%2E+In+Proceedings+of+the+IEEE+International+Confe)
13. Zhiwei He, Zhaopeng Tu, Xing Wang, Xingyu Chen, Zhijie Wang, Jiahao Xu, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, and Rui Wang. 2025. RaSA: Rank-sharing low-rank adaptation. In Proceedings of the 13th International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Zhiwei+He%2C+Zhaopeng+Tu%2C+Xing+Wang%2C+Xingyu+Chen%2C+Zhijie+Wang%2C+Jiahao+Xu%2C+Tian+Liang%2C+Wenxiang+Jiao%2C+Zhuosheng+Zhang%2C+and+Rui+Wang%2E+2025%2E+RaSA%3A+Rank%2Dsharing+low%2Drank+adaptation%2E+In+Proceedings+of+the+13)
14. Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. [Learning to solve arithmetic word problems with verb categorization](https://md.research.elamur.ai/free/en/papers/hosseini-2014-learning-solve-arithmetic-word-problems/). In EMNLP, pages 523–533.
15. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In Proceedings of the 36th International Conference on Machine Learning (ICML), pages 2790–2799. PMLR. [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+De+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+nlp%2E+In+)
16. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In Proceedings of the International Conference on Learning Representations (ICLR).
17. Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. 2023. [Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models](https://md.research.elamur.ai/free/en/papers/hu-2023-llm-adapters-adapter-family-parameter/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (…
18. Qiushi Huang, Tom Ko, Zhan Zhuang, Lilian Tang, and Yu Zhang. 2025. Hira: Parameter-efficient hadamard high-rank adaptation for large language models. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Qiushi+Huang%2C+Tom+Ko%2C+Zhan+Zhuang%2C+Lilian+Tang%2C+and+Yu+Zhang%2E+2025%2E+Hira%3A+Parameter%2Defficient+hadamard+high%2Drank+adaptation+for+large+language+models%2E+In+The+Thirteenth+International+Conference+on+Lea)
19. Yeonjoon Jung, Daehyun Ahn, Hyungjun Kim, Taesu Kim, and Eunhyeok Park. 2025. Gralora: Granular low-rank adaptation for parameter-efficient fine-tuning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Yeonjoon+Jung%2C+Daehyun+Ahn%2C+Hyungjun+Kim%2C+Taesu+Kim%2C+and+Eunhyeok+Park%2E+2025%2E+Gralora%3A+Granular+low%2Drank+adaptation+for+parameter%2Defficient+fine%2Dtuning%2E+In+The+Thirty%2Dninth+Annual+Conference+on+Neural)
20. Yuval Kluger, Ronen Basri, Joseph T Chang, and Mark Gerstein. 2003. Spectral biclustering of microarray data: coclustering genes and conditions. Genome Research, 13(4):703–716. [find](https://scholar.google.com/scholar?q=Yuval+Kluger%2C+Ronen+Basri%2C+Joseph+T+Chang%2C+and+Mark+Gerstein%2E+2003%2E+Spectral+biclustering+of+microarray+data%3A+coclustering+genes+and+conditions%2E+Genome+Research%2C+13%284%29%3A703%E2%80%93716%2E)
21. Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3:585–597. [find](https://scholar.google.com/scholar?q=Rik+Koncel%2DKedziorski%2C+Hannaneh+Hajishirzi%2C+Ashish+Sabharwal%2C+Oren+Etzioni%2C+and+Siena+Dumas+Ang%2E+2015%2E+Parsing+algebraic+word+problems+into+equations%2E+Transactions+of+the+Association+for+Computational)
22. Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. MAWPS: A math word problem repository. In Proceedings of NAACL, pages 1152–1157. [find](https://scholar.google.com/scholar?q=Rik+Koncel%2DKedziorski%2C+Subhro+Roy%2C+Aida+Amini%2C+Nate+Kushman%2C+and+Hannaneh+Hajishirzi%2E+2016%2E+MAWPS%3A+A+math+word+problem+repository%2E+In+Proceedings+of+NAACL%2C+pages+1152%E2%80%931157%2E)
23. Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. 2024. [VeRA: Vector-based random matrix adaptation](https://md.research.elamur.ai/free/en/papers/kopiczko-2023-vera-vector-based-random-matrix/). In Proceedings of the 12th International Conference on Learning Representations (ICLR).
24. Md Kowsher, Tara Esmaeilbeig, Chun-Nam Yu, Chen Chen, Mojtaba Soltanalian, and Niloofar Yousefi. 2025. [Rocoft: Efficient finetuning of large language models with row-column updates](https://md.research.elamur.ai/free/en/papers/kowsher-2025-rocoft-efficient-finetuning-large-language/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa…
25. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. [The power of scale for parameter-efficient prompt tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3045–3059.
26. Shiwei Li, Xiandi Luo, Haozhao Wang, Xing Tang, Ziqiang Cui, Dugang Liu, Yuhua Li, Ruixuan Li, and 1 others. 2025. Beyond higher rank: Token-wise input-output projections for efficient low-rank adaptation. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Shiwei+Li%2C+Xiandi+Luo%2C+Haozhao+Wang%2C+Xing+Tang%2C+Ziqiang+Cui%2C+Dugang+Liu%2C+Yuhua+Li%2C+Ruixuan+Li%2C+and+1+others%2E+2025%2E+Beyond+higher+rank%3A+Token%2Dwise+input%2Doutput+projections+for+efficient+low%2Drank+adapta)
27. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL), pages 4582–4597. [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL%29%2C+page)
28. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158–167. [find](https://scholar.google.com/scholar?q=Wang+Ling%2C+Dani+Yogatama%2C+Chris+Dyer%2C+and+Phil+Blunsom%2E+2017%2E+Program+induction+by+rationale+generation%3A+Learning+to+solve+and+explain+algebraic+word+problems%2E+In+Proceedings+of+the+55th+Annual+Meetin)
29. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024a. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. [find](https://scholar.google.com/scholar?q=Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+and+1+others%2E+2024a%2E+Deepseek%2Dv3+technical+report%2E+Preprint%2C+arXiv%3A2412%2E194)
30. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024b. Dora: Weight-decomposed low-rank adaptation. In Proceedings of the 41st International Conference on Machine Learning (ICML). [find](https://scholar.google.com/scholar?q=Shih%2DYang+Liu%2C+Chien%2DYi+Wang%2C+Hongxu+Yin%2C+Pavlo+Molchanov%2C+Yu%2DChiang+Frank+Wang%2C+Kwang%2DTing+Cheng%2C+and+Min%2DHung+Chen%2E+2024b%2E+Dora%3A+Weight%2Ddecomposed+low%2Drank+adaptation%2E+In+Proceedings+of+the+41st+Int)
31. Zequan Liu, Yi Zhao, Ming Tan, Wei Zhu, and Aaron Xuxiang Tian. 2024c. Para: Parameter-efficient fine-tuning with prompt-aware representation adjustment. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 728–737. [find](https://scholar.google.com/scholar?q=Zequan+Liu%2C+Yi+Zhao%2C+Ming+Tan%2C+Wei+Zhu%2C+and+Aaron+Xuxiang+Tian%2E+2024c%2E+Para%3A+Parameter%2Defficient+fine%2Dtuning+with+prompt%2Daware+representation+adjustment%2E+In+Proceedings+of+the+2024+Conference+on+Empir)
32. Fanxu Meng, Zhaohui Wang, and Muhan Zhang. 2024. [Pissa: Principal singular values and singular vectors adaptation of large language models](https://md.research.elamur.ai/free/en/papers/meng-2024-pissa-principal-singular-values-singular/). In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pages 121038–121072.
33. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2381–2391. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2018%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+In+Proceedings+of+the+2018+conference+on+)
34. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. [Are NLP models really able to solve simple math word problems?](https://md.research.elamur.ai/free/en/papers/patel-2021-are-nlp-models-really-able/) In Proceedings of NAACL, pages 2080–2094.
35. Jing Qian and Venkatesh Saligrama. 2013. Spectral clustering with unbalanced data. Preprint, arXiv:1302.5134. — [arXiv:1302.5134](https://arxiv.org/abs/1302.5134) [find](https://scholar.google.com/scholar?q=Jing+Qian+and+Venkatesh+Saligrama%2E+2013%2E+Spectral+clustering+with+unbalanced+data%2E+Preprint%2C+arXiv%3A1302%2E5134%2E+%E2%80%94+%5BarXiv%3A1302%2E5134%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1302%2E5134%29)
36. Olivier Roy and Martin Vetterli. 2007. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pages 606–610. IEEE. [find](https://scholar.google.com/scholar?q=Olivier+Roy+and+Martin+Vetterli%2E+2007%2E+The+effective+rank%3A+A+measure+of+effective+dimensionality%2E+In+2007+15th+European+signal+processing+conference%2C+pages+606%E2%80%93610%2E+IEEE%2E)
37. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2021%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Communications+of+the+ACM%2C+64%289%29%3A99%E2%80%93106%2E)
38. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. [Socialiqa: Commonsense reasoning about social interactions](https://md.research.elamur.ai/free/en/papers/sap-2019-socialiqa-commonsense-reasoning-about-social/). arXiv preprint arXiv:1904.09728.
39. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). Preprint, arXiv:2307.09288.
40. Runyu Wang, Peng Ping, Zhengyu Guo, Xiaoye Zhang, Quan Shi, Liting Zhou, and Tianbo Ji. 2025. Loki: Low-damage knowledge implanting of large language models. arXiv preprint arXiv:2505.22120. — [arXiv:2505.22120](https://arxiv.org/abs/2505.22120) [find](https://scholar.google.com/scholar?q=Runyu+Wang%2C+Peng+Ping%2C+Zhengyu+Guo%2C+Xiaoye+Zhang%2C+Quan+Shi%2C+Liting+Zhou%2C+and+Tianbo+Ji%2E+2025%2E+Loki%3A+Low%2Ddamage+knowledge+implanting+of+large+language+models%2E+arXiv+preprint+arXiv%3A2505%2E22120%2E+%E2%80%94+%5BarXiv%3A)
41. Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan, and Jianfeng Gao. 2022. [Adamix: Mixture-of-adaptations for parameter-efficient model tuning](https://md.research.elamur.ai/free/en/papers/wang-2022-adamix-mixture-adaptations-parameter-efficient/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5744–5760.
42. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and 1 others. 2020. Transformers: State-ofthe-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lang… [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pier%2Dric+Cistac%2C+Tim+Rault%2C+R%C3%A9mi+Louf%2C+Morgan+Funtowicz%2C+and+1+others%2E+2020%2E+Transformers%3A+State%2Dofthe%2Dart+nat)
43. Yifeng Xiong and Xiaohui Xie. 2025. Oplora: Orthogonal projection lora prevents catastrophic forgetting during parameter-efficient fine-tuning. arXiv preprint arXiv:2510.13003. — [arXiv:2510.13003](https://arxiv.org/abs/2510.13003) [find](https://scholar.google.com/scholar?q=Yifeng+Xiong+and+Xiaohui+Xie%2E+2025%2E+Oplora%3A+Orthogonal+projection+lora+prevents+catastrophic+forgetting+during+parameter%2Defficient+fine%2Dtuning%2E+arXiv+preprint+arXiv%3A2510%2E13003%2E+%E2%80%94+%5BarXiv%3A2510%2E13003%5D%28ht)
44. Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. 2023. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. Preprint, arXiv:2312.12148. [find](https://scholar.google.com/scholar?q=Lingling+Xu%2C+Haoran+Xie%2C+Si%2DZhao+Joe+Qin%2C+Xiaohui+Tao%2C+and+Fu+Lee+Wang%2E+2023%2E+Parameter%2Defficient+fine%2Dtuning+methods+for+pretrained+language+models%3A+A+critical+review+and+assessment%2E+Preprint%2C+arXiv%3A)
45. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2.5 technical report. Preprint, arXiv:2412.15115. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+and+1+others%2E+2024%2E+Qwen2%2E5+technical+report%2E+Preprint%2C+arXiv%3A2412%2E15115%2E)
46. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Com)
47. Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. Adaptive budget allocation for parameter-efficient fine-tuning. In Proceedings of the 11th International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Qingru+Zhang%2C+Minshuo+Chen%2C+Alexander+Bukharin%2C+Pengcheng+He%2C+Yu+Cheng%2C+Weizhu+Chen%2C+and+Tuo+Zhao%2E+2023%2E+Adaptive+budget+allocation+for+parameter%2Defficient+fine%2Dtuning%2E+In+Proceedings+of+the+11th+Inte)
48. Ruiyi Zhang, Rushi Qiang, Sai Ashish Somayajula, and Pengtao Xie. 2024. Autolora: Automatically tuning matrix ranks in low-rank adaptation based on meta learning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech… [find](https://scholar.google.com/scholar?q=Ruiyi+Zhang%2C+Rushi+Qiang%2C+Sai+Ashish+Somayajula%2C+and+Pengtao+Xie%2E+2024%2E+Autolora%3A+Automatically+tuning+matrix+ranks+in+low%2Drank+adaptation+based+on+meta+learning%2E+In+Proceedings+of+the+2024+Conference)
49. A Proof. The update matrix ∆W l admits a factorization as the product of two matrices, M A ∈ R n×4r and M B ∈ R 4r×n ,satisfying ∆W l = M A M B :   A 1 A 3 0 0 M A = , 0 0 A 2 A 4   B 1 0  0 B 3   M B =   B 2 0  , 0 B 4 n [find](https://scholar.google.com/scholar?q=A+Proof%2E+The+update+matrix+%E2%88%86W+l+admits+a+factorization+as+the+product+of+two+matrices%2C+M+A+%E2%88%88+R+n%C3%974r+and+M+B+%E2%88%88+R+4r%C3%97n+%2Csatisfying+%E2%88%86W+l+%3D+M+A+M+B+%3A+%14+%15+A+1+A+3+0+0+M+A+%3D+%2C+0+0+A+2+A+4+%EF%A3%AE+%EF%A3%B9+B+1+0+%EF%A3%AF+0+B+3+)

*Updated 2 November 2026*