---
title: "Hebbian-Guided Bi-Directional Rank Adaptation for"
description: "LoRA gives every adapted layer the same rank, even though layers differ in how much they matter for a downstream task. 21 references."
authors: ["Xing Zhao", "Liu Yang", "Xi-Le Zhao", "Yueqi Yang", "Tianqi Jiang"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.954/"
doi: "10.18653/v1/2026.findings-acl.954"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhao-2026-hebbian-guided-bi-directional-rank/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhao-2026-hebbian-guided-bi-directional-rank/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/36/) / Hebbian-Guided Bi-Directional Rank Adaptation for Parameter-Efficient Fine-Tuning

# Hebbian-Guided Bi-Directional Rank Adaptation for Parameter-Efficient Fine-Tuning

**Authors:** Xing Zhao, [Liu Yang](https://md.research.elamur.ai/free/en/authors/liu-yang/), Xi-Le Zhao, Yueqi Yang and Tianqi Jiang · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.954/) · [PDF](https://aclanthology.org/2026.findings-acl.954.pdf) · [DOI 10.18653/v1/2026.findings-acl.954](https://doi.org/10.18653/v1/2026.findings-acl.954) · [OpenAlex](https://openalex.org/W7166811859)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
LoRA gives every adapted layer the same rank, even though layers differ in how much they matter for a downstream task. Earlier dynamic-rank methods address this by either removing ranks (AdaLoRA, PriLoRA) or adding them (IncreLoRA, TriAdaptLoRA). Doing only one leaves either too few effective directions or many underused ones. HeBiRA does both. It writes the LoRA update in SVD form (B·E·A) and treats each rank triple (column of B, singular value, row of A) as a separate unit. Each unit gets an importance score inspired by Hebbian learning, which combines how strongly the input activates the direction with how much its output gradient affects the loss. At fixed intervals, the weakest triples are removed and new ones are added in layers judged important. The method also works on top of DoRA, HiRA and QLoRA. It was tested on GLUE with DeBERTaV3-base, code generation (MBPP/MBPP+) with LLaMA2-7B and LLaMA3-8B, and XSum summarization, and it generally beats fixed-rank and single-direction baselines. This suggests that combining pruning with expansion makes better use of a limited parameter budget.

## Key points
- **Ablation on GLUE (DeBERTaV3-base).** Average score was 88.50 for plain LoRA, 89.23 with pruning only and 89.44 with expansion only. Full HeBiRA reached 89.80.
- **Importance criteria compared.** The combined activation-gradient score did better than either signal alone. On RTE it scored 88.09, versus 86.64 for activation only and 87.72 for gradient only.
- **Code generation with QLoRA on LLaMA3-8B.** Adding HeBiRA raised MBPP from 70.6 to 74.3 and MBPP+ from 60.6 to 63.8. DoRA and LLaMA2-7B also improved.
- **XSum.** ROUGE-L was 35.31 for LoRA, 35.63 for AdaLoRA and 35.96 for HeBiRA.
- **Limitation.** The adjustment size k (tested at 4, 8, 12 and 24 with initial rank 8) is a fixed hyperparameter, and its best value depends on the model architecture and the initial rank.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tianjin University](https://md.research.elamur.ai/free/en/universities/tianjin-university/) — University, Tianjin, China · 52 works · 1,128 citations
- [University of Electronic Science and Technology of China](https://md.research.elamur.ai/free/en/universities/university-of-electronic-science-and-technology-of-china/) — University, Chengdu, China · 38 works · 2,088 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Xing Zhao, Liu Yang, Xi-Le Zhao, Yueqi Yang and Tianqi Jiang. 2026. Hebbian-Guided Bi-Directional Rank Adaptation for Parameter-Efficient Fine-Tuning. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhao2026hebbian,
  title = {{Hebbian-Guided Bi-Directional Rank Adaptation for Parameter-Efficient Fine-Tuning}},
  author = {Xing Zhao, Liu Yang, Xi-Le Zhao, Yueqi Yang and Tianqi Jiang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.954},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.954}
}
```

- [← previous HEBATRON: A Hebrew-Specialized Open-Weight Mixture-of-Experts Language Model](https://md.research.elamur.ai/free/en/papers/kayzer-2026-hebatron-hebrew-specialized-open-weight/)
- [next → Helpful or Harmful? The Dual Role of Linguistic Features in LLM-Based Dialectal Machine Translation](https://md.research.elamur.ai/free/en/papers/dahou-2026-helpful-or-harmful-dual-role/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/36/)

## References

21 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Hebb Do. 1949. The organization of behavior: A neuropsychological theory. Wiley. [find](https://scholar.google.com/scholar?q=Hebb+Do%2E+1949%2E+The+organization+of+behavior%3A+A+neuropsychological+theory%2E+Wiley%2E)
2. Paul Albert, Frederic Z. Zhang, Hemanth Saratchandran, Cristian Rodriguez Opazo, Anton van den Hengel, and Ehsan Abbasnejad. 2025. [Randlora: Full rank Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, A…](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
3. Soufiane Hayou, Nikhil Ghosh, and Bin Yu. 2024. [Lora+: Efficient low rank adaptation of large models](https://md.research.elamur.ai/free/en/papers/hayou-2024-lora-efficient-low-rank-adaptation/). In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.
4. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. [Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing](https://md.research.elamur.ai/free/en/papers/he-2021-debertav3-improving-deberta-using-electra/). In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
5. Zhiwei He, Zhaopeng Tu, Xing Wang, Xingyu Chen, Zhijie Wang, Jiahao Xu, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, and Rui Wang. 2025. Rasa: Rank-sharing low-rank adaptation. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.… [find](https://scholar.google.com/scholar?q=Zhiwei+He%2C+Zhaopeng+Tu%2C+Xing+Wang%2C+Xingyu+Chen%2C+Zhijie+Wang%2C+Jiahao+Xu%2C+Tian+Liang%2C+Wenxiang+Jiao%2C+Zhuosheng+Zhang%2C+and+Rui+Wang%2E+2025%2E+Rasa%3A+Rank%2Dsharing+low%2Drank+adaptation%2E+In+The+Thirteenth+Intern)
6. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In NeurIPS Datasets and Benchmarks.
7. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Lon… [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+de+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+NLP%2E+In+)
8. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
9. Qiushi Huang, Tom Ko, Zhan Zhuang, Lilian Tang, and Yu Zhang. 2025. Hira: Parameter-efficient hadamard high-rank adaptation for large language models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. [find](https://scholar.google.com/scholar?q=Qiushi+Huang%2C+Tom+Ko%2C+Zhan+Zhuang%2C+Lilian+Tang%2C+and+Yu+Zhang%2E+2025%2E+Hira%3A+Parameter%2Defficient+hadamard+high%2Drank+adaptation+for+large+language+models%2E+In+The+Thirteenth+International+Conference+on+Lea)
10. Damjan Kalajdzievski. 2023. A rank stabilization scaling factor for fine-tuning with lora. CoRR, abs/2312.03732. — [arXiv:2312.03732](https://arxiv.org/abs/2312.03732) [find](https://scholar.google.com/scholar?q=Damjan+Kalajdzievski%2E+2023%2E+A+rank+stabilization+scaling+factor+for+fine%2Dtuning+with+lora%2E+CoRR%2C+abs%2F2312%2E03732%2E+%E2%80%94+%5BarXiv%3A2312%2E03732%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2312%2E03732%29)
11. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. [The power of scale for parameter-efficient prompt tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 3045– 3059.…
12. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. [Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). CoRR, abs/1910.13461.
13. Yao Liang, Yuwei Wang, and Yi Zeng. 2025. Triadaptlora: Brain-inspired triangular adaptive low-rank adaptation for parameter-efficient fine-tuning. CoRR, abs/2501.08008. — [arXiv:2501.08008](https://arxiv.org/abs/2501.08008) [find](https://scholar.google.com/scholar?q=Yao+Liang%2C+Yuwei+Wang%2C+and+Yi+Zeng%2E+2025%2E+Triadaptlora%3A+Brain%2Dinspired+triangular+adaptive+low%2Drank+adaptation+for+parameter%2Defficient+fine%2Dtuning%2E+CoRR%2C+abs%2F2501%2E08008%2E+%E2%80%94+%5BarXiv%3A2501%2E08008%5D%28https%3A%2F%2Fa)
14. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net. [find](https://scholar.google.com/scholar?q=Shih%2DYang+Liu%2C+Chien%2DYi+Wang%2C+Hongxu+Yin%2C+Pavlo+Molchanov%2C+Yu%2DChiang+Frank+Wang%2C+Kwang%2DTing+Cheng%2C+and+Min%2DHung+Chen%2E+2024%2E+Dora%3A+Weight%2Ddecomposed+low%2Drank+adaptation%2E+In+Forty%2Dfirst+International+Co)
15. Yulong Mao, Kaiyu Huang, Changhao Guan, Ganglin Bao, Fengran Mo, and Jinan Xu. 2024. Dora: Enhancing parameter-efficient fine-tuning with dynamic rank distribution. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok,… [find](https://scholar.google.com/scholar?q=Yulong+Mao%2C+Kaiyu+Huang%2C+Changhao+Guan%2C+Ganglin+Bao%2C+Fengran+Mo%2C+and+Jinan+Xu%2E+2024%2E+Dora%3A+Enhancing+parameter%2Defficient+fine%2Dtuning+with+dynamic+rank+distribution%2E+In+Proceedings+of+the+62nd+Annual+M)
16. Fanxu Meng, Zhaohui Wang, and Muhan Zhang. 2024. [Pissa: Principal singular values and singular vectors adaptation of large language models](https://md.research.elamur.ai/free/en/papers/meng-2024-pissa-principal-singular-values-singular/). In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, D…
17. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. [Don’t give me the details, just the summary!](https://md.research.elamur.ai/free/en/papers/narayan-2018-don-t-give-me-details/) topic-aware convolutional neural networks for extreme summarization. CoRR, abs/1808.08745.
18. Olivier Roy and Martin Vetterli. 2007. The effective rank: A measure of effective dimensionality. In 15th European Signal Processing Conference, EUSIPCO 2007, Poznan, Poland, September 3-7, 2007, pages 606–610. IEEE. [find](https://scholar.google.com/scholar?q=Olivier+Roy+and+Martin+Vetterli%2E+2007%2E+The+effective+rank%3A+A+measure+of+effective+dimensionality%2E+In+15th+European+Signal+Processing+Conference%2C+EUSIPCO+2007%2C+Poznan%2C+Poland%2C+September+3%2D7%2C+2007%2C+page)
19. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
20. Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. 2023. [Dylora: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation](https://md.research.elamur.ai/free/en/papers/valipour-2023-dylora-parameter-efficient-tuning-pre/). In EACL, pages 3266–3279.
21. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRevie…

*Updated 2 November 2026*