---
title: "Context-Conditioned Masked LoRA: Dynamic Rank Routing for"
description: "LoRA reduces how many parameters are trained, but it does not reduce how much computation the adapter performs. 27 references."
authors: ["Rifat Rafiuddin", "Rafae Abdullah"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1329/"
doi: "10.18653/v1/2026.findings-acl.1329"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rafiuddin-2026-context-conditioned-masked-lora-dynamic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rafiuddin-2026-context-conditioned-masked-lora-dynamic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/26/) / Context-Conditioned Masked LoRA: Dynamic Rank Routing for Compute-Efficient Parameter-Efficient Fine-Tuning

# Context-Conditioned Masked LoRA: Dynamic Rank Routing for Compute-Efficient Parameter-Efficient Fine-Tuning

**Authors:** [Rifat Rafiuddin](https://md.research.elamur.ai/free/en/authors/rifat-rafiuddin/) and Rafae Abdullah · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1329/) · [PDF](https://aclanthology.org/2026.findings-acl.1329.pdf) · [DOI 10.18653/v1/2026.findings-acl.1329](https://doi.org/10.18653/v1/2026.findings-acl.1329) · [OpenAlex](https://openalex.org/W7166884255)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
LoRA reduces how many parameters are trained, but it does not reduce how much computation the adapter performs. Once the rank is chosen, every token passes through the full low-rank update. CCM-LoRA addresses this by adding a small router. The router reads the backbone's hidden states, using either a pooled summary or only a short prefix, and selects which rank directions to activate for each input. Selection uses top-k gating with a straight-through estimator. Inactive rank rows and columns are skipped, so the savings are real rather than only on paper. Training combines the task loss with two extra terms:
- a Lagrangian penalty that pulls the expected number of active ranks toward a target budget;
- an entropy regularizer that prevents routing from collapsing into always-on or always-off masks.

The authors test the method on a T5-family model and Mistral-7B, using English NLU and multilingual benchmarks. Compared with static LoRA and adaptive-rank baselines, CCM-LoRA matches or exceeds accuracy at a lower average inference-time rank. The benefit is largest for longer sequences and heterogeneous, multilingual inputs. The work matters because it treats adapter capacity as a resource allocated per input, without changing the frozen backbone.

## Key points
- **Benchmarks:** GLUE and SuperGLUE for English NLU; XNLI (English training, zero-shot evaluation in other languages) and TyDi QA (EM/F1) for multilingual tasks.
- **Baselines:** full fine-tuning, static LoRA (r ∈ {8, 16, 32}), AdaLoRA, PRILoRA, DropLoRA, SparseLoRA, IA3 and DoRA.
- **Profiling protocol:**
  - Adapter FLOPs/token are reported separately from end-to-end latency, measured at batch sizes 1 and 8.
  - Router overhead is isolated.
  - Statistics on the per-input rank k(x) are reported as mean, P50 and P90.
- **Router variants:** token-pooling, prefix and module-conditioned. Per-module top-k routing proved the most stable configuration.
- **Limitations:**
  - Router overhead can cancel the gains for short inputs or small batches.
  - Speedups depend on kernels that actually skip inactive ranks.
  - Routing may allocate capacity unevenly across dialects or demographic groups, so the authors recommend group-wise auditing.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Oklahoma State University](https://md.research.elamur.ai/free/en/universities/oklahoma-state-university/) — University, Stillwater, United States · 3 works

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 242 works

## Cite

Cite as: Rifat Rafiuddin and Rafae Abdullah. 2026. Context-Conditioned Masked LoRA: Dynamic Rank Routing for Compute-Efficient Parameter-Efficient Fine-Tuning. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{rafiuddin2026context,
  title = {{Context-Conditioned Masked LoRA: Dynamic Rank Routing for Compute-Efficient Parameter-Efficient Fine-Tuning}},
  author = {Rifat Rafiuddin and Rafae Abdullah},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1329},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1329}
}
```

- [← previous Context-Aware Language Understanding in Human-Robot Dialogue with LLMs](https://md.research.elamur.ai/free/en/papers/stoyanchev-2026-context-aware-language-understanding-human/)
- [next → Contextual Diversity Measure (CDM) for Controllable Story Generation in Large Language Models](https://md.research.elamur.ai/free/en/papers/susilo-2026-contextual-diversity-measure-cdm-controllable/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/26/)

## References

27 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yash Akhauri, Ahmed F. AbouElhamayed, Jordan Dotzel, Zhiru Zhang, Alexander M. Rush, Safeen Huda, and Mohamed S. Abdelfattah. 2024. ShadowLLM: Predictor-based contextual sparsity for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pag… [find](https://scholar.google.com/scholar?q=Yash+Akhauri%2C+Ahmed+F%2E+AbouElhamayed%2C+Jordan+Dotzel%2C+Zhiru+Zhang%2C+Alexander+M%2E+Rush%2C+Safeen+Huda%2C+and+Mohamed+S%2E+Abdelfattah%2E+2024%2E+ShadowLLM%3A+Predictor%2Dbased+contextual+sparsity+for+large+language+mo)
2. Nadav Benedek and Lior Wolf. 2024. PRILoRA: Pruned and rank-increasing low-rank adaptation. In Findings of the Association for Computational Linguistics: EACL 2024, pages 252–263, St. Julian’s, Malta. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nadav+Benedek+and+Lior+Wolf%2E+2024%2E+PRILoRA%3A+Pruned+and+rank%2Dincreasing+low%2Drank+adaptation%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EACL+2024%2C+pages+252%E2%80%93263%2C+St%2E+Julian%E2%80%99s%2C+Malta%2E)
3. Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2C+Nicholas+L%C3%A9onard%2C+and+Aaron+Courville%2E+2013%2E+Estimating+or+propagating+gradients+through+stochastic+neurons+for+conditional+computation%2E+arXiv+preprint+arXiv%3A1308%2E3432%2E)
4. Minping Chen, Ruijia Yang, and Zeyi Wen. 2025. Efficient mask learning for language model fine-tuning. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management (CIKM ’25), pages 301–311, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Minping+Chen%2C+Ruijia+Yang%2C+and+Zeyi+Wen%2E+2025%2E+Efficient+mask+learning+for+language+model+fine%2Dtuning%2E+In+Proceedings+of+the+34th+ACM+International+Conference+on+Information+and+Knowledge+Management+%28)
5. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. [Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/). arXiv preprint arXiv:2003.05002. To appear in TACL 2020.
6. Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. arXiv preprint arXiv:1809.05053. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Guillaume+Lample%2C+Ruty+Rinott%2C+Adina+Williams%2C+Samuel+R%2E+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+2018%2E+Xnli%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+arXiv+preprint+arX)
7. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. [Qlora: Efficient finetuning of quantized LLMs](https://md.research.elamur.ai/free/en/papers/dettmers-2023-qlora-efficient-finetuning-quantized-llms/). arXiv preprint arXiv:2305.14314.
8. Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. 2023. Sparse low-rank adaptation of pre-trained language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4133–4145, Singapore. Association for Computati… [find](https://scholar.google.com/scholar?q=Ning+Ding%2C+Xingtai+Lv%2C+Qiaosen+Wang%2C+Yulin+Chen%2C+Bowen+Zhou%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2023%2E+Sparse+low%2Drank+adaptation+of+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+2023+Conference+on+Emp)
9. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024. DoRA: Weight-decomposed low-rank adaptation. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pa… [find](https://scholar.google.com/scholar?q=Shih%2DYang+Liu%2C+Chien%2DYi+Wang%2C+Hongxu+Yin%2C+Pavlo+Molchanov%2C+Yu%2DChiang+Frank+Wang%2C+Kwang%2DTing+Cheng%2C+and+Min%2DHung+Chen%2E+2024%2E+DoRA%3A+Weight%2Ddecomposed+low%2Drank+adaptation%2E+In+Proceedings+of+the+41st+Inte)
10. William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39. [find](https://scholar.google.com/scholar?q=William+Fedus%2C+Barret+Zoph%2C+and+Noam+Shazeer%2E+2022%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+Journal+of+Machine+Learning+Research%2C+23%28120%29%3A1%E2%80%9339%2E)
11. Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, and Beidi Chen. 2023. Deja vu: Contextual sparsity for efficient LLMs at inference time. In Proceedings of the 40th International Conference on Machine Learning, volu… [find](https://scholar.google.com/scholar?q=Zichang+Liu%2C+Jue+Wang%2C+Tri+Dao%2C+Tianyi+Zhou%2C+Binhang+Yuan%2C+Zhao+Song%2C+Anshumali+Shrivastava%2C+Ce+Zhang%2C+Yuandong+Tian%2C+Christopher+Re%2C+and+Beidi+Chen%2E+2023%2E+Deja+vu%3A+Contextual+sparsity+for+efficient+L)
12. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Ma… [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+De+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+NLP%2E+In+)
13. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). arXiv preprint arXiv:2106.09685.
14. Eric Jang, Shixiang Gu, and Ben Poole. 2017. [Categorical reparameterization with Gumbel-Softmax](https://md.research.elamur.ai/free/en/papers/jang-2016-categorical-reparameterization-gumbel-softmax/). arXiv preprint arXiv:1611.01144.
15. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+Las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
16. Samir Khaki, Xiuyu Li, Junxian Guo, Ligeng Zhu, Konstantinos N. Plataniotis, Amir Yazdanbakhsh, Kurt Keutzer, Song Han, and Zhijian Liu. 2025. SparseLoRA: Accelerating LLM fine-tuning with contextual sparsity. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Pro… — [arXiv:2506.16500](https://arxiv.org/abs/2506.16500) [find](https://scholar.google.com/scholar?q=Samir+Khaki%2C+Xiuyu+Li%2C+Junxian+Guo%2C+Ligeng+Zhu%2C+Konstantinos+N%2E+Plataniotis%2C+Amir+Yazdanbakhsh%2C+Kurt+Keutzer%2C+Song+Han%2C+and+Zhijian+Liu%2E+2025%2E+SparseLoRA%3A+Accelerating+LLM+fine%2Dtuning+with+contextual+)
17. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. [The power of scale for parameter-efficient prompt tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguist…
18. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190. [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+arXiv+preprint+arXiv%3A2101%2E00190%2E)
19. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022. [Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning](https://md.research.elamur.ai/free/en/papers/liu-2022-few-shot-parameter-efficient-fine/). arXiv preprint arXiv:2205.05638.
20. Christos Louizos, Max Welling, and Diederik P. Kingma. 2018. Learning sparse neural networks through L 0 regularization. In International Conference on Learning Representations (ICLR). ArXiv:1712.01312. [find](https://scholar.google.com/scholar?q=Christos+Louizos%2C+Max+Welling%2C+and+Diederik+P%2E+Kingma%2E+2018%2E+Learning+sparse+neural+networks+through+L+0+regularization%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E+ArXiv%3A1712%2E0131)
21. Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. 2017. [The concrete distribution: A continuous relaxation of discrete random variables](https://md.research.elamur.ai/free/en/papers/maddison-2016-concrete-distribution-continuous-relaxation-discrete/). In International Conference on Learning Representations (ICLR). ArXiv:1611.00712.
22. Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. [AdapterHub: A framework for adapting transformers](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterhub-framework-adapting-transformers/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,…
23. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67.
24. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). arXiv preprint arXiv:1905.00537.
25. Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). arXiv preprint arXiv:1804.07461.
26. Haojie Zhang. 2025. Droplora: Sparse low-rank adaptation for parameter-efficient fine-tuning. arXiv preprint arXiv:2508.17337. — [arXiv:2508.17337](https://arxiv.org/abs/2508.17337) [find](https://scholar.google.com/scholar?q=Haojie+Zhang%2E+2025%2E+Droplora%3A+Sparse+low%2Drank+adaptation+for+parameter%2Defficient+fine%2Dtuning%2E+arXiv+preprint+arXiv%3A2508%2E17337%2E+%E2%80%94+%5BarXiv%3A2508%2E17337%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2508%2E17337%29)
27. Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. [AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning](https://md.research.elamur.ai/free/en/papers/zhang-2023-adalora-adaptive-budget-allocation-parameter/). In The Eleventh International Conference on Learning Representations (ICLR 2023). ArXiv:2303.10512.

*Updated 1 November 2026*