---
title: "Federated LoRA Fine-Tuning with Pipelined Error-Mitigated"
description: "This introduces a systematic aggregation error, because the product of averaged factors is not the average of the per-client… 41 references."
authors: ["Haoran Wang", "Xiong Wang", "Yuqing Li", "Jing Chen", "Junyi Zhang", "Nan Yan", "Kun He", "Wei Wang"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.284/"
doi: "10.18653/v1/2026.findings-acl.284"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2026-federated-lora-fine-tuning-pipelined/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2026-federated-lora-fine-tuning-pipelined/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/30/) / Federated LoRA Fine-Tuning with Pipelined Error-Mitigated Aggregation and Matrix-Wise Freezing

# Federated LoRA Fine-Tuning with Pipelined Error-Mitigated Aggregation and Matrix-Wise Freezing

**Authors:** [Haoran Wang](https://md.research.elamur.ai/free/en/authors/haoran-wang/), [Xiong Wang](https://md.research.elamur.ai/free/en/authors/xiong-wang/), Yuqing Li, [Jing Chen](https://md.research.elamur.ai/free/en/authors/jing-chen/), Junyi Zhang, [Nan Yan](https://md.research.elamur.ai/free/en/authors/nan-yan/), [Kun He](https://md.research.elamur.ai/free/en/authors/kun-he/) and [Wei Wang](https://md.research.elamur.ai/free/en/authors/wei-wang/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.284/) · [PDF](https://aclanthology.org/2026.findings-acl.284.pdf) · [DOI 10.18653/v1/2026.findings-acl.284](https://doi.org/10.18653/v1/2026.findings-acl.284) · [OpenAlex](https://openalex.org/W7166840892)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Federated LoRA lets several clients jointly adapt an LLM without sharing raw data. Standard systems average the two low-rank factors B and A separately under FedAvg. This introduces a systematic aggregation error, because the product of averaged factors is not the average of the per-client products. Edge clients also face heavy communication and compute costs. The authors propose iFLoRA to address both problems. The server rebuilds each client's full update as BA, averages these updates, and factorizes the result with SVD. It then keeps only the important singular values, which sets the LoRA rank for the next round. To hide the extra SVD cost, server aggregation runs per matrix in parallel with clients' backward passes and uploads. A matrix-wise freezing scheme detects weight matrices that have stabilized and pauses them for adaptively chosen periods. Experiments used BERT-large, RoBERTa and Llama-3.2-3B on 20NEWS, AGNEWS, SQuAD2.0 and SAMSum. iFLoRA reached target accuracy 2.17–8.48× faster than prior methods and sharply cut network traffic. This makes collaborative, privacy-preserving LLM tuning more practical on resource-limited devices.

## Key points
- **Baselines and traffic:** iFLoRA was compared with FedPETuning, FFA-LoRA and FedEx-LoRA. It reduced network traffic by 74.47–97.96%, 90.19–97.81% and 94.09–99.36% respectively, and lowered client memory use by 9.42–25.47%.
- **Ablation on BERT-large / 20NEWS:**
  - SVD-based aggregation with importance-aware rank gave a 2.18× speedup.
  - Freezing alone gave 2.12×.
  - The full system (aggregation, pipelining and freezing) reached 2.37×.
- **SVD overhead:** SVD took about 68.04 ms per aggregated matrix for Llama-3.2-3B on RTX 4090 GPUs. Storing low-rank factors instead of full updates cut server memory for freezing from 75 GB to 1.03 GB.
- **Motivating experiment (Llama-3.2-3B on SQuAD2.0):**
  - Layers near the input converged earlier than those near the output.
  - Freezing the first 12 layers lowered F1 only from 92.89% to 91.16%.
- **Limitations:** Differential privacy was not evaluated. The matrix multiplication and SVD steps are incompatible with current secure aggregation protocols. Code is released on GitHub (whr819987540/iflora).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Hong Kong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/hong-kong-university-of-science-and-technology/) — University, Hong Kong, China · 142 works · 25,136 citations
- [Huazhong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/huazhong-university-of-science-and-technology/) — University, Wuhan, China · 45 works · 6,345 citations
- [Wuhan University](https://md.research.elamur.ai/free/en/universities/wuhan-university/) — University, Wuhan, China · 35 works · 1,127 citations

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works

## Cite

Cite as: Haoran Wang, Xiong Wang, Yuqing Li, Jing Chen, Junyi Zhang, Nan Yan, Kun He and Wei Wang. 2026. Federated LoRA Fine-Tuning with Pipelined Error-Mitigated Aggregation and Matrix-Wise Freezing. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2026federated,
  title = {{Federated LoRA Fine-Tuning with Pipelined Error-Mitigated Aggregation and Matrix-Wise Freezing}},
  author = {Haoran Wang, Xiong Wang, Yuqing Li, Jing Chen, Junyi Zhang, Nan Yan, Kun He and Wei Wang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.284},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.284}
}
```

- [← previous Faulkner, Amy, and Francis Leneghan, eds. 2024. The Age of Alfred: Rethinking English Literary Culture c. 850–950. Studies in Old English Literature 3. Turnhout: Brepols. Pp. 598. ISBN 9782503606651.](https://md.research.elamur.ai/free/en/papers/dance-2026-faulkner-amy-francis-leneghan-eds/)
- [next → FedGUI: Benchmarking Federated GUI Agents across Heterogeneous Platforms, Devices, and Operating Systems](https://md.research.elamur.ai/free/en/papers/wang-2026-fedgui-benchmarking-federated-gui-agents/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/30/)

## References

41 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, and 1 others. 2024. [Phi-3 technical report: A highly capable language model locally on your phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/). arXiv preprint arXiv:2404.14219.
2. Chen Chen, Hong Xu, Wei Wang, Baochun Li, Bo Li, Li Chen, and Gong Zhang. 2021. Communication-efficient federated learning with adaptive parameter freezing. In Proceedings of the 41st IEEE International Conference on Distributed Computing Systems (ICDCS), pages 1–11. [find](https://scholar.google.com/scholar?q=Chen+Chen%2C+Hong+Xu%2C+Wei+Wang%2C+Baochun+Li%2C+Bo+Li%2C+Li+Chen%2C+and+Gong+Zhang%2E+2021%2E+Communication%2Defficient+federated+learning+with+adaptive+parameter+freezing%2E+In+Proceedings+of+the+41st+IEEE+Internation)
3. Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. 2024. Heterogeneous lora for federated fine-tuning of on-device foundation models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 12903–12913. [find](https://scholar.google.com/scholar?q=Yae+Jee+Cho%2C+Luyang+Liu%2C+Zheng+Xu%2C+Aldi+Fahrezi%2C+and+Gauri+Joshi%2E+2024%2E+Heterogeneous+lora+for+federated+fine%2Dtuning+of+on%2Ddevice+foundation+models%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+)
4. Rohan Deb, Kiran Koshy Thekumparampil, Kousha Kalantari, Gaurush Hiranandani, Shoham Sabach, and Branislav Kveton. 2025. Fishersft: Data-efficient supervised fine-tuning of language models using information gain. In Proceedings of the 42nd International Conference on Machine Learning (ICML), pages 1… [find](https://scholar.google.com/scholar?q=Rohan+Deb%2C+Kiran+Koshy+Thekumparampil%2C+Kousha+Kalantari%2C+Gaurush+Hiranandani%2C+Shoham+Sabach%2C+and+Branislav+Kveton%2E+2025%2E+Fishersft%3A+Data%2Defficient+supervised+fine%2Dtuning+of+language+models+using+infor)
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies…
6. Minxin Du, Xiang Yue, Sherman S. M. Chow, Tianhao Wang, Chenyu Huang, and Huan Sun. 2023. Dp-forward: Fine-tuning and inference on language models with differential privacy in forward pass. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 2665–267… [find](https://scholar.google.com/scholar?q=Minxin+Du%2C+Xiang+Yue%2C+Sherman+S%2E+M%2E+Chow%2C+Tianhao+Wang%2C+Chenyu+Huang%2C+and+Huan+Sun%2E+2023%2E+Dp%2Dforward%3A+Fine%2Dtuning+and+inference+on+language+models+with+differential+privacy+in+forward+pass%2E+In+Proceed)
7. Carl Eckart and Gale Young. 1936. The approximation of one matrix by another of lower rank. Psychometrika, 1(3):211–218. [find](https://scholar.google.com/scholar?q=Carl+Eckart+and+Gale+Young%2E+1936%2E+The+approximation+of+one+matrix+by+another+of+lower+rank%2E+Psychometrika%2C+1%283%29%3A211%E2%80%93218%2E)
8. Zhidong Gao, Zhenxiao Zhang, Yuanxiong Guo, and Yanmin Gong. 2025. Federated adaptive fine-tuning of large language models with heterogeneous quantization and lora. In Proceedings of IEEE INFOCOM 2025 - IEEE Conference on Computer Communications, pages 1–10. [find](https://scholar.google.com/scholar?q=Zhidong+Gao%2C+Zhenxiao+Zhang%2C+Yuanxiong+Guo%2C+and+Yanmin+Gong%2E+2025%2E+Federated+adaptive+fine%2Dtuning+of+large+language+models+with+heterogeneous+quantization+and+lora%2E+In+Proceedings+of+IEEE+INFOCOM+2025)
9. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. [Samsum corpus: A human-annotated dialogue dataset for abstractive summarization](https://md.research.elamur.ai/free/en/papers/gliwa-2019-samsum-corpus-human-annotated-dialogue/). In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70–79.
10. Per Christian Hansen. 1987. The truncated svd as a method for regularization. BIT Numerical Mathematics, 27(4):534–553. [find](https://scholar.google.com/scholar?q=Per+Christian+Hansen%2E+1987%2E+The+truncated+svd+as+a+method+for+regularization%2E+BIT+Numerical+Mathematics%2C+27%284%29%3A534%E2%80%93553%2E)
11. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In Proceedings of the 10th International Conference on Learning Representations (ICLR).
12. Ian T Jolliffe. 2002. Principal Component Analysis for Special Types of Data. Springer New York, New York, NY. [find](https://scholar.google.com/scholar?q=Ian+T+Jolliffe%2E+2002%2E+Principal+Component+Analysis+for+Special+Types+of+Data%2E+Springer+New+York%2C+New+York%2C+NY%2E)
13. Fan Lai, Yinwei Dai, Sanjay Sri Vallabh Singapuram, Jiachen Liu, Xiangfeng Zhu, Harsha V. Madhyastha, and Mosharaf Chowdhury. 2022. Fedscale: Benchmarking model and system performance of federated learning at scale. In Proceedings of the 39th International Conference on Machine Learning (ICML), page… [find](https://scholar.google.com/scholar?q=Fan+Lai%2C+Yinwei+Dai%2C+Sanjay+Sri+Vallabh+Singapuram%2C+Jiachen+Liu%2C+Xiangfeng+Zhu%2C+Harsha+V%2E+Madhyastha%2C+and+Mosharaf+Chowdhury%2E+2022%2E+Fedscale%3A+Benchmarking+model+and+system+performance+of+federated+lea)
14. Fan Lai, Xiangfeng Zhu, Harsha V. Madhyastha, and Mosharaf Chowdhury. 2021. Oort: Efficient federated learning via guided participant selection. In Proceedings of the 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI), pages 19–35. [find](https://scholar.google.com/scholar?q=Fan+Lai%2C+Xiangfeng+Zhu%2C+Harsha+V%2E+Madhyastha%2C+and+Mosharaf+Chowdhury%2E+2021%2E+Oort%3A+Efficient+federated+learning+via+guided+participant+selection%2E+In+Proceedings+of+the+15th+USENIX+Symposium+on+Operatin)
15. Ken Lang. 1995. Newsweeder: Learning to filter netnews. In Proceedings of the 12th International Conference on Machine Learning (ICML), pages 331– 339. [find](https://scholar.google.com/scholar?q=Ken+Lang%2E+1995%2E+Newsweeder%3A+Learning+to+filter+netnews%2E+In+Proceedings+of+the+12th+International+Conference+on+Machine+Learning+%28ICML%29%2C+pages+331%E2%80%93+339%2E)
16. Chenning Li, Xiao Zeng, Mi Zhang, and Zhichao Cao. 2022. Pyramidfl: a fine-grained client selection framework for efficient federated learning. In Proceedings of ACM MobiCom ’22: the 28th Annual International Conference on Mobile Computing and Networking, pages 158–171. [find](https://scholar.google.com/scholar?q=Chenning+Li%2C+Xiao+Zeng%2C+Mi+Zhang%2C+and+Zhichao+Cao%2E+2022%2E+Pyramidfl%3A+a+fine%2Dgrained+client+selection+framework+for+efficient+federated+learning%2E+In+Proceedings+of+ACM+MobiCom+%E2%80%9922%3A+the+28th+Annual+Inter)
17. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692.
18. Yuhan Liu, Saurabh Agarwal, and Shivaram Venkataraman. 2021. [Autofreeze: Automatically freezing model blocks to accelerate fine-tuning](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692.
19. Bing Luo, Wenli Xiao, Shiqiang Wang, Jianwei Huang, and Leandros Tassiulas. 2022. Tackling system and statistical heterogeneity for federated learning with adaptive client sampling. In Proceedings of IEEE INFOCOM 2022 - IEEE Conference on Computer Communications, pages 1739–1748. [find](https://scholar.google.com/scholar?q=Bing+Luo%2C+Wenli+Xiao%2C+Shiqiang+Wang%2C+Jianwei+Huang%2C+and+Leandros+Tassiulas%2E+2022%2E+Tackling+system+and+statistical+heterogeneity+for+federated+learning+with+adaptive+client+sampling%2E+In+Proceedings+of+)
20. M-Lab. 2022. The m-lab mobiperf data set. [find](https://scholar.google.com/scholar?q=M%2DLab%2E+2022%2E+The+m%2Dlab+mobiperf+data+set%2E)
21. Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, Benjamin Bossan, and Marian Tietz. 2022. PEFT: State-of-the-art parameter-efficient fine-tuning methods. [find](https://scholar.google.com/scholar?q=Sourab+Mangrulkar%2C+Sylvain+Gugger%2C+Lysandre+Debut%2C+Younes+Belkada%2C+Sayak+Paul%2C+Benjamin+Bossan%2C+and+Marian+Tietz%2E+2022%2E+PEFT%3A+State%2Dof%2Dthe%2Dart+parameter%2Defficient+fine%2Dtuning+methods%2E)
22. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS), pages 1273–1282. [find](https://scholar.google.com/scholar?q=Brendan+McMahan%2C+Eider+Moore%2C+Daniel+Ramage%2C+Seth+Hampson%2C+and+Blaise+Ag%C3%BCera+y+Arcas%2E+2017%2E+Communication%2Defficient+learning+of+deep+networks+from+decentralized+data%2E+In+Proceedings+of+the+20th+Intern)
23. Sachin Mehta, Mohammad Sekhavat, Qingqing Cao, Max Horton, Yanzi Jin, Frank Sun, Iman Mirzadeh, Mahyar Najibikohnehshahri, Dmitry Belenko, Peter Zatloukal, and Mohammad Rastegari. 2024. Openelm: An efficient language model family with open training and inference framework. In Proceedings of the 2nd… [find](https://scholar.google.com/scholar?q=Sachin+Mehta%2C+Mohammad+Sekhavat%2C+Qingqing+Cao%2C+Max+Horton%2C+Yanzi+Jin%2C+Frank+Sun%2C+Iman+Mirzadeh%2C+Mahyar+Najibikohnehshahri%2C+Dmitry+Belenko%2C+Peter+Zatloukal%2C+and+Mohammad+Rastegari%2E+2024%2E+Openelm%3A+An+ef)
24. NVIDIA. 2025. Nvidia jetson modules. [find](https://scholar.google.com/scholar?q=NVIDIA%2E+2025%2E+Nvidia+jetson+modules%2E)
25. Zhaopeng Peng, Xiaoliang Fan, Yufan Chen, Zheng Wang, Shirui Pan, Chenglu Wen, Ruisheng Zhang, and Cheng Wang. 2024. Fedpft: Federated proxy fine-tuning of foundation models. In Proceedings of the 33rd International Joint Conference on Artificial Intelligence (IJCAI), pages 4806–4814. [find](https://scholar.google.com/scholar?q=Zhaopeng+Peng%2C+Xiaoliang+Fan%2C+Yufan+Chen%2C+Zheng+Wang%2C+Shirui+Pan%2C+Chenglu+Wen%2C+Ruisheng+Zhang%2C+and+Cheng+Wang%2E+2024%2E+Fedpft%3A+Federated+proxy+fine%2Dtuning+of+foundation+models%2E+In+Proceedings+of+the+33r)
26. Wenqi Qiu, Yipeng Zhou, Jinzhi Wang, Quan Z. Sheng, and Laizhong Cui. 2025. Flm-topk: Expediting federated large language model tuning by sparsifying intervalized gradients. In Proceedings of IEEE IN-FOCOM 2025 - IEEE Conference on Computer Communications, pages 1–10. [find](https://scholar.google.com/scholar?q=Wenqi+Qiu%2C+Yipeng+Zhou%2C+Jinzhi+Wang%2C+Quan+Z%2E+Sheng%2C+and+Laizhong+Cui%2E+2025%2E+Flm%2Dtopk%3A+Expediting+federated+large+language+model+tuning+by+sparsifying+intervalized+gradients%2E+In+Proceedings+of+IEEE+IN%2D)
27. Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. 2017. SVCCA: singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), volume 30, pages 6076–6085. [find](https://scholar.google.com/scholar?q=Maithra+Raghu%2C+Justin+Gilmer%2C+Jason+Yosinski%2C+and+Jascha+Sohl%2DDickstein%2E+2017%2E+SVCCA%3A+singular+vector+canonical+correlation+analysis+for+deep+learning+dynamics+and+interpretability%2E+In+Proceedings+of+)
28. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. [Know what you don’t know: Unanswerable questions for squad](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 784–789.
29. Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten de Rijke, Zhumin Chen, and Jiahuan Pei. 2024. Melora: Mini-ensemble low-rank adapters for parameter-efficient fine-tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pa… [find](https://scholar.google.com/scholar?q=Pengjie+Ren%2C+Chengshun+Shi%2C+Shiguang+Wu%2C+Mengqi+Zhang%2C+Zhaochun+Ren%2C+Maarten+de+Rijke%2C+Zhumin+Chen%2C+and+Jiahuan+Pei%2E+2024%2E+Melora%3A+Mini%2Densemble+low%2Drank+adapters+for+parameter%2Defficient+fine%2Dtuning%2E+)
30. Xinyi Shang, Yang Lu, Gang Huang, and Hanzi Wang. 2022. Federated learning on heterogeneous and long-tailed data via classifier re-training with federated features. In Proceedings of the 31st International Joint Conference on Artificial Intelligence (IJCAI), pages 2218–2224. [find](https://scholar.google.com/scholar?q=Xinyi+Shang%2C+Yang+Lu%2C+Gang+Huang%2C+and+Hanzi+Wang%2E+2022%2E+Federated+learning+on+heterogeneous+and+long%2Dtailed+data+via+classifier+re%2Dtraining+with+federated+features%2E+In+Proceedings+of+the+31st+Internat)
31. Raghav Singhal, Kaustubh Ponkshe, and Praneeth Vepakomma. 2025. Fedex-lora: Exact aggregation for federated and efficient fine-tuning of large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), pages 1316–1336. learning. In Proceedings… [find](https://scholar.google.com/scholar?q=Raghav+Singhal%2C+Kaustubh+Ponkshe%2C+and+Praneeth+Vepakomma%2E+2025%2E+Fedex%2Dlora%3A+Exact+aggregation+for+federated+and+efficient+fine%2Dtuning+of+large+language+models%2E+In+Proceedings+of+the+63rd+Annual+Meetin)
32. Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, Xin He, Bo Han, and Xiaowen Chu. 2022. Virtual homogeneity learning: Defending against data heterogeneity in federated learning. In Proceedings of the 39th International Conference on Machine Learning (ICML), pages 21111–21132. [find](https://scholar.google.com/scholar?q=Zhenheng+Tang%2C+Yonggang+Zhang%2C+Shaohuai+Shi%2C+Xin+He%2C+Bo+Han%2C+and+Xiaowen+Chu%2E+2022%2E+Virtual+homogeneity+learning%3A+Defending+against+data+heterogeneity+in+federated+learning%2E+In+Proceedings+of+the+39th)
33. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and 1 others. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Morgane+Riviere%2C+Shreya+Pathak%2C+Pier+Giuseppe+Sessa%2C+Cassidy+Hardin%2C+Surya+Bhupatiraju%2C+L%C3%A9onard+Hussenot%2C+Thomas+Mesnard%2C+Bobak+Shahriari%2C+Alexandre+Ram%C3%A9%2C+and+1+others%2E+2024%2E+Gemma+2%3A+Impr)
34. Madeleine Udell and Alex Townsend. 2019. Why are big data matrices approximately low rank? SIAM Journal on Mathematics of Data Science, 1(1):144– 160. [find](https://scholar.google.com/scholar?q=Madeleine+Udell+and+Alex+Townsend%2E+2019%2E+Why+are+big+data+matrices+approximately+low+rank%3F+SIAM+Journal+on+Mathematics+of+Data+Science%2C+1%281%29%3A144%E2%80%93+160%2E)
35. Yiding Wang, Decang Sun, Kai Chen, Fan Lai, and Mosharaf Chowdhury. 2023. Egeria: Efficient DNN training with knowledge-guided layer freezing. In Proceedings of the 18th European Conference on Computer Systems (EuroSys), pages 851–866. [find](https://scholar.google.com/scholar?q=Yiding+Wang%2C+Decang+Sun%2C+Kai+Chen%2C+Fan+Lai%2C+and+Mosharaf+Chowdhury%2E+2023%2E+Egeria%3A+Efficient+DNN+training+with+knowledge%2Dguided+layer+freezing%2E+In+Proceedings+of+the+18th+European+Conference+on+Compute)
36. Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. 2024. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), volume 37, pages 22513– 22533. [find](https://scholar.google.com/scholar?q=Ziyao+Wang%2C+Zheyu+Shen%2C+Yexiao+He%2C+Guoheng+Sun%2C+Hongyi+Wang%2C+Lingjuan+Lyu%2C+and+Ang+Li%2E+2024%2E+Flora%3A+Federated+fine%2Dtuning+large+language+models+with+heterogeneous+low%2Drank+adaptations%2E+In+Proceedings+)
37. Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023a. Adaptive budget allocation for parameter-efficient fine-tuning. In Proceedings of the 11th International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Qingru+Zhang%2C+Minshuo+Chen%2C+Alexander+Bukharin%2C+Pengcheng+He%2C+Yu+Cheng%2C+Weizhu+Chen%2C+and+Tuo+Zhao%2E+2023a%2E+Adaptive+budget+allocation+for+parameter%2Defficient+fine%2Dtuning%2E+In+Proceedings+of+the+11th+Int)
38. Ruiyi Zhang, Rushi Qiang, Sai Ashish Somayajula, and Pengtao Xie. 2024. Autolora: Automatically tuning matrix ranks in low-rank adaptation based on meta learning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech… [find](https://scholar.google.com/scholar?q=Ruiyi+Zhang%2C+Rushi+Qiang%2C+Sai+Ashish+Somayajula%2C+and+Pengtao+Xie%2E+2024%2E+Autolora%3A+Automatically+tuning+matrix+ranks+in+low%2Drank+adaptation+based+on+meta+learning%2E+In+Proceedings+of+the+2024+Conference)
39. Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. [Character-level convolutional networks for text classification](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/). In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), volume 28, pages 649–657.
40. Zhuo Zhang, Yuanhang Yang, Yong Dai, Qifan Wang, Yue Yu, Lizhen Qu, and Zenglin Xu. 2023b. Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models. In Findings of the Association for Computational Linguistics (ACL), pages 9963–9977. [find](https://scholar.google.com/scholar?q=Zhuo+Zhang%2C+Yuanhang+Yang%2C+Yong+Dai%2C+Qifan+Wang%2C+Yue+Yu%2C+Lizhen+Qu%2C+and+Zenglin+Xu%2E+2023b%2E+Fedpetuning%3A+When+federated+learning+meets+the+parameter%2Defficient+tuning+methods+of+pre%2Dtrained+language+mod)
41. Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. 2024. Improving lora in privacy-preserving federated [find](https://scholar.google.com/scholar?q=Youbang+Sun%2C+Zitao+Li%2C+Yaliang+Li%2C+and+Bolin+Ding%2E+2024%2E+Improving+lora+in+privacy%2Dpreserving+federated)

*Updated 1 November 2026*