--- title: "Group-Merger — paper by yi et al. (2026) — summary and ideas" description: "The paper addresses multilingual continual learning (MCL). 3 references. Among them: 0." authors: ["Weijian yi", "Hongliang Li", "Jinan Xu"] year: 2026 venue: "Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026)" source: "https://aclanthology.org/2026.mellm-1.30/" doi: "10.18653/v1/2026.mellm-1.30" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/yi-2026-group-merger-lora-based-framework/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yi-2026-group-merger-lora-based-framework/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/35/) / Group-Merger: A LoRA-based Framework for Multilingual Continual Learning # Group-Merger: A LoRA-based Framework for Multilingual Continual Learning **Authors:** Weijian yi, [Hongliang Li](https://md.research.elamur.ai/free/en/authors/hongliang-li/) and [Jinan Xu](https://md.research.elamur.ai/free/en/authors/jinan-xu/) · **Year:** 2026 · **Venue:** [Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026)](https://md.research.elamur.ai/free/en/venues/mellm/) ([Workshop on Multilinguality in the Era of Large Language Models 2026](https://md.research.elamur.ai/free/en/venues/mellm/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.mellm-1.30/) · [PDF](https://aclanthology.org/2026.mellm-1.30.pdf) · [DOI 10.18653/v1/2026.mellm-1.30](https://doi.org/10.18653/v1/2026.mellm-1.30) · [OpenAlex](https://openalex.org/W7166819263) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary The paper addresses multilingual continual learning (MCL). In this setting, a language model receives new language-specific tasks one after another. It must learn them without forgetting earlier ones and without replaying old data. Assigning a separate LoRA module to each task prevents forgetting, but the number of parameters grows with every task and the modules cannot learn from each other. Merging all modules into one avoids the growth but causes interference that worsens as tasks accumulate. Group-Merger takes a middle path. Incoming tasks are placed into groups with a fixed capacity n. Once a group is full, its LoRA modules are merged with SVD into a single module. To limit interference, training adds an orthogonality penalty only among tasks in the same group. The strength of that penalty is scaled by how similar the tasks are, using sentence-embedding similarity, so that related languages can share knowledge. Modules from earlier groups also take part in the forward pass. On the XNLI and MLQA benchmarks, the framework outperforms existing LoRA-based continual learning baselines while storing fewer extra modules. ## Key points - Storage stays bounded: the number of extra modules at inference equals ⌈T/n⌉ for T tasks and group size n. No historical data needs to be replayed. - Average performance improves over prior state-of-the-art by 1.75% on XNLI. The gain on MLQA is only 0.02%. - In the Llama-3B ablation, removing previous-module incorporation lowers XNLI AP from 74.23 to 63.68 and raises forgetting from 1.51 to 11.82. Task-aware orthogonality outperforms plain orthogonality. - Group size is a trade-off between accuracy and storage. As the number of extra modules goes from 1 to 2 to 3, XNLI AP rises from 72.25 to 74.23 to 76.52. - The paper's own merging method outperforms Weight Average, Task Arithmetic and Ties-Merging. For example, it reaches 77.05 on XNLI at n=2, compared with 68.59 for Ties-Merging. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Beijing Jiaotong University](https://md.research.elamur.ai/free/en/universities/beijing-jiaotong-university/) — University, Beijing, China · 28 works · 1,754 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works 🧭 Topics (1): [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 882 works ## Cite Cite as: Weijian yi, Hongliang Li and Jinan Xu. 2026. Group-Merger: A LoRA-based Framework for Multilingual Continual Learning. Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{yi2026group, title = {{Group-Merger: A LoRA-based Framework for Multilingual Continual Learning}}, author = {Weijian yi, Hongliang Li and Jinan Xu}, booktitle = {Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026)}, year = {2026}, doi = {10.18653/v1/2026.mellm-1.30}, url = {https://doi.org/10.18653/v1/2026.mellm-1.30} } ``` - [← previous Grounding Sign Language Representation Learning in Phonology](https://md.research.elamur.ai/free/en/papers/vandendriessche-2026-grounding-sign-language-representation-learning/) - [next → GroupRank: A Groupwise Paradigm for Effective and Efficient Passage Reranking with LLMs](https://md.research.elamur.ai/free/en/papers/long-2026-grouprank-groupwise-paradigm-effective-efficient/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/35/) ## References 3 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Qwen3-1.7B Llama3-3B Llama3-8B Qwen3-8B 73.66 74.23 79.35 79.43 60.31 62.71 64.28 63.41 [find](https://scholar.google.com/scholar?q=Qwen3%2D1%2E7B+Llama3%2D3B+Llama3%2D8B+Qwen3%2D8B+73%2E66+74%2E23+79%2E35+79%2E43+60%2E31+62%2E71+64%2E28+63%2E41) 2. Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marcaurelio Ranzato. 2019. On tiny episodic memories in continual learning. arxiv. Learning, 6(7). *Table 5: The results of the XNLI and MLQA benchmarks with different architectures an… [find](https://scholar.google.com/scholar?q=Arslan+Chaudhry%2C+Marcus+Rohrbach%2C+Mohamed+Elhoseiny%2C+Thalaiyasingam+Ajanthan%2C+Puneet+K+Dokania%2C+Philip+HS+Torr%2C+and+Marcaurelio+Ranzato%2E+2019%2E+On+tiny+episodic+memories+in+continual+learning%2E+arxiv%2E+L) 3. Rajas Chitale, Ankit Vaidya, Aditya Kane, and Archana Ghotkar. 2023. Task arithmetic with lora for continual learning. arXiv preprint arXiv:2311.02428. \#\# Conclusion In this paper, we introduce a novel framework Group-Merger that addresses catastrophic forgetting in incrementally learning new langua… — [arXiv:2311.02428](https://arxiv.org/abs/2311.02428) [find](https://scholar.google.com/scholar?q=Rajas+Chitale%2C+Ankit+Vaidya%2C+Aditya+Kane%2C+and+Archana+Ghotkar%2E+2023%2E+Task+arithmetic+with+lora+for+continual+learning%2E+arXiv+preprint+arXiv%3A2311%2E02428%2E+%5C%23%5C%23+Conclusion+In+this+paper%2C+we+introduce+a+no) *Updated 2 November 2026*