---
title: "Less, but Better — paper by Zhang et al. (2025)"
description: "This paper introduces LayerMoE, which questions whether every layer needs the same number of… 38 references, cited by 12 papers here."
authors: ["Xue Zhang", "Yunlong Liang", "Fandong Meng", "Songming Zhang", "Yufeng Chen", "Jinan Xu", "Jie Zhou"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.acl-long.878/"
doi: "10.18653/v1/2025.acl-long.878"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2025-less-but-better-efficient-multilingual/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2025-less-but-better-efficient-multilingual/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/) / Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts

# Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts

**Authors:** [Xue Zhang](https://md.research.elamur.ai/free/en/authors/xue-zhang/), [Yunlong Liang](https://md.research.elamur.ai/free/en/authors/yunlong-liang/), [Fandong Meng](https://md.research.elamur.ai/free/en/authors/fandong-meng/), [Songming Zhang](https://md.research.elamur.ai/free/en/authors/songming-zhang/), [Yufeng Chen](https://md.research.elamur.ai/free/en/authors/yufeng-chen/), [Jinan Xu](https://md.research.elamur.ai/free/en/authors/jinan-xu/) and [Jie Zhou](https://md.research.elamur.ai/free/en/authors/jie-zhou/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.878/) · [PDF](https://aclanthology.org/2025.acl-long.878.pdf) · [DOI 10.18653/v1/2025.acl-long.878](https://doi.org/10.18653/v1/2025.acl-long.878) · [OpenAlex](https://openalex.org/W4412944946)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [12](https://md.research.elamur.ai/free/en/papers/zhang-2025-less-but-better-efficient-multilingual/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Adding new languages to an existing LLM risks erasing skills in the languages it already handles well. The earlier MoE-based approach, MoE-LPR, also inflates model size: Qwen1.5-1.8B grows 3.2× to support three extra languages. This paper introduces LayerMoE, which questions whether every layer needs the same number of new experts. The authors measure cosine similarity between post-attention hidden states of different languages, sampled from CulturalX, and find that it varies sharply across layers. LayerMoE then gives more new experts to layers where languages diverge and fewer to layers where representations look language-agnostic. In the high-similarity layers, a classifier placed before the router helps send old-language tokens back to the original frozen expert. LayerMoE outperforms MoE-LPR with 60% fewer experts in single expansion and 33.3% fewer in lifelong (sequential) expansion. It learns new languages better and forgets old ones less. This makes ongoing multilingual growth cheaper, which matters for languages poorly served by current LLMs.

## Key points
- The baseline, MoE-LPR, trains in two stages: frozen dense experts plus new FFN experts with top-2 routing, followed by router-only "review" training with a language-priors routing loss.
- The similarity analysis uses Qwen1.5-1.8B and 100,000 sampled tokens per language. Similarity is higher in the middle and final layers and lower in layers 0–4 and 17–21.
- Evaluation covers ARC-C, MMLU, Hellaswag and Belebele. For Llama-3.2-3B, FLORES translation scored with COMET is added.
- On Llama-3.2-3B (G0→G1), LayerMoE reaches an average of 59.64, compared with 59.47 for MoE-LPR (3*24) and 57.79 for the base model. Its new-language average is 56.47.
- Adding router classifiers to the 7 most-similar layers works best; random or least-similar layers preserve old languages poorly. Code is available at github.com/XZhang00/LayerMoE.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works

## Cite

Cite as: Xue Zhang, Yunlong Liang, Fandong Meng, Songming Zhang, Yufeng Chen, Jinan Xu and Jie Zhou. 2025. Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2025less,
  title = {{Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts}},
  author = {Xue Zhang, Yunlong Liang, Fandong Meng, Songming Zhang, Yufeng Chen, Jinan Xu and Jie Zhou},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.878},
  url = {https://doi.org/10.18653/v1/2025.acl-long.878}
}
```

- [← previous Learning to Refuse: Towards Mitigating Privacy Risks in LLMs](https://md.research.elamur.ai/free/en/papers/liu-2025-learning-refuse-mitigating-privacy-risks/)
- [next → LoGU: Long-form Generation with Uncertainty Expressions](https://md.research.elamur.ai/free/en/papers/yang-2025-logu-long-form-generation-uncertainty/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/)

## References

38 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. 2023. [Do all languages cost the same?](https://md.research.elamur.ai/free/en/papers/ai-2024-yi-open-foundation-models-01/) tokenization in the era of commercial language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9904…
2. Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, Charvi Jain, Alexander Weber, Lena Jurkschat, Hammam Abdelwahab, Chelsea John, Pedro Ortiz Suarez, Malte Ostendorff, Samuel Weinbach, Rafet Sifa, Ste… — [Tokenizer Choice For LLM Training: Negligible or Crucial?](https://md.research.elamur.ai/free/en/papers/ali-2024-tokenizer-choice-llm-training-negligible/)
3. Catherine Arnett and Benjamin K. Bergen. 2024. [Why do language models perform worse for morphologically complex languages?](https://md.research.elamur.ai/free/en/papers/arnett-2025-why-do-language-models-perform/) Preprint, arXiv:2411.14198.
4. Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. [The belebele benchmark: a parallel reading comprehension dataset in 122 language variants](https://md.research.elamur.ai/free/en/papers/bandarkar-2023-belebele-benchmark-parallel-reading-comprehension/). In Proceedings of the 62nd Annual M…
5. Yuheng Chen, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. 2023. Journey to the center of the knowledge neurons: Discoveries of language-independent knowledge neurons and degenerate knowledge neurons. Preprint, arXiv:2308.13198. — [arXiv:2308.13198](https://arxiv.org/abs/2308.13198) [find](https://scholar.google.com/scholar?q=Yuheng+Chen%2C+Pengfei+Cao%2C+Yubo+Chen%2C+Kang+Liu%2C+and+Jun+Zhao%2E+2023%2E+Journey+to+the+center+of+the+knowledge+neurons%3A+Discoveries+of+language%2Dindependent+knowledge+neurons+and+degenerate+knowledge+neuron)
6. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). Preprint, arXiv:1803.05457.
7. Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. Loramoe: Alleviate world knowledge forgetting in large language models via moe-style plugin. Preprint, arX… — [arXiv:2312.09979](https://arxiv.org/abs/2312.09979) [find](https://scholar.google.com/scholar?q=Shihan+Dou%2C+Enyu+Zhou%2C+Yan+Liu%2C+Songyang+Gao%2C+Jun+Zhao%2C+Wei+Shen%2C+Yuhao+Zhou%2C+Zhiheng+Xi%2C+Xiao+Wang%2C+Xiaoran+Fan%2C+Shiliang+Pu%2C+Jiang+Zhu%2C+Rui+Zheng%2C+Tao+Gui%2C+Qi+Zhang%2C+and+Xuanjing+Huang%2E+2024%2E+Loramo)
8. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process… [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2E+2019%2E+How+contextual+are+contextualized+word+representations%3F+Comparing+the+geometry+of+BERT%2C+ELMo%2C+and+GPT%2D2+embeddings%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+i)
9. William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Preprint, arXiv:2101.03961. [find](https://scholar.google.com/scholar?q=William+Fedus%2C+Barret+Zoph%2C+and+Noam+Shazeer%2E+2022%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+Preprint%2C+arXiv%3A2101%2E03961%2E)
10. Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, and VS Subrahmanian. 2024a. Higher layers need more lora experts. Preprint, arXiv:2402.08562. — [arXiv:2402.08562](https://arxiv.org/abs/2402.08562) [find](https://scholar.google.com/scholar?q=Chongyang+Gao%2C+Kezhen+Chen%2C+Jinmeng+Rao%2C+Baochen+Sun%2C+Ruibo+Liu%2C+Daiyi+Peng%2C+Yawen+Zhang%2C+Xiaoyuan+Guo%2C+Jie+Yang%2C+and+VS+Subrahmanian%2E+2024a%2E+Higher+layers+need+more+lora+experts%2E+Preprint%2C+arXiv%3A2402)
11. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Ani… [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Alain+Le+Noac%E2%80%99h%2C+Haonan+Li%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Chris+Oci)
12. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). Preprint, arXiv:2407.21783.
13. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). Preprint, arXiv:2009.03300.
14. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). Preprint, arXiv:2106.09685.
15. Kaiyu Huang, Fengran Mo, Hongliang Li, You Li, Yuanchi Zhang, Weijian Yi, Yulong Mao, Jinchen Liu, Yuzhuang Xu, Jinan Xu, Jian-Yun Nie, and Yang Liu. 2024. A survey on large language models with multilingualism: Recent advances and new frontiers. Preprint, arXiv:2405.10936. — [arXiv:2405.10936](https://arxiv.org/abs/2405.10936) [find](https://scholar.google.com/scholar?q=Kaiyu+Huang%2C+Fengran+Mo%2C+Hongliang+Li%2C+You+Li%2C+Yuanchi+Zhang%2C+Weijian+Yi%2C+Yulong+Mao%2C+Jinchen+Liu%2C+Yuzhuang+Xu%2C+Jinan+Xu%2C+Jian%2DYun+Nie%2C+and+Yang+Liu%2E+2024%2E+A+survey+on+large+language+models+with+multi)
16. Kaiyu Huang, Fengran Mo, Xinyu Zhang, Hongliang Li, You Li, Yuanchi Zhang, Weijian Yi, Yulong Mao, Jinchen Liu, Yuzhuang Xu, Jinan Xu, Jian-Yun Nie, and Yang Liu. 2025. A survey on large language models with multilingualism: Recent advances and new frontiers. Preprint, arXiv:2405.10936. — [arXiv:2405.10936](https://arxiv.org/abs/2405.10936) [find](https://scholar.google.com/scholar?q=Kaiyu+Huang%2C+Fengran+Mo%2C+Xinyu+Zhang%2C+Hongliang+Li%2C+You+Li%2C+Yuanchi+Zhang%2C+Weijian+Yi%2C+Yulong+Mao%2C+Jinchen+Liu%2C+Yuzhuang+Xu%2C+Jinan+Xu%2C+Jian%2DYun+Nie%2C+and+Yang+Liu%2E+2025%2E+A+survey+on+large+language+mode)
17. Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. 2023a. [Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback](https://md.research.elamur.ai/free/en/papers/lai-2023-okapi-instruction-tuned-large-language/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Langua…
18. Viet Dac Lai, Nghia Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023b. [ChatGPT beyond English: Towards a comprehensive evaluation of large language models in multilingual learning](https://md.research.elamur.ai/free/en/papers/lai-2023-chatgpt-beyond-english-comprehensive-evaluation/). In Findings of the Association for Computational Linguistics: EMNLP 202…
19. Tianhao Li, Shangjie Li, Binbin Xie, Deyi Xiong, and Baosong Yang. 2024. Moe-ct: A novel approach for large language models training with resistance to catastrophic forgetting. Preprint, arXiv:2407.00875. — [arXiv:2407.00875](https://arxiv.org/abs/2407.00875) [find](https://scholar.google.com/scholar?q=Tianhao+Li%2C+Shangjie+Li%2C+Binbin+Xie%2C+Deyi+Xiong%2C+and+Baosong+Yang%2E+2024%2E+Moe%2Dct%3A+A+novel+approach+for+large+language+models+training+with+resistance+to+catastrophic+forgetting%2E+Preprint%2C+arXiv%3A2407%2E00)
20. Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2024a. [CulturaX: A cleaned, enormous, and multilingual dataset for large language models in 167 languages](https://md.research.elamur.ai/free/en/papers/nguyen-2023-culturax-cleaned-enormous-multilingual-dataset/). In Proceedings of the 2024 Joint International Conference on Co…
21. Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Zhiqiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li, Jianyu Wang, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, and Lidong Bing. 2024b. [Seallms – large language models for southeast asia](https://md.research.elamur.ai/free/en/papers/nguyen-2023-seallms-large-language-models-southeast/). Preprint, arX…
22. Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, and Adel Bibi. 2023. [Language model tokenizers introduce unfairness between languages](https://md.research.elamur.ai/free/en/papers/petrov-2023-language-model-tokenizers-introduce-unfairness/). Preprint, arXiv:2305.15425.
23. Peijun Qing, Chongyang Gao, Yefan Zhou, Xingjian Diao, Yaoqing Yang, and Soroush Vosoughi. 2024. AlphaLoRA: Assigning LoRA experts based on layer training quality. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20511–20523, Miami, Florida, USA. Assoc… [find](https://scholar.google.com/scholar?q=Peijun+Qing%2C+Chongyang+Gao%2C+Yefan+Zhou%2C+Xingjian+Diao%2C+Yaoqing+Yang%2C+and+Soroush+Vosoughi%2E+2024%2E+AlphaLoRA%3A+Assigning+LoRA+experts+based+on+layer+training+quality%2E+In+Proceedings+of+the+2024+Conferenc)
24. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. [COMET: A neural framework for MT evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.
25. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. [How good is your tokenizer? on the monolingual performance of multilingual language models](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Join…
26. Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. [find](https://scholar.google.com/scholar?q=Daria+Soboleva%2C+Faisal+Al%2DKhateeb%2C+Robert+Myers%2C+Jacob+R+Steeves%2C+Joel+Hestness%2C+and+Nolan+Dey%2E+2023%2E+SlimPajama%3A+A+627B+token+cleaned+and+deduplicated+version+of+RedPajama%2E)
27. Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. 2024. [Language-specific neurons: The key to multilingual capabilities in large language models](https://md.research.elamur.ai/free/en/papers/tang-2024-language-specific-neurons-key-multilingual/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volu…
28. NLLB Team. 2022. No language left behind: Scaling human-centered machine translation. [find](https://scholar.google.com/scholar?q=NLLB+Team%2E+2022%2E+No+language+left+behind%3A+Scaling+human%2Dcentered+machine+translation%2E)
29. Qwen Team. 2024. Introducing qwen1.5. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+2024%2E+Introducing+qwen1%2E5%2E)
30. Ahmet Üstün, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. Aya model: An instruction fine-tun… [find](https://scholar.google.com/scholar?q=Ahmet+%C3%9Cst%C3%BCn%2C+Viraat+Aryabumi%2C+Zheng+Yong%2C+Wei%2DYin+Ko%2C+Daniel+D%E2%80%99souza%2C+Gbemileke+Onilude%2C+Neel+Bhandari%2C+Shivalika+Singh%2C+Hui%2DLee+Ooi%2C+Amr+Kayid%2C+Freddie+Vargus%2C+Phil+Blunsom%2C+Shayne+Longpre%2C+Niklas+Mu)
31. Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, Chenxia Li, Liu Yang, Xilin Luo, Xuejie Wu, Lunan Liu, Wenjun Cheng, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Lei Lin, Xiaokun Wang, Yutuan Ma, Chuanhai Dong, Yanqi Sun, Yifu Chen, Yongy… — [Skywork: A More Open Bilingual Foundation Model](https://md.research.elamur.ai/free/en/papers/wei-2023-skywork-more-open-bilingual/)
32. Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, Tianxiang Hu, Shangjie Li, Binyuan Hui, Bowen Yu, Dayiheng Liu, Baosong Yang, Fei Huang, and Jun Xie. 2023b. Polylm: An open source polyglot large language model. Preprint, arXiv:2307.0… [find](https://scholar.google.com/scholar?q=Xiangpeng+Wei%2C+Haoran+Wei%2C+Huan+Lin%2C+Tianhao+Li%2C+Pei+Zhang%2C+Xingzhang+Ren%2C+Mei+Li%2C+Yu+Wan%2C+Zhiwei+Cao%2C+Binbin+Xie%2C+Tianxiang+Hu%2C+Shangjie+Li%2C+Binyuan+Hui%2C+Bowen+Yu%2C+Dayiheng+Liu%2C+Baosong+Yang%2C+Fei+Hua)
33. Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ying Shan, and Ping Luo. 2024. Llama pro: Progressive llama with block expansion. Preprint, arXiv:2401.02415. [find](https://scholar.google.com/scholar?q=Chengyue+Wu%2C+Yukang+Gan%2C+Yixiao+Ge%2C+Zeyu+Lu%2C+Jiahao+Wang%2C+Ye+Feng%2C+Ying+Shan%2C+and+Ping+Luo%2E+2024%2E+Llama+pro%3A+Progressive+llama+with+block+expansion%2E+Preprint%2C+arXiv%3A2401%2E02415%2E)
34. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+HellaSwag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Com)
35. Shaolei Zhang, Kehao Zhang, Qingkai Fang, Shoutao Guo, Yan Zhou, Xiaodong Liu, and Yang Feng. 2024a. Bayling 2: A multilingual large language model with efficient language alignment. Preprint, arXiv:2411.16300. — [arXiv:2411.16300](https://arxiv.org/abs/2411.16300) [find](https://scholar.google.com/scholar?q=Shaolei+Zhang%2C+Kehao+Zhang%2C+Qingkai+Fang%2C+Shoutao+Guo%2C+Yan+Zhou%2C+Xiaodong+Liu%2C+and+Yang+Feng%2E+2024a%2E+Bayling+2%3A+A+multilingual+large+language+model+with+efficient+language+alignment%2E+Preprint%2C+arXiv%3A2)
36. Xue Zhang, Yunlong Liang, Fandong Meng, Songming Zhang, Yufeng Chen, Jinan Xu, and Jie Zhou. 2024b. [Multilingual knowledge editing with language-agnostic factual neurons](https://md.research.elamur.ai/free/en/papers/zhang-2025-multilingual-knowledge-editing-language-agnostic/). Preprint, arXiv:2406.16416.
37. Yuanchi Zhang, Yile Wang, Zijun Liu, Shuo Wang, Xiaolong Wang, Peng Li, Maosong Sun, and Yang Liu. 2024c. [Enhancing multilingual capabilities of large language models through self-distillation from resource-rich languages](https://md.research.elamur.ai/free/en/papers/zhang-2024-enhancing-multilingual-capabilities-large-language/). In Proceedings of the 62nd Annual Meeting of the Association for Computationa…
38. Hao Zhou, Zhijun Wang, Shujian Huang, Xin Huang, Xue Han, Junlan Feng, Chao Deng, Weihua Luo, and Jiajun Chen. 2024. Moe-lpr: Multilingual extension of large language models through mixture-of-experts with language priors routing. Preprint, arXiv:2408.11396. — [arXiv:2408.11396](https://arxiv.org/abs/2408.11396) [find](https://scholar.google.com/scholar?q=Hao+Zhou%2C+Zhijun+Wang%2C+Shujian+Huang%2C+Xin+Huang%2C+Xue+Han%2C+Junlan+Feng%2C+Chao+Deng%2C+Weihua+Luo%2C+and+Jiajun+Chen%2E+2024%2E+Moe%2Dlpr%3A+Multilingual+extension+of+large+language+models+through+mixture%2Dof%2Dexperts)

*Updated 30 October 2026*