--- title: "HMoE: Heterogeneous Mixture of Experts for Language Modeling" description: "They propose HMoE, in which the FFN experts in a layer have different hidden dimensions and therefore… 1 reference, cited by 9 papers here." authors: ["An Wang", "Xingwu Sun", "Ruobing Xie", "Shuaipeng Li", "Jiaqi Zhu", "Zhen Yang", "Pinxue Zhao", "Weidong Han", "Zhanhui Kang", "Di Wang", "Naoaki Okazaki", "Cheng-zhong Xu"] year: 2025 venue: "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing" source: "https://aclanthology.org/2025.emnlp-main.1115/" doi: "10.18653/v1/2025.emnlp-main.1115" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/wang-2025-hmoe-heterogeneous-mixture-experts-language/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2025-hmoe-heterogeneous-mixture-experts-language/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/8/) / HMoE: Heterogeneous Mixture of Experts for Language Modeling # HMoE: Heterogeneous Mixture of Experts for Language Modeling **Authors:** An Wang, [Xingwu Sun](https://md.research.elamur.ai/free/en/authors/xingwu-sun/), [Ruobing Xie](https://md.research.elamur.ai/free/en/authors/ruobing-xie/), [Shuaipeng Li](https://md.research.elamur.ai/free/en/authors/shuaipeng-li/), [Jiaqi Zhu](https://md.research.elamur.ai/free/en/authors/jiaqi-zhu/), [Zhen Yang](https://md.research.elamur.ai/free/en/authors/zhen-yang/), Pinxue Zhao, Weidong Han, [Zhanhui Kang](https://md.research.elamur.ai/free/en/authors/zhanhui-kang/), [Di Wang](https://md.research.elamur.ai/free/en/authors/di-wang/), [Naoaki Okazaki](https://md.research.elamur.ai/free/en/authors/naoaki-okazaki/) and Cheng-zhong Xu · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2025](https://md.research.elamur.ai/free/en/venues/emnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2025.emnlp-main.1115/) · [PDF](https://aclanthology.org/2025.emnlp-main.1115.pdf) · [DOI 10.18653/v1/2025.emnlp-main.1115](https://doi.org/10.18653/v1/2025.emnlp-main.1115) · [OpenAlex](https://openalex.org/W4416035996) **Citations:** 2 citations (OpenAlex, 2026-10-14) · [9](https://md.research.elamur.ai/free/en/papers/wang-2025-hmoe-heterogeneous-mixture-experts-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Standard Mixture of Experts (MoE) language models use experts of identical size, even though some tokens are harder to process than others. The authors argue this uniformity limits how much experts specialize and wastes parameters. They propose HMoE, in which the FFN experts in a layer have different hidden dimensions and therefore different capacities. They also tested a naive version with no auxiliary objective, and it did not beat homogeneous MoE: routing favored the large experts and left the small ones mostly unused. To fix this, they add the P-Penalty Loss, a training term that pushes the router toward smaller experts. They evaluate HMoE with both Top-K and Top-P routing and compare several schemes for choosing expert sizes. In the reported pre-training experiments, HMoE reaches lower loss than conventional MoE while activating fewer parameters, and it scores better on downstream benchmarks. The work matters because it shows expert size is a useful design dimension for sparse LLMs, giving better accuracy at lower compute. ## Key points - Experts follow the LLaMa-style SiLU-gated FFN. Heterogeneity comes only from varying each expert's hidden size, because mixing different expert architectures made training unstable. - The P-Penalty Loss promotes activation of small experts and outperforms the standard load balancing loss for HMoE. Without guidance, HMoE (e.g., relative sizes {9, 11, …, 23} for experts a–h) underperforms homogeneous MoE. - Among expert-size distributions, a hybrid mix of similar and varied sizes worked best, ahead of arithmetic. Geometric was worst because its smallest experts were too weak and seldom chosen. - Wasserstein-distance heatmaps show more differentiated experts in HMoE than in homogeneous MoE. In HMoE, experts of similar size form clusters, while homogeneous experts split into roughly two groups. - Rare or ambiguous tokens are routed more often to larger experts. Code is available at github.com/AnWang-AI/HMoE (EMNLP 2025, Tencent Hunyuan et al.). *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Tencent](https://md.research.elamur.ai/free/en/companies/tencent/) — Company, Shenzhen, China · 163 works · 21,032 citations - [University of Macau](https://md.research.elamur.ai/free/en/universities/university-of-macau/) — University, Macao, China · 43 works · 2,368 citations - [Institute of Science Tokyo](https://md.research.elamur.ai/free/en/universities/institute-of-science-tokyo/) — University, Tokyo, Japan · 14 works · 20 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works 🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works ## Cite Cite as: An Wang, Xingwu Sun, Ruobing Xie, Shuaipeng Li, Jiaqi Zhu, Zhen Yang, Pinxue Zhao, Weidong Han, Zhanhui Kang, Di Wang, Naoaki Okazaki and Cheng-zhong Xu. 2025. HMoE: Heterogeneous Mixture of Experts for Language Modeling. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{wang2025hmoe, title = {{HMoE: Heterogeneous Mixture of Experts for Language Modeling}}, author = {An Wang, Xingwu Sun, Ruobing Xie, Shuaipeng Li, Jiaqi Zhu, Zhen Yang, Pinxue Zhao, Weidong Han, Zhanhui Kang, Di Wang, Naoaki Okazaki and Cheng-zhong Xu}, booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing}, year = {2025}, doi = {10.18653/v1/2025.emnlp-main.1115}, url = {https://doi.org/10.18653/v1/2025.emnlp-main.1115} } ``` - [← previous Hit the Sweet Spot! Span-Level Ensemble for Large Language Models](https://md.research.elamur.ai/free/en/papers/xu-2025-hit-sweet-spot-span-level/) - [next → How well can LLMs Grade Essays in Arabic?](https://md.research.elamur.ai/free/en/papers/ghazawi-2025-well-llms-grade-essays-arabic/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/8/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work. 1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774. \#\# Conclusion *Updated 2 November 2026*