---
title: "The Illusion of Specialization — paper by Wang et al. (2026)"
description: "The framework builds domain-conditioned routing profiles using an Expert Contribution Index (ECI), defined as the average… 29 references."
authors: ["Yan Wang", "Yitao Xu", "Nanhan Shen", "Jinyan Su", "Jimin Huang", "Zining Zhu"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.665/"
doi: "10.18653/v1/2026.acl-long.665"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2026-illusion-specialization-unveiling-domain-invariant/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2026-illusion-specialization-unveiling-domain-invariant/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/45/) / The Illusion of Specialization: Unveiling the Domain-Invariant “Standing Committee” in Mixture-of-Experts Models

# The Illusion of Specialization: Unveiling the Domain-Invariant “Standing Committee” in Mixture-of-Experts Models

**Authors:** [Yan Wang](https://md.research.elamur.ai/free/en/authors/yan-wang/), Yitao Xu, Nanhan Shen, [Jinyan Su](https://md.research.elamur.ai/free/en/authors/jinyan-su/), [Jimin Huang](https://md.research.elamur.ai/free/en/authors/jimin-huang/) and [Zining Zhu](https://md.research.elamur.ai/free/en/authors/zining-zhu/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.665/) · [PDF](https://aclanthology.org/2026.acl-long.665.pdf) · [DOI 10.18653/v1/2026.acl-long.665](https://doi.org/10.18653/v1/2026.acl-long.665) · [OpenAlex](https://openalex.org/W7166830732)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Mixture-of-Experts (MoE) language models are usually thought to gain efficiency by sending different domains to different specialist experts. This paper tests that belief with COMMITTEEAUDIT, a post hoc, model-agnostic framework that analyzes routing by looking at groups of experts together instead of examining each expert in isolation. The framework builds domain-conditioned routing profiles using an Expert Contribution Index (ECI), defined as the average router weight an expert receives on a domain. It then measures how much routing diverges between domains and ranks experts with Pareto-optimal criteria and stability diagnostics. The authors applied it to OLMoE, Qwen3-30B-A3B and DeepSeek-V2-Lite on MMLU, with additional evidence from C4. They found a small, domain-invariant "Standing Committee" of routed experts that takes most of the routing mass across domains, layers and top-k budgets. This holds even in models that already have shared experts. Committee members appear to support reasoning structure and syntax, while peripheral experts supply domain knowledge. The findings suggest MoE specialization is much less widespread than assumed, and that uniform load-balancing losses may work against the model's natural drift toward centralized computation.

## Key points
- Models audited: OLMoE, Qwen3-30B-A3B and DeepSeek-V2-Lite, covering both plain top-k routing and architectures with explicit shared experts. The main benchmark is MMLU, with supplementary analysis on C4.
- ECI ranks experts by expected routing weight rather than activation frequency, so it keeps information about how strongly the router prefers each expert.
- In a masking intervention on DeepSeek-V2-Lite, committee experts were suppressed and the remaining weights were re-normalized. MMLU correct rate fell from 0.39 to 0.12, 0.09 and 0.03 when masking at layers 2, 10 and 26 respectively. The share of "No Answer" outputs rose from 0.03 to 0.36–0.38.
- The authors note several limitations: only a few architectures were tested, there are no random or frequency-matched masking controls, the evaluation focuses mainly on MMLU, and training dynamics are not analyzed.
- Code is released at github.com/The-FinAI/CommitteeAudit.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

## Cite

Cite as: Yan Wang, Yitao Xu, Nanhan Shen, Jinyan Su, Jimin Huang and Zining Zhu. 2026. The Illusion of Specialization: Unveiling the Domain-Invariant “Standing Committee” in Mixture-of-Experts Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2026illusion,
  title = {{The Illusion of Specialization: Unveiling the Domain-Invariant “Standing Committee” in Mixture-of-Experts Models}},
  author = {Yan Wang, Yitao Xu, Nanhan Shen, Jinyan Su, Jimin Huang and Zining Zhu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.665},
  url = {https://doi.org/10.18653/v1/2026.acl-long.665}
}
```

- [← previous The Icelandic Language Biobank: Data Collection through a Clinical Analysis Platform](https://md.research.elamur.ai/free/en/papers/nowenstein-2026-icelandic-language-biobank-data-collection/)
- [next → The Impact of Highlighting Subjective Language on Perceived News Trustworthiness](https://md.research.elamur.ai/free/en/papers/shokri-2026-impact-highlighting-subjective-language-perceived/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/45/)

## References

29 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jun Bai, Minghao Tong, Yang Liu, Zixia Jia, and Zilong Zheng. 2025. Understanding and leveraging the expert specialization of context faithfulness in mixture-of-experts llms. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 21938–21953. [find](https://scholar.google.com/scholar?q=Jun+Bai%2C+Minghao+Tong%2C+Yang+Liu%2C+Zixia+Jia%2C+and+Zilong+Zheng%2E+2025%2E+Understanding+and+leveraging+the+expert+specialization+of+context+faithfulness+in+mixture%2Dof%2Dexperts+llms%2E+In+Proceedings+of+the+202)
2. Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, and 1 others. 2022. On the representation collapse of sparse mixture of experts. Advances in Neural Information Processing Systems, 35:34600–34613. [find](https://scholar.google.com/scholar?q=Zewen+Chi%2C+Li+Dong%2C+Shaohan+Huang%2C+Damai+Dai%2C+Shuming+Ma%2C+Barun+Patra%2C+Saksham+Singhal%2C+Payal+Bajaj%2C+Xia+Song%2C+Xian%2DLing+Mao%2C+and+1+others%2E+2022%2E+On+the+representation+collapse+of+sparse+mixture+of+ex)
3. Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, and 1 others. 2024. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066. [find](https://scholar.google.com/scholar?q=Damai+Dai%2C+Chengqi+Deng%2C+Chenggang+Zhao%2C+RX+Xu%2C+Huazuo+Gao%2C+Deli+Chen%2C+Jiashi+Li%2C+Wangding+Zeng%2C+Xingkai+Yu%2C+Yu+Wu%2C+and+1+others%2E+2024%2E+Deepseekmoe%3A+Towards+ultimate+expert+specialization+in+mixture%2Do)
4. DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, and 138 others. 2024. Deepseek-v2: A strong, economical, and efficient… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Aixin+Liu%2C+Bei+Feng%2C+Bin+Wang%2C+Bingxuan+Wang%2C+Bo+Liu%2C+Chenggang+Zhao%2C+Chengqi+Dengr%2C+Chong+Ruan%2C+Damai+Dai%2C+Daya+Guo%2C+Dejian+Yang%2C+Deli+Chen%2C+Dongjie+Ji%2C+Erhang+Li%2C+Fangyun+Lin%2C+Fuli+Luo%2C)
5. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025. Deepseek-v3 technical report. Preprint, arXiv:2… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+Damai+Dai%2C+Daya+Guo%2C+Dejian+Yang%2C+Deli+Chen%2C+Dongjie+Ji%2C+Erhan)
6. Ka Man Lo, Zeyu Huang, Zihan Qiu, Zili Wang, and Jie Fu. 2025. A closer look into mixture-of-experts in large language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4427–4447. [find](https://scholar.google.com/scholar?q=Ka+Man+Lo%2C+Zeyu+Huang%2C+Zihan+Qiu%2C+Zili+Wang%2C+and+Jie+Fu%2E+2025%2E+A+closer+look+into+mixture%2Dof%2Dexperts+in+large+language+models%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+NAACL+2025%2C)
7. Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, and 5 others. 2025. Olmoe: O… [find](https://scholar.google.com/scholar?q=Niklas+Muennighoff%2C+Luca+Soldaini%2C+Dirk+Groeneveld%2C+Kyle+Lo%2C+Jacob+Morrison%2C+Sewon+Min%2C+Weijia+Shi%2C+Pete+Walsh%2C+Oyvind+Tafjord%2C+Nathan+Lambert%2C+Yuling+Gu%2C+Shane+Arora%2C+Akshita+Bhagia%2C+Dustin+Schwenk%2C+)
8. Matthew Lyle Olson, Neale Ratzlaff, Musashi Hinck, Man Luo, Sungduk Yu, Chendi Xue, and Vasudev Lal. 2025. Probing semantic routing in large mixture-of-expert models. arXiv preprint arXiv:2502.10928. [find](https://scholar.google.com/scholar?q=Matthew+Lyle+Olson%2C+Neale+Ratzlaff%2C+Musashi+Hinck%2C+Man+Luo%2C+Sungduk+Yu%2C+Chendi+Xue%2C+and+Vasudev+Lal%2E+2025%2E+Probing+semantic+routing+in+large+mixture%2Dof%2Dexpert+models%2E+arXiv+preprint+arXiv%3A2502%2E10928%2E)
9. Giang Do, Hung Le, and Truyen Tran. 2025. Simsmoe: Toward efficient training mixture of experts via solving representational collapse. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 2012–2025. [find](https://scholar.google.com/scholar?q=Giang+Do%2C+Hung+Le%2C+and+Truyen+Tran%2E+2025%2E+Simsmoe%3A+Toward+efficient+training+mixture+of+experts+via+solving+representational+collapse%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+NAA)
10. Lingfei Qian, Weipeng Zhou, Yan Wang, Xueqing Peng, Han Yi, Yilun Zhao, Jimin Huang, Qianqian Xie, and Jian yun Nie. 2025. Fino1: On the transferability of reasoning-enhanced llms and reinforcement learning to finance. Preprint, arXiv:2502.08127. — [arXiv:2502.08127](https://arxiv.org/abs/2502.08127) [find](https://scholar.google.com/scholar?q=Lingfei+Qian%2C+Weipeng+Zhou%2C+Yan+Wang%2C+Xueqing+Peng%2C+Han+Yi%2C+Yilun+Zhao%2C+Jimin+Huang%2C+Qianqian+Xie%2C+and+Jian+yun+Nie%2E+2025%2E+Fino1%3A+On+the+transferability+of+reasoning%2Denhanced+llms+and+reinforcement+le)
11. Dongyang Fan, Bettina Messmer, and Martin Jaggi. 2024. Towards an empirical understanding of moe design choices. Preprint, arXiv:2402.13089. — [arXiv:2402.13089](https://arxiv.org/abs/2402.13089) [find](https://scholar.google.com/scholar?q=Dongyang+Fan%2C+Bettina+Messmer%2C+and+Martin+Jaggi%2E+2024%2E+Towards+an+empirical+understanding+of+moe+design+choices%2E+Preprint%2C+arXiv%3A2402%2E13089%2E+%E2%80%94+%5BarXiv%3A2402%2E13089%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2402%2E13089%29)
12. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of machine learning research, 21(140):1–67.
13. William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39. [find](https://scholar.google.com/scholar?q=William+Fedus%2C+Barret+Zoph%2C+and+Noam+Shazeer%2E+2022%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+Journal+of+Machine+Learning+Research%2C+23%28120%29%3A1%E2%80%9339%2E)
14. Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. 2024. Patchscopes: A unifying framework for inspecting hidden representations of language models. arXiv preprint arXiv:2401.06102. [find](https://scholar.google.com/scholar?q=Asma+Ghandeharioun%2C+Avi+Caciularu%2C+Adam+Pearce%2C+Lucas+Dixon%2C+and+Mor+Geva%2E+2024%2E+Patchscopes%3A+A+unifying+framework+for+inspecting+hidden+representations+of+language+models%2E+arXiv+preprint+arXiv%3A2401%2E0)
15. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300.
16. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, and 1 others. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+and+1+others%2E+2024%2E+Mi)
17. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668. [find](https://scholar.google.com/scholar?q=Dmitry+Lepikhin%2C+HyoukJoong+Lee%2C+Yuanzhong+Xu%2C+Dehao+Chen%2C+Orhan+Firat%2C+Yanping+Huang%2C+Maxim+Krikun%2C+Noam+Shazeer%2C+and+Zhifeng+Chen%2E+2020%2E+Gshard%3A+Scaling+giant+models+with+conditional+computation+and)
18. Ziyue Li and Tianyi Zhou. 2024. Your mixture-of-experts llm is secretly an embedding model for free. arXiv preprint arXiv:2410.10814. — [arXiv:2410.10814](https://arxiv.org/abs/2410.10814) [find](https://scholar.google.com/scholar?q=Ziyue+Li+and+Tianyi+Zhou%2E+2024%2E+Your+mixture%2Dof%2Dexperts+llm+is+secretly+an+embedding+model+for+free%2E+arXiv+preprint+arXiv%3A2410%2E10814%2E+%E2%80%94+%5BarXiv%3A2410%2E10814%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2410%2E10814%29)
19. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2C+Azalia+Mirhoseini%2C+Krzysztof+Maziarz%2C+Andy+Davis%2C+Quoc+Le%2C+Geoffrey+Hinton%2C+and+Jeff+Dean%2E+2017%2E+Outrageously+large+neural+networks%3A+The+sparsely%2Dgated+mixture%2Dof%2Dexperts+layer%2E+arXiv+pr)
20. Zunhai Su, Qingyuan Li, Hao Zhang, Weihao Ye, Qibo Xue, YuLei Qian, Yuchen Xie, Ngai Wong, and Kehong Yuan. 2025. Unveiling super experts in mixture-of-experts large language models. Preprint, arXiv:2507.23279. — [arXiv:2507.23279](https://arxiv.org/abs/2507.23279) [find](https://scholar.google.com/scholar?q=Zunhai+Su%2C+Qingyuan+Li%2C+Hao+Zhang%2C+Weihao+Ye%2C+Qibo+Xue%2C+YuLei+Qian%2C+Yuchen+Xie%2C+Ngai+Wong%2C+and+Kehong+Yuan%2E+2025%2E+Unveiling+super+experts+in+mixture%2Dof%2Dexperts+large+language+models%2E+Preprint%2C+arXiv%3A2)
21. Vinitra Swamy, Syrielle Montariol, Julian Blackwell, Jibril Frej, Martin Jaggi, and Tanja Käser. 2024. Intrinsic user-centric interpretability through global mixture of experts. arXiv preprint arXiv:2402.02933. — [arXiv:2402.02933](https://arxiv.org/abs/2402.02933) [find](https://scholar.google.com/scholar?q=Vinitra+Swamy%2C+Syrielle+Montariol%2C+Julian+Blackwell%2C+Jibril+Frej%2C+Martin+Jaggi%2C+and+Tanja+K%C3%A4ser%2E+2024%2E+Intrinsic+user%2Dcentric+interpretability+through+global+mixture+of+experts%2E+arXiv+preprint+arXiv%3A2)
22. Yan Wang, Yueru He, Ruoyu Xiang, and Jeff Zhao. 2025a. Rkefino1: A regulation knowledge-enhanced large language model. In 2025 IEEE 11th International Conference on Intelligent Data and Security (IDS), pages 49–51. IEEE. [find](https://scholar.google.com/scholar?q=Yan+Wang%2C+Yueru+He%2C+Ruoyu+Xiang%2C+and+Jeff+Zhao%2E+2025a%2E+Rkefino1%3A+A+regulation+knowledge%2Denhanced+large+language+model%2E+In+2025+IEEE+11th+International+Conference+on+Intelligent+Data+and+Security+%28IDS%29)
23. Yan Wang, Yang Ren, Lingfei Qian, Xueqing Peng, Keyi Wang, Yi Han, Dongji Feng, Fengran Mo, Shengyuan Lin, Qinchuan Zhang, Kaiwen He, Chenri Luo, Jianxing Chen, Junwei Wu, Jimin Huang, Guojun Xiong, Xiao-Yang Liu, Qianqian Xie, and Jian-Yun Nie. 2025b. Fintagging: Benchmarking llms for extracting an… — [arXiv:2505.20650](https://arxiv.org/abs/2505.20650) [find](https://scholar.google.com/scholar?q=Yan+Wang%2C+Yang+Ren%2C+Lingfei+Qian%2C+Xueqing+Peng%2C+Keyi+Wang%2C+Yi+Han%2C+Dongji+Feng%2C+Fengran+Mo%2C+Shengyuan+Lin%2C+Qinchuan+Zhang%2C+Kaiwen+He%2C+Chenri+Luo%2C+Jianxing+Chen%2C+Junwei+Wu%2C+Jimin+Huang%2C+Guojun+Xiong%2C+X)
24. Yan Wang, Keyi Wang, Shanshan Yang, Jaisal Patel, Jeff Zhao, Fengran Mo, Xueqing Peng, Lingfei Qian, Jimin Huang, Guojun Xiong, Xiao-Yang Liu, and Jian-Yun Nie. 2025c. Finauditing: A financial taxonomy-structured multi-document benchmark for evaluating llms. Preprint, arXiv:2510.08886. — [arXiv:2510.08886](https://arxiv.org/abs/2510.08886) [find](https://scholar.google.com/scholar?q=Yan+Wang%2C+Keyi+Wang%2C+Shanshan+Yang%2C+Jaisal+Patel%2C+Jeff+Zhao%2C+Fengran+Mo%2C+Xueqing+Peng%2C+Lingfei+Qian%2C+Jimin+Huang%2C+Guojun+Xiong%2C+Xiao%2DYang+Liu%2C+and+Jian%2DYun+Nie%2E+2025c%2E+Finauditing%3A+A+financial+taxonom)
25. Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. 2024. [Openmoe: An early effort on open mixture-of-experts language models](https://md.research.elamur.ai/free/en/papers/xue-2024-openmoe-early-effort-open-mixture/). arXiv preprint arXiv:2402.01739.
26. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025a. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). Preprint, arXiv:2505.09388.
27. Xingyi Yang, Constantin Venhoff, Ashkan Khakzar, Christian Schroeder de Witt, Puneet K Dokania, Adel Bibi, and Philip Torr. 2025b. Mixture of experts made intrinsically interpretable. arXiv preprint arXiv:2503.07639. — [arXiv:2503.07639](https://arxiv.org/abs/2503.07639) [find](https://scholar.google.com/scholar?q=Xingyi+Yang%2C+Constantin+Venhoff%2C+Ashkan+Khakzar%2C+Christian+Schroeder+de+Witt%2C+Puneet+K+Dokania%2C+Adel+Bibi%2C+and+Philip+Torr%2E+2025b%2E+Mixture+of+experts+made+intrinsically+interpretable%2E+arXiv+preprint+a)
28. Guorui Zheng, Xidong Wang, Juhao Liang, Nuo Chen, Yuping Zheng, and Benyou Wang. 2024. Efficiently democratizing medical llms for 50 languages via a mixture of language family experts. arXiv preprint arXiv:2410.10626. — [arXiv:2410.10626](https://arxiv.org/abs/2410.10626) [find](https://scholar.google.com/scholar?q=Guorui+Zheng%2C+Xidong+Wang%2C+Juhao+Liang%2C+Nuo+Chen%2C+Yuping+Zheng%2C+and+Benyou+Wang%2E+2024%2E+Efficiently+democratizing+medical+llms+for+50+languages+via+a+mixture+of+language+family+experts%2E+arXiv+preprint+)
29. Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. 2022. St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906. [find](https://scholar.google.com/scholar?q=Barret+Zoph%2C+Irwan+Bello%2C+Sameer+Kumar%2C+Nan+Du%2C+Yanping+Huang%2C+Jeff+Dean%2C+Noam+Shazeer%2C+and+William+Fedus%2E+2022%2E+St%2Dmoe%3A+Designing+stable+and+transferable+sparse+expert+models%2E+arXiv+preprint+arXiv%3A22)

*Updated 1 November 2026*