---
title: "Specialization through Collaboration — paper by Tang et al. (2026)"
description: "Drawing on the neuroscience idea that functions are carried by neural populations, the authors study groups of experts that… 27 references."
authors: ["Yuanbo Tang", "Naifan Zhang", "Yan Tang", "Meixuan Chen", "Shuhan Huang", "Tingyu Cao", "Yang Li"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.eacl-long.104/"
doi: "10.18653/v1/2026.eacl-long.104"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tang-2026-specialization-through-collaboration-understanding-expert/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tang-2026-specialization-through-collaboration-understanding-expert/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/50/) / Specialization through Collaboration: Understanding Expert Interaction in Mixture-of-Expert Large Language Models

# Specialization through Collaboration: Understanding Expert Interaction in Mixture-of-Expert Large Language Models

**Authors:** Yuanbo Tang, Naifan Zhang, Yan Tang, Meixuan Chen, Shuhan Huang, Tingyu Cao and [Yang Li](https://md.research.elamur.ai/free/en/authors/yang-li/) · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-long.104/) · [PDF](https://aclanthology.org/2026.eacl-long.104.pdf) · [DOI 10.18653/v1/2026.eacl-long.104](https://doi.org/10.18653/v1/2026.eacl-long.104) · [OpenAlex](https://openalex.org/W7140117125)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Mixture-of-Experts LLMs send each token to a few expert subnetworks, but it remains unclear whether individual experts really specialize. Drawing on the neuroscience idea that functions are carried by neural populations, the authors study groups of experts that activate together across layers. They aggregate token routing weights into sentence-level activation matrices for MMLU-pro inputs. These matrices are factorized with Hierarchical Sparse Dictionary Learning (HSDL), a multi-level extension of an existing Bayesian dictionary learning method, which yields collaboration patterns from coarse to fine. GPT-4 labels the patterns, and they correspond to capabilities like solving math problems or handling sentence structure. Groups are more domain-focused than single experts. The authors then use these groups for expert pruning (CAEP), keeping functionally important combinations. On DeepSeek-MoE and Phi-MoE, CAEP outperforms SEER-MoE and GEM by about 2.5% on average and still leads when half of the experts are removed. The work gives an interpretability lens on MoE models, a cheap compression recipe, and an interactive visualization platform.

## Key points
- Analysis data: 2,812 MMLU-pro samples covering mathematics, computer science, physics, law and psychology; the models studied are DeepSeek-MoE (16B) and Phi-MoE.
- With 25% of experts pruned, CAEP achieves the following average scores (OBQA, ARC-C, HellaSwag, WinoGrande, RTE):
  - DeepSeek: 0.650, versus 0.628 for GEM, 0.626 for SEER-MoE and 0.692 for the original model.
  - Phi-MoE: 0.652, versus 0.636 for GEM.
- With 50% of experts pruned on DeepSeek-MoE, CAEP scores 0.552, compared with 0.529 for GEM and 0.512 for SEER-MoE.
- Domain ablations confirm the groups matter:
  - Removing the math-specific expert group lowers math accuracy by 21.7%.
  - Keeping only domain-relevant experts costs just 5.7%.
  - Related domains (Math/Physics/CS) affect each other more than distant ones (Law/Physics).
- HSDL adds roughly 1,230 TFLOPs, about 2 minutes on one GTX 1080 Ti. Pruning 25% of experts saves 3.7B parameters (14.7GB). The semantic labels come from GPT-4 and have not been checked by humans.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 415 works · 94,028 citations
- [Tsinghua Shenzhen International Graduate School](https://md.research.elamur.ai/free/en/universities/tsinghua-shenzhen-international-graduate-school/) — University, Shenzhen, China · 29 works · 3,035 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Yuanbo Tang, Naifan Zhang, Yan Tang, Meixuan Chen, Shuhan Huang, Tingyu Cao and Yang Li. 2026. Specialization through Collaboration: Understanding Expert Interaction in Mixture-of-Expert Large Language Models. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{tang2026specialization,
  title = {{Specialization through Collaboration: Understanding Expert Interaction in Mixture-of-Expert Large Language Models}},
  author = {Yuanbo Tang, Naifan Zhang, Yan Tang, Meixuan Chen, Shuhan Huang, Tingyu Cao and Yang Li},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.104},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.104}
}
```

- [← previous SpecEdit: A Spectral Approach for Multi-Round Knowledge Editing](https://md.research.elamur.ai/free/en/papers/liu-2026-specedit-spectral-approach-multi-round/)
- [next → Specialization without Sparsity: Efficient and Expressive Split-Path Experts for LLM Fine-Tuning](https://md.research.elamur.ai/free/en/papers/liu-2026-specialization-without-sparsity-efficient-expressive/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/50/)

## References

27 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. 2024. A Survey on Mixture of Experts. arXiv preprint. [find](https://scholar.google.com/scholar?q=Weilin+Cai%2C+Juyong+Jiang%2C+Fan+Wang%2C+Jing+Tang%2C+Sunghun+Kim%2C+and+Jiayi+Huang%2E+2024%2E+A+Survey+on+Mixture+of+Experts%2E+arXiv+preprint%2E)
2. Chen Chen, Hao Su, Qixing Huang, Lin Zhang, and Leonidas Guibas. 2013. Pathlet learning for compressing and planning trajectories. In Proceedings of the 21st ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, pages 392–395, Orlando Florida. ACM. [find](https://scholar.google.com/scholar?q=Chen+Chen%2C+Hao+Su%2C+Qixing+Huang%2C+Lin+Zhang%2C+and+Leonidas+Guibas%2E+2013%2E+Pathlet+learning+for+compressing+and+planning+trajectories%2E+In+Proceedings+of+the+21st+ACM+SIGSPATIAL+International+Conference+on)
3. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). Preprint, arXiv:1803.05457.
4. Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. 2024. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Mo… [find](https://scholar.google.com/scholar?q=Damai+Dai%2C+Chengqi+Deng%2C+Chenggang+Zhao%2C+R%2E+X%2E+Xu%2C+Huazuo+Gao%2C+Deli+Chen%2C+Jiashi+Li%2C+Wangding+Zeng%2C+Xingkai+Yu%2C+Y%2E+Wu%2C+Zhenda+Xie%2C+Y%2E+K%2E+Li%2C+Panpan+Huang%2C+Fuli+Luo%2C+Chong+Ruan%2C+Zhifang+Sui%2C+and+Wenfen)
5. William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39. [find](https://scholar.google.com/scholar?q=William+Fedus%2C+Barret+Zoph%2C+and+Noam+Shazeer%2E+2022%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+Journal+of+Machine+Learning+Research%2C+23%28120%29%3A1%E2%80%9339%2E)
6. Leo Gao, Gabriel Goh, and Ilya Sutskever. Scaling and evaluating sparse autoencoders. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Pha… [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Gabriel+Goh%2C+and+Ilya+Sutskever%2E+Scaling+and+evaluating+sparse+autoencoders%2E+Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C)
7. Shwai He, Daize Dong, Liang Ding, and Ang Li. 2024. Demystifying the Compression of Mixture-of-Experts Through a Unified Framework. arXiv preprint. [find](https://scholar.google.com/scholar?q=Shwai+He%2C+Daize+Dong%2C+Liang+Ding%2C+and+Ang+Li%2E+2024%2E+Demystifying+the+Compression+of+Mixture%2Dof%2DExperts+Through+a+Unified+Framework%2E+arXiv+preprint%2E)
8. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations.
9. Zhi Hou, Xiaojiang Peng, Yu Qiao, and Dacheng Tao. 2020. Visual Compositional Learning for Human-Object Interaction Detection. In Computer Vision – ECCV 2020, pages 584–600, Cham. Springer International Publishing. [find](https://scholar.google.com/scholar?q=Zhi+Hou%2C+Xiaojiang+Peng%2C+Yu+Qiao%2C+and+Dacheng+Tao%2E+2020%2E+Visual+Compositional+Learning+for+Human%2DObject+Interaction+Detection%2E+In+Computer+Vision+%E2%80%93+ECCV+2020%2C+pages+584%E2%80%93600%2C+Cham%2E+Springer+Internation)
10. Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng, and Dacheng Tao. 2021. Detecting Human-Object Interaction via Fabricated Compositional Learning. pages 14646–14655. [find](https://scholar.google.com/scholar?q=Zhi+Hou%2C+Baosheng+Yu%2C+Yu+Qiao%2C+Xiaojiang+Peng%2C+and+Dacheng+Tao%2E+2021%2E+Detecting+Human%2DObject+Interaction+via+Fabricated+Compositional+Learning%2E+pages+14646%E2%80%9314655%2E)
11. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guill)
12. Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. 2023. Merge, then compress: Demystify efficient smoe with hints from its routing policy. CoRR. [find](https://scholar.google.com/scholar?q=Pingzhi+Li%2C+Zhenyu+Zhang%2C+Prateek+Yadav%2C+Yi%2DLin+Sung%2C+Yu+Cheng%2C+Mohit+Bansal%2C+and+Tianlong+Chen%2E+2023%2E+Merge%2C+then+compress%3A+Demystify+efficient+smoe+with+hints+from+its+routing+policy%2E+CoRR%2E)
13. Ka Man Lo, Zeyu Huang, Zihan Qiu, Zili Wang, and Jie Fu. 2024. A Closer Look into Mixture-of-Experts in Large Language Models. arXiv preprint. [find](https://scholar.google.com/scholar?q=Ka+Man+Lo%2C+Zeyu+Huang%2C+Zihan+Qiu%2C+Zili+Wang%2C+and+Jie+Fu%2E+2024%2E+A+Closer+Look+into+Mixture%2Dof%2DExperts+in+Large+Language+Models%2E+arXiv+preprint%2E)
14. Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. 2024. [Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models](https://md.research.elamur.ai/free/en/papers/lu-2024-not-all-experts-are-equal/). CoRR, abs/2402.14800.
15. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2018%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+In+Proceedings+of+the+2018+Conference+on+)
16. Alexandre Muzio, Alex Sun, and Churan He. 2024. SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts. arXiv preprint. [find](https://scholar.google.com/scholar?q=Alexandre+Muzio%2C+Alex+Sun%2C+and+Churan+He%2E+2024%2E+SEER%2DMoE%3A+Sparse+Expert+Efficiency+through+Regularization+for+Mixture%2Dof%2DExperts%2E+arXiv+preprint%2E)
17. Stefano Panzeri, Monica Moroni, Houman Safaai, and Christopher D. Harvey. 2022. The structures and functions of correlations in neural population codes. Nature Reviews Neuroscience, 23(9):551–567. [find](https://scholar.google.com/scholar?q=Stefano+Panzeri%2C+Monica+Moroni%2C+Houman+Safaai%2C+and+Christopher+D%2E+Harvey%2E+2022%2E+The+structures+and+functions+of+correlations+in+neural+population+codes%2E+Nature+Reviews+Neuroscience%2C+23%289%29%3A551%E2%80%93567%2E)
18. Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. 2024. [Improving Dictionary Learning with Gated Sparse Autoencoders](https://md.research.elamur.ai/free/en/papers/rajamanoharan-2024-improving-dictionary-learning-gated-sparse/). arXiv preprint.
19. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2021%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Communications+of+the+ACM%2C+64%289%29%3A99%E2%80%93106%2E)
20. Yuanbo Tang, Zhiyuan Peng, and Yang Li. 2023. Explainable Trajectory Representation through Dictionary Learning. In Proceedings of the 31st ACM International Conference on Advances in Geographic Information Systems, SIGSPATIAL ’23, pages 1–4, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Yuanbo+Tang%2C+Zhiyuan+Peng%2C+and+Yang+Li%2E+2023%2E+Explainable+Trajectory+Representation+through+Dictionary+Learning%2E+In+Proceedings+of+the+31st+ACM+International+Conference+on+Advances+in+Geographic+Infor)
21. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In the Proceedings of ICLR.
22. J. Wright, A. Y. Yang, A. Ganesh, S. S. Sastry, and Yi Ma. 2009. Robust Face Recognition via Sparse Representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2(31):210–227. [find](https://scholar.google.com/scholar?q=J%2E+Wright%2C+A%2E+Y%2E+Yang%2C+A%2E+Ganesh%2C+S%2E+S%2E+Sastry%2C+and+Yi+Ma%2E+2009%2E+Robust+Face+Recognition+via+Sparse+Representation%2E+IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%2C+2%2831%29%3A210%E2%80%93227%2E)
23. Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. 2024. [Openmoe: An early effort on open mixture-of-experts language models](https://md.research.elamur.ai/free/en/papers/xue-2024-openmoe-early-effort-open-mixture/). In Forty-first International Conference on Machine Learning.
24. Jianchao Yang, John Wright, Thomas S. Huang, and Yi Ma. 2010. Image super-resolution via sparse representation. IEEE transactions on image processing: a publication of the IEEE Signal Processing Society, 19(11):2861–2873. [find](https://scholar.google.com/scholar?q=Jianchao+Yang%2C+John+Wright%2C+Thomas+S%2E+Huang%2C+and+Yi+Ma%2E+2010%2E+Image+super%2Dresolution+via+sparse+representation%2E+IEEE+transactions+on+image+processing%3A+a+publication+of+the+IEEE+Signal+Processing+Socie)
25. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+HellaSwag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Com)
26. Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, and Jianfeng Gao. 2024. Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts. arXiv preprint. [find](https://scholar.google.com/scholar?q=Zeliang+Zhang%2C+Xiaodong+Liu%2C+Hao+Cheng%2C+Chenliang+Xu%2C+and+Jianfeng+Gao%2E+2024%2E+Diversifying+the+Expert+Knowledge+for+Task%2DAgnostic+Pruning+in+Sparse+Mixture%2Dof%2DExperts%2E+arXiv+preprint%2E)
27. Zihui Zhao, Yuanbo Tang, Jieyu Ren, Xiaoping Zhang, and Yang Li. 2025. A Unified Probabilistic Framework for Dictionary Learning with Parsimonious Activation. arXiv preprint. ArXiv:2509.25690 [cs]. — [arXiv:2509.25690](https://arxiv.org/abs/2509.25690) [find](https://scholar.google.com/scholar?q=Zihui+Zhao%2C+Yuanbo+Tang%2C+Jieyu+Ren%2C+Xiaoping+Zhang%2C+and+Yang+Li%2E+2025%2E+A+Unified+Probabilistic+Framework+for+Dictionary+Learning+with+Parsimonious+Activation%2E+arXiv+preprint%2E+ArXiv%3A2509%2E25690+%5Bcs%5D%2E+%E2%80%94+)

*Updated 1 November 2026*