---
title: "Infinity-MoE: Generalizing Mixture of Experts to Infinite Experts"
description: "The authors propose ∞-MoE, which replaces the discrete expert set with a continuous expert space. 20 references."
authors: ["Shota Takashiro", "Takeshi Kojima", "Shohei Taniguchi", "Yusuke Iwasawa", "Yutaka Matsuo"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)"
source: "https://aclanthology.org/2026.eacl-short.33/"
doi: "10.18653/v1/2026.eacl-short.33"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/takashiro-2026-infinity-moe-generalizing-mixture-experts/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/takashiro-2026-infinity-moe-generalizing-mixture-experts/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/33/) / Infinity-MoE: Generalizing Mixture of Experts to Infinite Experts

# Infinity-MoE: Generalizing Mixture of Experts to Infinite Experts

**Authors:** Shota Takashiro, [Takeshi Kojima](https://md.research.elamur.ai/free/en/authors/takeshi-kojima/), [Shohei Taniguchi](https://md.research.elamur.ai/free/en/authors/shohei-taniguchi/), [Yusuke Iwasawa](https://md.research.elamur.ai/free/en/authors/yusuke-iwasawa/) and [Yutaka Matsuo](https://md.research.elamur.ai/free/en/authors/yutaka-matsuo/) · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-short.33/) · [PDF](https://aclanthology.org/2026.eacl-short.33.pdf) · [DOI 10.18653/v1/2026.eacl-short.33](https://doi.org/10.18653/v1/2026.eacl-short.33) · [OpenAlex](https://openalex.org/W7140113546)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Mixture of Experts (MoE) layers grow model capacity without a matching rise in per-token compute. However, the experts are independent, discrete units, which makes training harder as their number increases. The authors propose ∞-MoE, which replaces the discrete expert set with a continuous expert space. For each token, a router predicts a diagonal Gaussian over a continuous index z. Each sampled z is projected onto the hidden neurons of one large shared FFN, and a top-N% mask keeps only the highest-scoring neurons. Every sample therefore activates its own subset of FFN parameters, so the number of possible experts is unbounded while compute stays sparse.

The models were built on GPT-2 Small and GPT-2 Medium, pre-trained on 10B tokens from FineWeb, and evaluated zero-shot on seven benchmarks against Dense, Switch Transformer and Top-2 MoE baselines. ∞-MoE achieved the best average accuracy at both sizes. The number of samples K can also be changed at inference time, which lets users trade accuracy for speed.

## Key points
- **Benchmarks:** BoolQ, HellaSwag, WinoGrande, ARC-e/c, OpenBookQA and RACE-high, all zero-shot.
- **GPT-2 Small results:** ∞-MoE (129M active / 186M total parameters) averaged 39.8, compared with 38.5 for Dense, 38.2 for Switch Transformer and 38.8 for MoE. The authors report this as roughly matching a dense 350M GPT-2 Medium.
- **GPT-2 Medium results:** ∞-MoE (362M / 568M parameters) reached the top average of 41.3.
- **Effect of K:** varying K over {1, 2, 3, 4, 8} produced smoother scaling than standard MoE, with gains of up to 2.5% even at low K.
- **Ablations and efficiency:**
  - An index dimension of d_z = 128 gave the lowest perplexity (3.213).
  - Normalized routing entropy settled at about 0.89 during training.
  - A fused CUDA kernel made the FFN roughly 1.25× faster.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [The University of Tokyo](https://md.research.elamur.ai/free/en/universities/the-university-of-tokyo/) — University, Tokyo, Japan · 90 works · 9,156 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

## Cite

Cite as: Shota Takashiro, Takeshi Kojima, Shohei Taniguchi, Yusuke Iwasawa and Yutaka Matsuo. 2026. Infinity-MoE: Generalizing Mixture of Experts to Infinite Experts. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{takashiro2026infinity,
  title = {{Infinity-MoE: Generalizing Mixture of Experts to Infinite Experts}},
  author = {Shota Takashiro, Takeshi Kojima, Shohei Taniguchi, Yusuke Iwasawa and Yutaka Matsuo},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-short.33},
  url = {https://doi.org/10.18653/v1/2026.eacl-short.33}
}
```

- [← previous Infinite Babble: Inflating 3D Vision-Language Model Inference Overhead via Adversarial Geometric Perturbation](https://md.research.elamur.ai/free/en/papers/sun-2026-infinite-babble-inflating-3d-vision/)
- [next → Information Asymmetry across Language Varieties: A Case Study on Cantonese-Mandarin and Bavarian-German QA](https://md.research.elamur.ai/free/en/papers/pei-2026-information-asymmetry-across-language-varieties/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/33/)

## References

20 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen… — [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/)
2. William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39. [find](https://scholar.google.com/scholar?q=William+Fedus%2C+Barret+Zoph%2C+and+Noam+Shazeer%2E+2022%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+Journal+of+Machine+Learning+Research%2C+23%28120%29%3A1%E2%80%9339%2E)
3. Xu Owen He. 2024. Mixture of a million experts. [find](https://scholar.google.com/scholar?q=Xu+Owen+He%2E+2024%2E+Mixture+of+a+million+experts%2E)
4. Pratyay Banerjee, Kuntal Kumar Pal, Arindam Mitra, and Chitta Baral. 2019. Careful selection of knowledge to solve open book question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6120– 6129, Florence, Italy. Association for Computationa… [find](https://scholar.google.com/scholar?q=Pratyay+Banerjee%2C+Kuntal+Kumar+Pal%2C+Arindam+Mitra%2C+and+Chitta+Baral%2E+2019%2E+Careful+selection+of+knowledge+to+solve+open+book+question+answering%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Associ)
5. Michael Boratko, Harshit Padigela, Divyendra Mikkilineni, Pritish Yuvraj, Rajarshi Das, Andrew McCallum, Maria Chang, Achille Fokoue-Nkoutche, Pavan Kapanipathi, Nicholas Mattei, Ryan Musa, Kartik Talamadupula, and Michael Witbrock. 2018. A systematic classification of knowledge, reasoning, and cont… [find](https://scholar.google.com/scholar?q=Michael+Boratko%2C+Harshit+Padigela%2C+Divyendra+Mikkilineni%2C+Pritish+Yuvraj%2C+Rajarshi+Das%2C+Andrew+McCallum%2C+Maria+Chang%2C+Achille+Fokoue%2DNkoutche%2C+Pavan+Kapanipathi%2C+Nicholas+Mattei%2C+Ryan+Musa%2C+Kartik+Tal)
6. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374.
7. Aidan Clark, Diego De Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, George Bm Van Den Driessche, Eliza Rutherford, Tom Hennigan, Matthew J Johnson, Albin Cassirer, Chris Jones, Elena Buchatskaya, David Budden,… [find](https://scholar.google.com/scholar?q=Aidan+Clark%2C+Diego+De+Las+Casas%2C+Aurelia+Guy%2C+Arthur+Mensch%2C+Michela+Paganini%2C+Jordan+Hoffmann%2C+Bogdan+Damoc%2C+Blake+Hechtman%2C+Trevor+Cai%2C+Sebastian+Borgeaud%2C+George+Bm+Van+Den+Driessche%2C+Eliza+Rutherf)
8. Robert Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. 1991. Adaptive mixtures of local experts. Neural Computation, 3:79–87. [find](https://scholar.google.com/scholar?q=Robert+Jacobs%2C+Michael+Jordan%2C+Steven+Nowlan%2C+and+Geoffrey+Hinton%2E+1991%2E+Adaptive+mixtures+of+local+experts%2E+Neural+Computation%2C+3%3A79%E2%80%9387%2E)
9. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guill)
10. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). CoRR, abs/2001.08361.
11. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. [RACE: Large-scale ReAding comprehension dataset from examinations](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785– 794, Copenhagen, Denmark. Association for Computational Lin…
12. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding. Preprint, arXiv:2006.16668. [find](https://scholar.google.com/scholar?q=Dmitry+Lepikhin%2C+HyoukJoong+Lee%2C+Yuanzhong+Xu%2C+Dehao+Chen%2C+Orhan+Firat%2C+Yanping+Huang%2C+Maxim+Krikun%2C+Noam+Shazeer%2C+and+Zhifeng+Chen%2E+2020%2E+Gshard%3A+Scaling+giant+models+with+conditional+computation+and)
13. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. [Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/). arXiv preprint arXiv:2107.13586.
14. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [BoolQ: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:…
15. Jan Ludziejewski, Jakub Krajewski, Kamil Adamczewski, Maciej Pióro, Michał Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóźdź, Piotr Sankowski, Marek Cygan, and Sebastian Jaszczur. 2024. Scaling laws for fine-grained mixture of experts. In Proceedings of the 41st Internation… [find](https://scholar.google.com/scholar?q=Jan+Ludziejewski%2C+Jakub+Krajewski%2C+Kamil+Adamczewski%2C+Maciej+Pi%C3%B3ro%2C+Micha%C5%82+Krutul%2C+Szymon+Antoniak%2C+Kamil+Ciebiera%2C+Krystian+Kr%C3%B3l%2C+Tomasz+Odrzyg%C3%B3z%CC%81dz%CC%81%2C+Piotr+Sankowski%2C+Marek+Cygan%2C+and+Sebastian+Jasz)
16. Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. 2024. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language mo… [find](https://scholar.google.com/scholar?q=Damai+Dai%2C+Chengqi+Deng%2C+Chenggang+Zhao%2C+R%2E+X%2E+Xu%2C+Huazuo+Gao%2C+Deli+Chen%2C+Jiashi+Li%2C+Wangding+Zeng%2C+Xingkai+Yu%2C+Y%2E+Wu%2C+Zhenda+Xie%2C+Y%2E+K%2E+Li%2C+Panpan+Huang%2C+Fuli+Luo%2C+Chong+Ruan%2C+Zhifang+Sui%2C+and+Wenfen)
17. Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. [The fineweb datasets: Decanting the web for the finest text data at scale](https://md.research.elamur.ai/free/en/papers/penedo-2024-fineweb-datasets-decanting-web-finest/). In The Thirty-eight Conference on Neural Information Processing Systems Datasets a…
18. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI. Accessed: 2024-11-15. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI%2E+Accessed%3A+2024%2D11%2D15%2E)
19. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99–106. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2021%2E+Winogrande%3A+an+adversarial+winograd+schema+challenge+at+scale%2E+Commun%2E+ACM%2C+64%289%29%3A99%E2%80%93106%2E)
20. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. Preprint, arXiv:1701.06538. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2C+Azalia+Mirhoseini%2C+Krzysztof+Maziarz%2C+Andy+Davis%2C+Quoc+Le%2C+Geoffrey+Hinton%2C+and+Jeff+Dean%2E+2017%2E+Outrageously+large+neural+networks%3A+The+sparsely%2Dgated+mixture%2Dof%2Dexperts+layer%2E+Preprint)

*Updated 1 November 2026*