---
title: "Scaling Vision with Sparse Mixture of Experts — Riquelme 2021"
description: "The authors build V-MoE on the Vision Transformer (ViT). 70 references, cited by 103 papers here."
authors: ["Carlos Riquelme", "Joan Puigcerver", "Basil Mustafa", "Maxim Neumann", "Rodolphe Jenatton", "Andre Susano Pinto", "Daniel Keysers", "Neil Houlsby"]
year: 2021
venue: "arXiv:2106.05974"
source: "https://arxiv.org/abs/2106.05974"
doi: "10.48550/arxiv.2106.05974"
arxiv: "2106.05974"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/riquelme-2021-scaling-vision-sparse-mixture-experts/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/riquelme-2021-scaling-vision-sparse-mixture-experts/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / Scaling Vision with Sparse Mixture of Experts

# Scaling Vision with Sparse Mixture of Experts

**Authors:** [Carlos Riquelme](https://md.research.elamur.ai/free/en/authors/carlos-riquelme/), [Joan Puigcerver](https://md.research.elamur.ai/free/en/authors/joan-puigcerver/), [Basil Mustafa](https://md.research.elamur.ai/free/en/authors/basil-mustafa/), [Maxim Neumann](https://md.research.elamur.ai/free/en/authors/maxim-neumann/), Rodolphe Jenatton, [Andre Susano Pinto](https://md.research.elamur.ai/free/en/authors/andre-susano-pinto/), [Daniel Keysers](https://md.research.elamur.ai/free/en/authors/daniel-keysers/) and [Neil Houlsby](https://md.research.elamur.ai/free/en/authors/neil-houlsby/) · **Year:** 2021 · **Venue:** arXiv:2106.05974 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2106.05974) · [PDF](https://arxiv.org/pdf/2106.05974) · [DOI 10.48550/arxiv.2106.05974](https://doi.org/10.48550/arxiv.2106.05974) · [arXiv 2106.05974](https://arxiv.org/abs/2106.05974) · [OpenAlex](https://openalex.org/W3170943566)

**Citations:** 25 citations (OpenAlex, 2026-10-14) · [103](https://md.research.elamur.ai/free/en/papers/riquelme-2021-scaling-vision-sparse-mixture-experts/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large vision networks are usually dense: each parameter touches every input, so growing capacity drives up compute. This paper brings sparse Mixture of Experts, already successful in NLP, to image recognition. The authors build V-MoE on the Vision Transformer (ViT). Some of its feedforward MLP blocks are swapped for banks of expert MLPs, and a learned router sends each image patch, not the whole image, to only a few experts. Models are pre-trained on JFT-300M and then transferred through linear few-shot evaluation and full fine-tuning. The paper also proposes Batch Prioritized Routing. This scheme ranks tokens across a batch so that low-value patches can be dropped, which allows compute to vary per image and to be reduced at test time without retraining. V-MoE equals or beats dense ViT models at roughly half the inference cost. The largest version, with about 15B parameters, is the biggest vision model reported so far. These results show that conditional computation is a workable way to scale vision models efficiently.

## Key points
- Largest model: ~15B parameters, up to 24 MoE layers with 32 experts each; fine-tuned ImageNet accuracy reaches 90.35%.
- Default setup: k = 2 experts selected per token, E = 32 total experts, and a capacity ratio C = 1.05 during upstream training. MoE layers sit on alternate blocks (Every-2) or only on the final blocks (Last-n).
- Routing applies top-k after the softmax, which makes k = 1 trainable, and adds small Gaussian noise. Fixed expert buffers and auxiliary losses keep the load balanced across experts.
- Pre-training data: JFT-300M, with ~305M images across 18,291 classes. Models still transfer well when fine-tuning uses only about 1,000 examples.
- Batch Prioritized Routing matches dense model quality while using 20% fewer training FLOPs. The values of k and C can be changed after training to trade accuracy against cost.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 492 works · 440,463 citations
- [Amazon](https://md.research.elamur.ai/free/en/companies/amazon/) — Company, Seattle, United States · 128 works · 41,700 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, Andre Susano Pinto, Daniel Keysers and Neil Houlsby. 2021. Scaling Vision with Sparse Mixture of Experts. arXiv:2106.05974.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{riquelme2021scaling,
  title = {{Scaling Vision with Sparse Mixture of Experts}},
  author = {Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, Andre Susano Pinto, Daniel Keysers and Neil Houlsby},
  year = {2021},
  doi = {10.48550/arxiv.2106.05974},
  eprint = {2106.05974},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2106.05974}
}
```

- [← previous A Comprehensive Assessment of Dialog Evaluation Metrics](https://md.research.elamur.ai/free/en/papers/yeh-2021-comprehensive-assessment-dialog-evaluation-metrics/)
- [next → Zero-shot Event Extraction via Transfer Learning: Challenges and Insights](https://md.research.elamur.ai/free/en/papers/lyu-2021-zero-shot-event-extraction-transfer/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

70 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Abbas and Y. Andreopoulos. Biased mixtures of experts: Enabling computer vision inference under data transfer limitations. IEEE Trans. Image Processing, 29:7656–7667, 2020. [find](https://scholar.google.com/scholar?q=A%2E+Abbas+and+Y%2E+Andreopoulos%2E+Biased+mixtures+of+experts%3A+Enabling+computer+vision+inference+under+data+transfer+limitations%2E+IEEE+Trans%2E+Image+Processing%2C+29%3A7656%E2%80%937667%2C+2020%2E)
2. K. Ahmed, M. H. Baig, and L. Torresani. Network of experts for large-scale image categorization. In ECCV, 2016. [find](https://scholar.google.com/scholar?q=K%2E+Ahmed%2C+M%2E+H%2E+Baig%2C+and+L%2E+Torresani%2E+Network+of+experts+for+large%2Dscale+image+categorization%2E+In+ECCV%2C+2016%2E)
3. A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić, and C. Schmid. ViViT: A video vision transformer. arXiv preprint arXiv:2103.15691, 2021. [find](https://scholar.google.com/scholar?q=A%2E+Arnab%2C+M%2E+Dehghani%2C+G%2E+Heigold%2C+C%2E+Sun%2C+M%2E+Luc%CC%8Cic%CC%81%2C+and+C%2E+Schmid%2E+ViViT%3A+A+video+vision+transformer%2E+arXiv+preprint+arXiv%3A2103%2E15691%2C+2021%2E)
4. E. Bengio, P.-L. Bacon, J. Pineau, and D. Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015. — [arXiv:1511.06297](https://arxiv.org/abs/1511.06297) [find](https://scholar.google.com/scholar?q=E%2E+Bengio%2C+P%2E%2DL%2E+Bacon%2C+J%2E+Pineau%2C+and+D%2E+Precup%2E+Conditional+computation+in+neural+networks+for+faster+models%2E+arXiv+preprint+arXiv%3A1511%2E06297%2C+2015%2E+%E2%80%94+%5BarXiv%3A1511%2E06297%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1511%2E0)
5. Y. Bengio. Deep learning of representations: Looking forward. In International Conference on Statistical Language and Speech Processing, pages 1–37, 2013. [find](https://scholar.google.com/scholar?q=Y%2E+Bengio%2E+Deep+learning+of+representations%3A+Looking+forward%2E+In+International+Conference+on+Statistical+Language+and+Speech+Processing%2C+pages+1%E2%80%9337%2C+2013%2E)
6. Y. Bengio, N. Léonard, and A. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013. [find](https://scholar.google.com/scholar?q=Y%2E+Bengio%2C+N%2E+L%C3%A9onard%2C+and+A%2E+Courville%2E+Estimating+or+propagating+gradients+through+stochastic+neurons+for+conditional+computation%2E+arXiv+preprint+arXiv%3A1308%2E3432%2C+2013%2E)
7. L. Breiman, J. Friedman, C. J. Stone, and R. A. Olshen. Classification and regression trees. CRC press, 1984. [find](https://scholar.google.com/scholar?q=L%2E+Breiman%2C+J%2E+Friedman%2C+C%2E+J%2E+Stone%2C+and+R%2E+A%2E+Olshen%2E+Classification+and+regression+trees%2E+CRC+press%2C+1984%2E)
8. A. Brock, S. De, S. L. Smith, and K. Simonyan. High-performance large-scale image recognition without normalization. arXiv preprint arXiv:2102.06171, 2021. — [arXiv:2102.06171](https://arxiv.org/abs/2102.06171) [find](https://scholar.google.com/scholar?q=A%2E+Brock%2C+S%2E+De%2C+S%2E+L%2E+Smith%2C+and+K%2E+Simonyan%2E+High%2Dperformance+large%2Dscale+image+recognition+without+normalization%2E+arXiv+preprint+arXiv%3A2102%2E06171%2C+2021%2E+%E2%80%94+%5BarXiv%3A2102%2E06171%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
9. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). arXiv preprint arXiv:2005.14165, 2020.
10. K. Chen, L. Xu, and H. Chi. Improved learning algorithms for mixture of experts in multiclass classification. Neural networks, 12(9):1229–1252, 1999. [find](https://scholar.google.com/scholar?q=K%2E+Chen%2C+L%2E+Xu%2C+and+H%2E+Chi%2E+Improved+learning+algorithms+for+mixture+of+experts+in+multiclass+classification%2E+Neural+networks%2C+12%289%29%3A1229%E2%80%931252%2C+1999%2E)
11. G. Cheng, J. Han, and X. Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):1865–1883, Oct 2017. [find](https://scholar.google.com/scholar?q=G%2E+Cheng%2C+J%2E+Han%2C+and+X%2E+Lu%2E+Remote+sensing+image+scene+classification%3A+Benchmark+and+state+of+the+art%2E+Proceedings+of+the+IEEE%2C+105%2810%29%3A1865%E2%80%931883%2C+Oct+2017%2E)
12. K. Cho and Y. Bengio. Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning. arXiv preprint arXiv:1406.7362, 2014. [find](https://scholar.google.com/scholar?q=K%2E+Cho+and+Y%2E+Bengio%2E+Exponentially+increasing+the+capacity%2Dto%2Dcomputation+ratio+for+conditional+computation+in+deep+learning%2E+arXiv+preprint+arXiv%3A1406%2E7362%2C+2014%2E)
13. M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Computer Vision and Pattern Recognition (CVPR), 2014. [find](https://scholar.google.com/scholar?q=M%2E+Cimpoi%2C+S%2E+Maji%2C+I%2E+Kokkinos%2C+S%2E+Mohamed%2C+and+A%2E+Vedaldi%2E+Describing+textures+in+the+wild%2E+In+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+2014%2E)
14. E. D. Cubuk, B. Zoph, J. Shlens, and Q. Le. Randaugment: Practical automated data augmentation with a reduced search space. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, 2020. [find](https://scholar.google.com/scholar?q=E%2E+D%2E+Cubuk%2C+B%2E+Zoph%2C+J%2E+Shlens%2C+and+Q%2E+Le%2E+Randaugment%3A+Practical+automated+data+augmentation+with+a+reduced+search+space%2E+In+H%2E+Larochelle%2C+M%2E+Ranzato%2C+R%2E+Hadsell%2C+M%2E+F%2E+Balcan%2C+and+H%2E+Lin%2C+editors%2C)
15. A. Davis and I. Arel. Low-rank approximations for conditional feedforward computation in deep neural networks. arXiv preprint arXiv:1312.4461, 2013. — [arXiv:1312.4461](https://arxiv.org/abs/1312.4461) [find](https://scholar.google.com/scholar?q=A%2E+Davis+and+I%2E+Arel%2E+Low%2Drank+approximations+for+conditional+feedforward+computation+in+deep+neural+networks%2E+arXiv+preprint+arXiv%3A1312%2E4461%2C+2013%2E+%E2%80%94+%5BarXiv%3A1312%2E4461%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1312%2E4461)
16. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009. [find](https://scholar.google.com/scholar?q=J%2E+Deng%2C+W%2E+Dong%2C+R%2E+Socher%2C+L%2E%2DJ%2E+Li%2C+K%2E+Li%2C+and+L%2E+Fei%2DFei%2E+ImageNet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+CVPR%2C+2009%2E)
17. L. Denoyer and P. Gallinari. Deep sequential neural network. arXiv preprint arXiv:1410.0510, 2014. — [arXiv:1410.0510](https://arxiv.org/abs/1410.0510) [find](https://scholar.google.com/scholar?q=L%2E+Denoyer+and+P%2E+Gallinari%2E+Deep+sequential+neural+network%2E+arXiv+preprint+arXiv%3A1410%2E0510%2C+2014%2E+%E2%80%94+%5BarXiv%3A1410%2E0510%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1410%2E0510%29)
18. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In NAACL, 2019.
19. G. S. Dhillon, P. Chaudhari, A. Ravichandran, and S. Soatto. A baseline for few-shot image classification. In ICLR, 2020. [find](https://scholar.google.com/scholar?q=G%2E+S%2E+Dhillon%2C+P%2E+Chaudhari%2C+A%2E+Ravichandran%2C+and+S%2E+Soatto%2E+A+baseline+for+few%2Dshot+image+classification%2E+In+ICLR%2C+2020%2E)
20. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). In ICLR, 2021.
21. D. Eigen, M. Ranzato, and I. Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv:1312.4314, 2013. — [arXiv:1312.4314](https://arxiv.org/abs/1312.4314) [find](https://scholar.google.com/scholar?q=D%2E+Eigen%2C+M%2E+Ranzato%2C+and+I%2E+Sutskever%2E+Learning+factored+representations+in+a+deep+mixture+of+experts%2E+arXiv+preprint+arXiv%3A1312%2E4314%2C+2013%2E+%E2%80%94+%5BarXiv%3A1312%2E4314%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1312%2E4314%29)
22. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961, 2021. [find](https://scholar.google.com/scholar?q=W%2E+Fedus%2C+B%2E+Zoph%2C+and+N%2E+Shazeer%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+arXiv+preprint+arXiv%3A2101%2E03961%2C+2021%2E)
23. D. M. Gavrila and S. Munder. Multi-cue pedestrian detection and tracking from a moving vehicle. International journal of computer vision, 73(1):41–59, 2007. [find](https://scholar.google.com/scholar?q=D%2E+M%2E+Gavrila+and+S%2E+Munder%2E+Multi%2Dcue+pedestrian+detection+and+tracking+from+a+moving+vehicle%2E+International+journal+of+computer+vision%2C+73%281%29%3A41%E2%80%9359%2C+2007%2E)
24. A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? The KITTI vision benchmark suite. In Computer Vision and Pattern Recognition (CVPR), 2012. [find](https://scholar.google.com/scholar?q=A%2E+Geiger%2C+P%2E+Lenz%2C+and+R%2E+Urtasun%2E+Are+we+ready+for+autonomous+driving%3F+The+KITTI+vision+benchmark+suite%2E+In+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+2012%2E)
25. S. Gross, M. Ranzato, and A. Szlam. Hard mixtures of experts for large scale weakly supervised vision. In CVPR, 2017. [find](https://scholar.google.com/scholar?q=S%2E+Gross%2C+M%2E+Ranzato%2C+and+A%2E+Szlam%2E+Hard+mixtures+of+experts+for+large+scale+weakly+supervised+vision%2E+In+CVPR%2C+2017%2E)
26. J. V. Hansen. Combining predictors: comparison of five meta machine learning methods. Information Sciences, 119(1-2):91–105, 1999. [find](https://scholar.google.com/scholar?q=J%2E+V%2E+Hansen%2E+Combining+predictors%3A+comparison+of+five+meta+machine+learning+methods%2E+Information+Sciences%2C+119%281%2D2%29%3A91%E2%80%93105%2C+1999%2E)
27. T. Hastie, R. Tibshirani, and J. Friedman. The elements of statistical learning: data mining, inference, and prediction. Springer, 2017. [find](https://scholar.google.com/scholar?q=T%2E+Hastie%2C+R%2E+Tibshirani%2C+and+J%2E+Friedman%2E+The+elements+of+statistical+learning%3A+data+mining%2C+inference%2C+and+prediction%2E+Springer%2C+2017%2E)
28. P. Helber, B. Bischke, A. Dengel, and D. Borth. EuroSAT: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019. [find](https://scholar.google.com/scholar?q=P%2E+Helber%2C+B%2E+Bischke%2C+A%2E+Dengel%2C+and+D%2E+Borth%2E+EuroSAT%3A+A+novel+dataset+and+deep+learning+benchmark+for+land+use+and+land+cover+classification%2E+IEEE+Journal+of+Selected+Topics+in+Applied+Earth+Observ)
29. D. Hendrycks and K. Gimpel. arXiv:1606.08415, 2016. Gaussian error linear units (gelus). arXiv preprint [find](https://scholar.google.com/scholar?q=D%2E+Hendrycks+and+K%2E+Gimpel%2E+arXiv%3A1606%2E08415%2C+2016%2E+Gaussian+error+linear+units+%28gelus%29%2E+arXiv+preprint)
30. Y. H. Hu, S. Palreddy, and W. J. Tompkins. A patient-adaptable ECG beat classifier using a mixture of experts approach. IEEE Trans. Biomedical Engineering, 44(9):891–900, 1997. [find](https://scholar.google.com/scholar?q=Y%2E+H%2E+Hu%2C+S%2E+Palreddy%2C+and+W%2E+J%2E+Tompkins%2E+A+patient%2Dadaptable+ECG+beat+classifier+using+a+mixture+of+experts+approach%2E+IEEE+Trans%2E+Biomedical+Engineering%2C+44%289%29%3A891%E2%80%93900%2C+1997%2E)
31. R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79–87, 1991. [find](https://scholar.google.com/scholar?q=R%2E+A%2E+Jacobs%2C+M%2E+I%2E+Jordan%2C+S%2E+J%2E+Nowlan%2C+and+G%2E+E%2E+Hinton%2E+Adaptive+mixtures+of+local+experts%2E+Neural+Computation%2C+3%281%29%3A79%E2%80%9387%2C+1991%2E)
32. Z. Jiang, Q. Hou, L. Yuan, D. Zhou, X. Jin, A. Wang, and J. Feng. Token labeling: Training a 85.5% top-1 accuracy vision transformer with 56m parameters on imagenet. arXiv preprint arXiv:2104.10858, 2021. — [arXiv:2104.10858](https://arxiv.org/abs/2104.10858) [find](https://scholar.google.com/scholar?q=Z%2E+Jiang%2C+Q%2E+Hou%2C+L%2E+Yuan%2C+D%2E+Zhou%2C+X%2E+Jin%2C+A%2E+Wang%2C+and+J%2E+Feng%2E+Token+labeling%3A+Training+a+85%2E5%25+top%2D1+accuracy+vision+transformer+with+56m+parameters+on+imagenet%2E+arXiv+preprint+arXiv%3A2104%2E10858%2C+2)
33. J. Johnson, B. Hariharan, L. van der Maaten, F.-F. Li, C. Lawrence Zitnick, and R. Girshick. [CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning](https://md.research.elamur.ai/free/en/papers/johnson-2016-clevr-diagnostic-dataset-compositional-language/). In Computer Vision and Pattern Recognition (CVPR), 2017.
34. M. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Neural computation, 6(2):181–214, 1994. [find](https://scholar.google.com/scholar?q=M%2E+I%2E+Jordan+and+R%2E+A%2E+Jacobs%2E+Hierarchical+mixtures+of+experts+and+the+EM+algorithm%2E+Neural+computation%2C+6%282%29%3A181%E2%80%93214%2C+1994%2E)
35. Kaggle and EyePacs. Kaggle diabetic retinopathy detection, 2015. [find](https://scholar.google.com/scholar?q=Kaggle+and+EyePacs%2E+Kaggle+diabetic+retinopathy+detection%2C+2015%2E)
36. A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung, S. Gelly, and N. Houlsby. Big transfer (BiT): General visual representation learning. In ECCV, 2020. [find](https://scholar.google.com/scholar?q=A%2E+Kolesnikov%2C+L%2E+Beyer%2C+X%2E+Zhai%2C+J%2E+Puigcerver%2C+J%2E+Yung%2C+S%2E+Gelly%2C+and+N%2E+Houlsby%2E+Big+transfer+%28BiT%29%3A+General+visual+representation+learning%2E+In+ECCV%2C+2020%2E)
37. A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. [find](https://scholar.google.com/scholar?q=A%2E+Krizhevsky%2E+Learning+multiple+layers+of+features+from+tiny+images%2E+Technical+report%2C+University+of+Toronto%2C+2009%2E)
38. Y. LeCun, F. J. Huang, and L. Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Computer Vision and Pattern Recognition (CVPR), 2004. [find](https://scholar.google.com/scholar?q=Y%2E+LeCun%2C+F%2E+J%2E+Huang%2C+and+L%2E+Bottou%2E+Learning+methods+for+generic+object+recognition+with+invariance+to+pose+and+lighting%2E+In+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+2004%2E)
39. D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In ICLR, 2021. [find](https://scholar.google.com/scholar?q=D%2E+Lepikhin%2C+H%2E+Lee%2C+Y%2E+Xu%2C+D%2E+Chen%2C+O%2E+Firat%2C+Y%2E+Huang%2C+M%2E+Krikun%2C+N%2E+Shazeer%2C+and+Z%2E+Chen%2E+GShard%3A+Scaling+giant+models+with+conditional+computation+and+automatic+sharding%2E+In+ICLR%2C+2021%2E)
40. M. Lewis, S. Bhosale, T. Dettmers, N. Goyal, and L. Zettlemoyer. [Base layers: Simplifying training of large, sparse models](https://md.research.elamur.ai/free/en/papers/lewis-2021-base-layers-simplifying-training-large/). arXiv preprint arXiv:2103.16716, 2021.
41. F.-F. Li, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental Bayesian approach tested on 101 object categories. Computer Vision and Pattern Recognition Workshop, 2004. [find](https://scholar.google.com/scholar?q=F%2E%2DF%2E+Li%2C+R%2E+Fergus%2C+and+P%2E+Perona%2E+Learning+generative+visual+models+from+few+training+examples%3A+An+incremental+Bayesian+approach+tested+on+101+object+categories%2E+Computer+Vision+and+Pattern+Recognit)
42. L. Matthey, I. Higgins, D. Hassabis, and A. Lerchner. dSprites: Disentanglement testing sprites dataset, 2017. [find](https://scholar.google.com/scholar?q=L%2E+Matthey%2C+I%2E+Higgins%2C+D%2E+Hassabis%2C+and+A%2E+Lerchner%2E+dSprites%3A+Disentanglement+testing+sprites+dataset%2C+2017%2E)
43. Y. Netzer, T. Wang, A. Coates, A. Bissacco, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011. [find](https://scholar.google.com/scholar?q=Y%2E+Netzer%2C+T%2E+Wang%2C+A%2E+Coates%2C+A%2E+Bissacco%2C+and+A%2E+Y%2E+Ng%2E+Reading+digits+in+natural+images+with+unsupervised+feature+learning%2E+In+NIPS+Workshop+on+Deep+Learning+and+Unsupervised+Feature+Learning+2011%2C)
44. M.-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In Sixth Indian Conf. on Computer Vision, Graphics & Image Processing, 2008. [find](https://scholar.google.com/scholar?q=M%2E%2DE%2E+Nilsback+and+A%2E+Zisserman%2E+Automated+flower+classification+over+a+large+number+of+classes%2E+In+Sixth+Indian+Conf%2E+on+Computer+Vision%2C+Graphics+%26+Image+Processing%2C+2008%2E)
45. O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar. Cats and dogs. In CVPR, 2012. [find](https://scholar.google.com/scholar?q=O%2E+M%2E+Parkhi%2C+A%2E+Vedaldi%2C+A%2E+Zisserman%2C+and+C%2E+Jawahar%2E+Cats+and+dogs%2E+In+CVPR%2C+2012%2E)
46. D. Patterson, J. Gonzalez, Q. Le, C. Liang, L.-M. Munguia, D. Rothchild, D. So, M. Texier, and J. Dean. [Carbon emissions and large neural network training](https://md.research.elamur.ai/free/en/papers/patterson-2021-carbon-emissions-large-neural-network/). arXiv preprint arXiv:2104.10350, 2021.
47. S. Pavlitskaya, C. Hubschneider, M. Weber, R. Moritz, F. Huger, P. Schlicht, and M. Zollner. Using mixture of expert models to gain insights into semantic segmentation. In CVPR Workshops, 2020. [find](https://scholar.google.com/scholar?q=S%2E+Pavlitskaya%2C+C%2E+Hubschneider%2C+M%2E+Weber%2C+R%2E+Moritz%2C+F%2E+Huger%2C+P%2E+Schlicht%2C+and+M%2E+Zollner%2E+Using+mixture+of+expert+models+to+gain+insights+into+semantic+segmentation%2E+In+CVPR+Workshops%2C+2020%2E)
48. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In NAACL, 2018.
49. H. Pham, Z. Dai, Q. Xie, M.-T. Luong, and Q. V. Le. Meta pseudo labels. arXiv preprint arXiv:2003.10580, 2020. [find](https://scholar.google.com/scholar?q=H%2E+Pham%2C+Z%2E+Dai%2C+Q%2E+Xie%2C+M%2E%2DT%2E+Luong%2C+and+Q%2E+V%2E+Le%2E+Meta+pseudo+labels%2E+arXiv+preprint+arXiv%3A2003%2E10580%2C+2020%2E)
50. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). arXiv preprint arXiv:1910.10683, 2019.
51. P. Ramachandran and Q. V. Le. Diversity and depth in per-example routing models. In ICLR, 2018. [find](https://scholar.google.com/scholar?q=P%2E+Ramachandran+and+Q%2E+V%2E+Le%2E+Diversity+and+depth+in+per%2Dexample+routing+models%2E+In+ICLR%2C+2018%2E)
52. C. Rosenbaum, I. Cases, M. Riemer, and T. Klinger. Routing networks and the challenges of modular and compositional computation. arXiv preprint arXiv:1904.12774, 2019. — [arXiv:1904.12774](https://arxiv.org/abs/1904.12774) [find](https://scholar.google.com/scholar?q=C%2E+Rosenbaum%2C+I%2E+Cases%2C+M%2E+Riemer%2C+and+T%2E+Klinger%2E+Routing+networks+and+the+challenges+of+modular+and+compositional+computation%2E+arXiv+preprint+arXiv%3A1904%2E12774%2C+2019%2E+%E2%80%94+%5BarXiv%3A1904%2E12774%5D%28https%3A%2F%2Farx)
53. C. Rosenbaum, T. Klinger, and M. Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. arXiv preprint arXiv:1711.01239, 2017. — [arXiv:1711.01239](https://arxiv.org/abs/1711.01239) [find](https://scholar.google.com/scholar?q=C%2E+Rosenbaum%2C+T%2E+Klinger%2C+and+M%2E+Riemer%2E+Routing+networks%3A+Adaptive+selection+of+non%2Dlinear+functions+for+multi%2Dtask+learning%2E+arXiv+preprint+arXiv%3A1711%2E01239%2C+2017%2E+%E2%80%94+%5BarXiv%3A1711%2E01239%5D%28https%3A%2F%2Farxiv)
54. N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In ICLR, 2017. [find](https://scholar.google.com/scholar?q=N%2E+Shazeer%2C+A%2E+Mirhoseini%2C+K%2E+Maziarz%2C+A%2E+Davis%2C+Q%2E+Le%2C+G%2E+Hinton%2C+and+J%2E+Dean%2E+Outrageously+large+neural+networks%3A+The+sparsely%2Dgated+mixture%2Dof%2Dexperts+layer%2E+In+ICLR%2C+2017%2E)
55. C. Sminchisescu, A. Kanaujia, Z. Li, and D. Metaxas. Learning to reconstruct 3D human motion from Bayesian mixtures of experts. A probabilistic discriminative approach. Dept. Comput. Sci., Univ. Toronto, Tech. Rep. CSRG-502, 2004. [find](https://scholar.google.com/scholar?q=C%2E+Sminchisescu%2C+A%2E+Kanaujia%2C+Z%2E+Li%2C+and+D%2E+Metaxas%2E+Learning+to+reconstruct+3D+human+motion+from+Bayesian+mixtures+of+experts%2E+A+probabilistic+discriminative+approach%2E+Dept%2E+Comput%2E+Sci%2E%2C+Univ%2E+Toron)
56. E. Strubell, A. Ganesh, and A. McCallum. [Energy and policy considerations for deep learning in NLP](https://md.research.elamur.ai/free/en/papers/strubell-2019-energy-policy-considerations-deep-learning/). arXiv preprint arXiv:1906.02243, 2019.
57. C. Sun, A. Shrivastava, S. Singh, and A. Gupta. [Revisiting unreasonable effectiveness of data in deep learning era](https://md.research.elamur.ai/free/en/papers/sun-2017-revisiting-unreasonable-effectiveness-data-deep/). In ICCV, 2017.
58. J. Tani and S. Nolfi. Learning to perceive the world as articulated: an approach for hierarchical learning in sensory-motor systems. Neural Networks, 12(7-8):1131–1141, 1999. [find](https://scholar.google.com/scholar?q=J%2E+Tani+and+S%2E+Nolfi%2E+Learning+to+perceive+the+world+as+articulated%3A+an+approach+for+hierarchical+learning+in+sensory%2Dmotor+systems%2E+Neural+Networks%2C+12%287%2D8%29%3A1131%E2%80%931141%2C+1999%2E)
59. H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. [Training data-efficient image transformers & distillation through attention](https://md.research.elamur.ai/free/en/papers/touvron-2020-training-data-efficient-image-transformers/). arXiv preprint arXiv:2012.12877, 2020.
60. H. Touvron, M. Cord, A. Sablayrolles, G. Synnaeve, and H. Jégou. Going deeper with image transformers. arXiv preprint arXiv:2103.17239, 2021. [find](https://scholar.google.com/scholar?q=H%2E+Touvron%2C+M%2E+Cord%2C+A%2E+Sablayrolles%2C+G%2E+Synnaeve%2C+and+H%2E+J%C3%A9gou%2E+Going+deeper+with+image+transformers%2E+arXiv+preprint+arXiv%3A2103%2E17239%2C+2021%2E)
61. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In NeurIPS, 2017.
62. B. S. Veeling, J. Linmans, J. Winkens, T. Cohen, and M. Welling. Rotation equivariant CNNs for digital pathology. In Medical Image Computing and Computer Assisted Intervention (MICCAI), 2018. [find](https://scholar.google.com/scholar?q=B%2E+S%2E+Veeling%2C+J%2E+Linmans%2C+J%2E+Winkens%2C+T%2E+Cohen%2C+and+M%2E+Welling%2E+Rotation+equivariant+CNNs+for+digital+pathology%2E+In+Medical+Image+Computing+and+Computer+Assisted+Intervention+%28MICCAI%29%2C+2018%2E)
63. X. Wang, F. Yu, L. Dunlap, Y.-A. Ma, R. Wang, A. Mirhoseini, T. Darrell, and J. E. Gonzalez. Deep mixture of experts via shallow embedding. In Uncertainty in Artificial Intelligence, 2020. [find](https://scholar.google.com/scholar?q=X%2E+Wang%2C+F%2E+Yu%2C+L%2E+Dunlap%2C+Y%2E%2DA%2E+Ma%2C+R%2E+Wang%2C+A%2E+Mirhoseini%2C+T%2E+Darrell%2C+and+J%2E+E%2E+Gonzalez%2E+Deep+mixture+of+experts+via+shallow+embedding%2E+In+Uncertainty+in+Artificial+Intelligence%2C+2020%2E)
64. B. Yang, G. Bender, Q. V. Le, and J. Ngiam. Condconv: Conditionally parameterized convolutions for efficient inference. arXiv preprint arXiv:1904.04971, 2019. [find](https://scholar.google.com/scholar?q=B%2E+Yang%2C+G%2E+Bender%2C+Q%2E+V%2E+Le%2C+and+J%2E+Ngiam%2E+Condconv%3A+Conditionally+parameterized+convolutions+for+efficient+inference%2E+arXiv+preprint+arXiv%3A1904%2E04971%2C+2019%2E)
65. L. Yuan, Y. Chen, T. Wang, W. Yu, Y. Shi, F. E. Tay, J. Feng, and S. Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101.11986, 2021. [find](https://scholar.google.com/scholar?q=L%2E+Yuan%2C+Y%2E+Chen%2C+T%2E+Wang%2C+W%2E+Yu%2C+Y%2E+Shi%2C+F%2E+E%2E+Tay%2C+J%2E+Feng%2C+and+S%2E+Yan%2E+Tokens%2Dto%2Dtoken+vit%3A+Training+vision+transformers+from+scratch+on+imagenet%2E+arXiv+preprint+arXiv%3A2101%2E11986%2C+2021%2E)
66. S. E. Yuksel, J. N. Wilson, and P. D. Gader. Twenty years of mixture of experts. IEEE transactions on neural networks and learning systems, 23(8):1177–1193, 2012. [find](https://scholar.google.com/scholar?q=S%2E+E%2E+Yuksel%2C+J%2E+N%2E+Wilson%2C+and+P%2E+D%2E+Gader%2E+Twenty+years+of+mixture+of+experts%2E+IEEE+transactions+on+neural+networks+and+learning+systems%2C+23%288%29%3A1177%E2%80%931193%2C+2012%2E)
67. A. J. Zeevi, R. Meir, and R. J. Adler. Time series prediction using mixtures of experts. In NeurIPS, 1997. [find](https://scholar.google.com/scholar?q=A%2E+J%2E+Zeevi%2C+R%2E+Meir%2C+and+R%2E+J%2E+Adler%2E+Time+series+prediction+using+mixtures+of+experts%2E+In+NeurIPS%2C+1997%2E)
68. X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer. Scaling vision transformers, 2021. [find](https://scholar.google.com/scholar?q=X%2E+Zhai%2C+A%2E+Kolesnikov%2C+N%2E+Houlsby%2C+and+L%2E+Beyer%2E+Scaling+vision+transformers%2C+2021%2E)
69. X. Zhai, J. Puigcerver, A. Kolesnikov, P. Ruyssen, C. Riquelme, M. Lucic, J. Djolonga, A. S. Pinto, M. Neumann, A. Dosovitskiy, L. Beyer, O. Bachem, M. Tschannen, M. Michalski, O. Bousquet, S. Gelly, and N. Houlsby. A large-scale study of representation learning with the visual task adaptation bench… — [arXiv:1910.04867](https://arxiv.org/abs/1910.04867) [find](https://scholar.google.com/scholar?q=X%2E+Zhai%2C+J%2E+Puigcerver%2C+A%2E+Kolesnikov%2C+P%2E+Ruyssen%2C+C%2E+Riquelme%2C+M%2E+Lucic%2C+J%2E+Djolonga%2C+A%2E+S%2E+Pinto%2C+M%2E+Neumann%2C+A%2E+Dosovitskiy%2C+L%2E+Beyer%2C+O%2E+Bachem%2C+M%2E+Tschannen%2C+M%2E+Michalski%2C+O%2E+Bousquet%2C+S%2E+Gelly%2C+)
70. X. Zhai, J. Puigcerver, A. Kolesnikov, P. Ruyssen, C. Riquelme, M. Lucic, J. Djolonga, A. S. Pinto, M. Neumann, A. Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867, 2019. *Table 3: Comparison of routing func… — [arXiv:1910.04867](https://arxiv.org/abs/1910.04867) [find](https://scholar.google.com/scholar?q=X%2E+Zhai%2C+J%2E+Puigcerver%2C+A%2E+Kolesnikov%2C+P%2E+Ruyssen%2C+C%2E+Riquelme%2C+M%2E+Lucic%2C+J%2E+Djolonga%2C+A%2E+S%2E+Pinto%2C+M%2E+Neumann%2C+A%2E+Dosovitskiy%2C+et+al%2E+A+large%2Dscale+study+of+representation+learning+with+the+visual+ta)

*Updated 2 November 2026*