---
title: "BASE Layers: Simplifying Training of Large, Sparse Models (2021)"
description: "Lewis et al. propose BASE layers, which treat routing during training as a linear assignment problem. 31 references, cited by 116 papers here."
authors: ["Mike Lewis", "Shruti Bhosale", "Tim Dettmers", "Naman Goyal", "Luke Zettlemoyer"]
year: 2021
venue: "arXiv:2103.16716"
source: "https://arxiv.org/abs/2103.16716"
doi: "10.48550/arxiv.2103.16716"
arxiv: "2103.16716"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lewis-2021-base-layers-simplifying-training-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lewis-2021-base-layers-simplifying-training-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / BASE Layers: Simplifying Training of Large, Sparse Models

# BASE Layers: Simplifying Training of Large, Sparse Models

**Authors:** [Mike Lewis](https://md.research.elamur.ai/free/en/authors/mike-lewis/), [Shruti Bhosale](https://md.research.elamur.ai/free/en/authors/shruti-bhosale/), [Tim Dettmers](https://md.research.elamur.ai/free/en/authors/tim-dettmers/), [Naman Goyal](https://md.research.elamur.ai/free/en/authors/naman-goyal/) and [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/) · **Year:** 2021 · **Venue:** arXiv:2103.16716 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2103.16716) · [PDF](https://arxiv.org/pdf/2103.16716) · [DOI 10.48550/arxiv.2103.16716](https://doi.org/10.48550/arxiv.2103.16716) · [arXiv 2103.16716](https://arxiv.org/abs/2103.16716) · [OpenAlex](https://openalex.org/W3147874613)

**Citations:** 61 citations (OpenAlex, 2026-10-14) · [116](https://md.research.elamur.ai/free/en/papers/lewis-2021-base-layers-simplifying-training-large/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Sparse expert models increase capacity cheaply by sending each token to one of many expert sub-networks. In practice they are hard to train because routing tends to become uneven. Earlier systems such as Sparsely Gated MoE and Switch counter this with auxiliary balancing losses, extra hyperparameters and capacity limits that drop tokens. Lewis et al. propose BASE layers, which treat routing during training as a linear assignment problem. Token–expert affinity scores are maximised under the constraint that every expert gets exactly the same number of tokens. Each expert's output is added to the token through a residual connection scaled by a sigmoid gate, so experts can specialise without extra loss terms. At inference, each token simply goes to its best-scoring expert. In language-model experiments with up to 110B parameters, BASE matches or beats dense data/model-parallel training and earlier sparse methods at equal compute, and it trains faster because workers communicate less. Its value is that it works as a drop-in conditional-compute layer with no new hyperparameters. Code is in fairseq.

## Key points
- The balanced assignment is computed with the auction algorithm (Bertsekas, 1992), chosen because it parallelises on GPUs more easily than the Hungarian algorithm. The problem is sharded so that each worker solves it for its own batch, and tokens are randomly shuffled across workers beforehand.
- Training throughput on 128 GPUs, in tokens per second:

  | Model | Tokens/s |
  |---|---|
  | Data Parallel | 600k |
  | BASE | 545k |
  | BASE ×2 | 475k |
  | Switch | 469k |
  | Sparsely Gated MoE | 292k |
  | Model Parallel ×2 | 224k |

- A 43B-parameter BASE model trains at roughly 90% of the speed of a 1.5B data-parallel baseline. About 1.5B is the largest model that fits in pure data parallelism in the authors' setup.
- A single large BASE layer is already effective, and adding more BASE layers improves results further.
- Experts tend to specialise on shallow, local cues such as the preceding token. Examples include experts for numbers, auxiliary verbs, possessives and specific noun or adjective groups.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 402 works · 212,762 citations
- [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 214 works · 241,492 citations
- [Meta (United Kingdom)](https://md.research.elamur.ai/free/en/companies/meta-united-kingdom/) — Company, London, United Kingdom · 54 works · 19,348 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal and Luke Zettlemoyer. 2021. BASE Layers: Simplifying Training of Large, Sparse Models. arXiv:2103.16716.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{lewis2021base,
  title = {{BASE Layers: Simplifying Training of Large, Sparse Models}},
  author = {Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal and Luke Zettlemoyer},
  year = {2021},
  doi = {10.48550/arxiv.2103.16716},
  eprint = {2103.16716},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2103.16716}
}
```

- [← previous The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) Shared Task](https://md.research.elamur.ai/free/en/papers/aly-2021-fact-extraction-verification-over-unstructured/)
- [next → ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts](https://md.research.elamur.ai/free/en/papers/koreeda-2021-contractnli-dataset-document-level-natural/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

31 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ba, J. L., Kiros, J. R., and Hinton, G. E. [Layer normalization](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/). arXiv preprint arXiv:1607.06450, 2016.
2. Bapna, A., Arivazhagan, N., and Firat, O. [Simple, scalable adaptation for neural machine translation](https://md.research.elamur.ai/free/en/papers/bapna-2019-simple-scalable-adaptation-neural-machine/). arXiv preprint arXiv:1909.08478, 2019.
3. Bertsekas, D. P. Auction algorithms for network flow problems: A tutorial introduction. Computational optimization and applications, 1(1):7–66, 1992. [find](https://scholar.google.com/scholar?q=Bertsekas%2C+D%2E+P%2E+Auction+algorithms+for+network+flow+problems%3A+A+tutorial+introduction%2E+Computational+optimization+and+applications%2C+1%281%29%3A7%E2%80%9366%2C+1992%2E)
4. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). arXiv preprint arXiv:2005.14165, 2020.
5. Child, R., Gray, S., Radford, A., and Sutskever, I. [Generating long sequences with sparse transformers](https://md.research.elamur.ai/free/en/papers/child-2019-generating-long-sequences-sparse-transformers/). arXiv preprint arXiv:1904.10509, 2019.
6. Lample, G., Sablayrolles, A., Ranzato, M., Denoyer, L., and Jégou, H. Large memory layers with product keys. arXiv preprint arXiv:1907.05242, 2019. [find](https://scholar.google.com/scholar?q=Lample%2C+G%2E%2C+Sablayrolles%2C+A%2E%2C+Ranzato%2C+M%2E%2C+Denoyer%2C+L%2E%2C+and+Je%CC%81gou%2C+H%2E+Large+memory+layers+with+product+keys%2E+arXiv+preprint+arXiv%3A1907%2E05242%2C+2019%2E)
7. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. [Unsupervised cross-lingual representation learning at scale](https://md.research.elamur.ai/free/en/papers/conneau-2020-unsupervised-cross-lingual-representation-learning/). arXiv preprint arXiv:1911.02116, 2019.
8. Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020. [find](https://scholar.google.com/scholar?q=Lepikhin%2C+D%2E%2C+Lee%2C+H%2E%2C+Xu%2C+Y%2E%2C+Chen%2C+D%2E%2C+Firat%2C+O%2E%2C+Huang%2C+Y%2E%2C+Krikun%2C+M%2E%2C+Shazeer%2C+N%2E%2C+and+Chen%2C+Z%2E+Gshard%3A+Scaling+giant+models+with+conditional+computation+and+automatic+sharding%2E+arXiv+preprint+ar)
9. Correia, G. M., Niculae, V., and Martins, A. F. [Adaptively sparse transformers](https://md.research.elamur.ai/free/en/papers/correia-2019-adaptively-sparse-transformers/). arXiv preprint arXiv:1909.00015, 2019.
10. Dettmers, T. and Zettlemoyer, L. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.04840, 2019. [find](https://scholar.google.com/scholar?q=Dettmers%2C+T%2E+and+Zettlemoyer%2C+L%2E+Sparse+networks+from+scratch%3A+Faster+training+without+losing+performance%2E+arXiv+preprint+arXiv%3A1907%2E04840%2C+2019%2E)
11. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805, 2018.
12. Evci, U., Gale, T., Menick, J., Castro, P. S., and Elsen, E. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943– 2952. PMLR, 2020. [find](https://scholar.google.com/scholar?q=Evci%2C+U%2E%2C+Gale%2C+T%2E%2C+Menick%2C+J%2E%2C+Castro%2C+P%2E+S%2E%2C+and+Elsen%2C+E%2E+Rigging+the+lottery%3A+Making+all+tickets+winners%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+2943%E2%80%93+2952%2E+PMLR%2C+2020%2E)
13. Fan, A., Bhosale, S., Schwenk, H., Ma, Z., El-Kishky, A., Goyal, S., Baines, M., Celebi, O., Wenzek, G., Chaudhary, V., et al. Beyond english-centric multilingual machine translation. arXiv preprint arXiv:2010.11125, 2020. [find](https://scholar.google.com/scholar?q=Fan%2C+A%2E%2C+Bhosale%2C+S%2E%2C+Schwenk%2C+H%2E%2C+Ma%2C+Z%2E%2C+El%2DKishky%2C+A%2E%2C+Goyal%2C+S%2E%2C+Baines%2C+M%2E%2C+Celebi%2C+O%2E%2C+Wenzek%2C+G%2E%2C+Chaudhary%2C+V%2E%2C+et+al%2E+Beyond+english%2Dcentric+multilingual+machine+translation%2E+arXiv+preprint+a)
14. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. [Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). arXiv preprint arXiv:1910.13461, 2019.
15. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692, 2019.
16. Mostafa, H. and Wang, X. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pp. 4646–4655. PMLR, 2019. [find](https://scholar.google.com/scholar?q=Mostafa%2C+H%2E+and+Wang%2C+X%2E+Parameter+efficient+training+of+deep+convolutional+neural+networks+by+dynamic+sparse+reparameterization%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+4646%E2%80%934655%2E+PMLR%2C+)
17. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+Child%2C+R%2E%2C+Luan%2C+D%2E%2C+Amodei%2C+D%2E%2C+and+Sutskever%2C+I%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2C+2019%2E)
18. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). arXiv preprint arXiv:1910.10683, 2019.
19. Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961, 2021. [find](https://scholar.google.com/scholar?q=Fedus%2C+W%2E%2C+Zoph%2C+B%2E%2C+and+Shazeer%2C+N%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+arXiv+preprint+arXiv%3A2101%2E03961%2C+2021%2E)
20. Roy, A., Saffar, M., Vaswani, A., and Grangier, D. [Efficient content-based sparse attention with routing transformers](https://md.research.elamur.ai/free/en/papers/roy-2020-efficient-content-based-sparse-attention/). arXiv preprint arXiv:2003.05997, 2020.
21. Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790–2799. PMLR, 2019. [find](https://scholar.google.com/scholar?q=Houlsby%2C+N%2E%2C+Giurgiu%2C+A%2E%2C+Jastrzebski%2C+S%2E%2C+Morrone%2C+B%2E%2C+De+Laroussilhe%2C+Q%2E%2C+Gesmundo%2C+A%2E%2C+Attariyan%2C+M%2E%2C+and+Gelly%2C+S%2E+Parameter%2Defficient+transfer+learning+for+nlp%2E+In+International+Conference+on+Mac)
22. Sennrich, R., Haddow, B., and Birch, A. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). arXiv preprint arXiv:1508.07909, 2015.
23. Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., and Lewis, M. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00172, 2019. [find](https://scholar.google.com/scholar?q=Khandelwal%2C+U%2E%2C+Levy%2C+O%2E%2C+Jurafsky%2C+D%2E%2C+Zettlemoyer%2C+L%2E%2C+and+Lewis%2C+M%2E+Generalization+through+memorization%3A+Nearest+neighbor+language+models%2E+arXiv+preprint+arXiv%3A1911%2E00172%2C+2019%2E)
24. Khandelwal, U., Fan, A., Jurafsky, D., Zettlemoyer, L., and Lewis, M. Nearest neighbor machine translation. arXiv preprint arXiv:2010.00710, 2020. [find](https://scholar.google.com/scholar?q=Khandelwal%2C+U%2E%2C+Fan%2C+A%2E%2C+Jurafsky%2C+D%2E%2C+Zettlemoyer%2C+L%2E%2C+and+Lewis%2C+M%2E+Nearest+neighbor+machine+translation%2E+arXiv+preprint+arXiv%3A2010%2E00710%2C+2020%2E)
25. Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017. [find](https://scholar.google.com/scholar?q=Shazeer%2C+N%2E%2C+Mirhoseini%2C+A%2E%2C+Maziarz%2C+K%2E%2C+Davis%2C+A%2E%2C+Le%2C+Q%2E%2C+Hinton%2C+G%2E%2C+and+Dean%2C+J%2E+Outrageously+large+neural+networks%3A+The+sparsely%2Dgated+mixture%2Dof%2Dexperts+layer%2E+arXiv+preprint+arXiv%3A1701%2E06538%2C+)
26. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. [find](https://scholar.google.com/scholar?q=Shoeybi%2C+M%2E%2C+Patwary%2C+M%2E%2C+Puri%2C+R%2E%2C+LeGresley%2C+P%2E%2C+Casper%2C+J%2E%2C+and+Catanzaro%2C+B%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2E+arXiv+preprint+arXiv%3A1909%2E08053%2C)
27. Kingma, D. P. and Ba, J. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980, 2014.
28. Strubell, E., Ganesh, A., and McCallum, A. [Energy and policy considerations for deep learning in nlp](https://md.research.elamur.ai/free/en/papers/strubell-2019-energy-policy-considerations-deep-learning/). arXiv preprint arXiv:1906.02243, 2019.
29. Kuhn, H. W. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83– 97, 1955. [find](https://scholar.google.com/scholar?q=Kuhn%2C+H%2E+W%2E+The+hungarian+method+for+the+assignment+problem%2E+Naval+research+logistics+quarterly%2C+2%281%2D2%29%3A83%E2%80%93+97%2C+1955%2E)
30. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). arXiv preprint arXiv:1706.03762, 2017.
31. Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., and Liu, T. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pp. 10524– 10533. PMLR, 2020. def balanced_assignment(scores, max_iterations=100): num_worke… [find](https://scholar.google.com/scholar?q=Xiong%2C+R%2E%2C+Yang%2C+Y%2E%2C+He%2C+D%2E%2C+Zheng%2C+K%2E%2C+Zheng%2C+S%2E%2C+Xing%2C+C%2E%2C+Zhang%2C+H%2E%2C+Lan%2C+Y%2E%2C+Wang%2C+L%2E%2C+and+Liu%2C+T%2E+On+layer+normalization+in+the+transformer+architecture%2E+In+International+Conference+on+Machine+Le)

*Updated 2 November 2026*