---
title: "Adaptive Input Representations for Neural Language Modeling"
description: "Word-level neural language models with very large vocabularies spend many… 10.5 points. 34 references, cited by 129 papers here."
authors: ["Alexei Baevski", "Michael Auli"]
year: 2018
venue: "arXiv:1809.10853"
source: "https://arxiv.org/abs/1809.10853"
doi: "10.48550/arxiv.1809.10853"
arxiv: "1809.10853"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/baevski-2018-adaptive-input-representations-neural-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/baevski-2018-adaptive-input-representations-neural-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/) / Adaptive Input Representations for Neural Language Modeling

# Adaptive Input Representations for Neural Language Modeling

**Authors:** [Alexei Baevski](https://md.research.elamur.ai/free/en/authors/alexei-baevski/) and [Michael Auli](https://md.research.elamur.ai/free/en/authors/michael-auli/) · **Year:** 2018 · **Venue:** arXiv:1809.10853 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1809.10853) · [PDF](https://arxiv.org/pdf/1809.10853) · [DOI 10.48550/arxiv.1809.10853](https://doi.org/10.48550/arxiv.1809.10853) · [arXiv 1809.10853](https://arxiv.org/abs/1809.10853) · [OpenAlex](https://openalex.org/W2894175714)

**Citations:** 48 citations (OpenAlex, 2026-10-14) · [129](https://md.research.elamur.ai/free/en/papers/baevski-2018-adaptive-input-representations-neural-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Word-level neural language models with very large vocabularies spend many parameters and much computation on embedding layers, and rare words tend to be overfit. Baevski and Auli carry the idea behind the adaptive softmax of Grave et al. over to the input side. The vocabulary is sorted by frequency and split into clusters. Frequent words receive wide embeddings, and each later cluster is narrower by a factor k (typically 4). Every cluster is then linearly projected to a shared model dimension. These adaptive input embeddings can share weights with an adaptive softmax output layer. The authors compare word, character-CNN and BPE sub-word factorizations within a decoder-only Transformer, using the BILLION WORD and WIKITEXT-103 benchmarks. Adaptive inputs need fewer parameters than the character CNN, train more than twice as fast, and reach better perplexity. The resulting models set new state-of-the-art scores on both benchmarks. The work shows that allocating embedding capacity according to word frequency is a cheap and effective way to scale word-level language models. Code and models are released in fairseq.

## Key points
- **Results:** WIKITEXT-103 test perplexity of 18.7, which is 10.5 points better than the prior best. BILLION WORD perplexity of 23.02, nearly 5 points better than the previous best.
- **Parameter savings:** On BILLION WORD, adaptive inputs shrink the input and output layers by 23% compared with fixed-size embeddings. Tying them with an adaptive softmax raises the total reduction to 61%.
- **Architecture:** The decoder-only Transformer uses 16 blocks with 16 attention heads, model width 1024 and FFN size 4096. Layer norm is placed before each sub-block. An extra dropout in the adaptive softmax tail projection improves results.
- **Datasets:** BILLION WORD has about 768M tokens and an ~800K vocabulary. WIKITEXT-103 has about 100M tokens and a ~260K vocabulary.
- **Weight sharing:** On WIKITEXT-103, sharing both the embeddings and the tail projections works best. On BILLION WORD, sharing only the embeddings performs better.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Meta (Israel)](https://md.research.elamur.ai/free/en/companies/meta-israel/) — Company, Tel Aviv, Israel · 46 works · 42,810 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Alexei Baevski and Michael Auli. 2018. Adaptive Input Representations for Neural Language Modeling. arXiv:1809.10853.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{baevski2018adaptive,
  title = {{Adaptive Input Representations for Neural Language Modeling}},
  author = {Alexei Baevski and Michael Auli},
  year = {2018},
  doi = {10.48550/arxiv.1809.10853},
  eprint = {1809.10853},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1809.10853}
}
```

- [← previous Integrating Transformer and Paraphrase Rules for Sentence Simplification](https://md.research.elamur.ai/free/en/papers/zhao-2018-integrating-transformer-paraphrase-rules-sentence/)
- [next → The IIT Bombay English-Hindi Parallel Corpus](https://md.research.elamur.ai/free/en/papers/kunchukuttan-2018-iit-bombay-english-hindi-parallel/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/)

## References

34 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. [Character-level language modeling with deeper self-attention](https://md.research.elamur.ai/free/en/papers/al-rfou-2018-character-level-language-modeling-deeper/). arXiv, abs/1808.04444, 2018.
2. Ebru Arisoy, Tara N. Sainath, Brian Kingsbury, and Bhuvana Ramabhadran. Deep Neural Network Language Models. In NAACL-HLT Workshop on the Future of Language Modeling for HLT, 2012. [find](https://scholar.google.com/scholar?q=Ebru+Arisoy%2C+Tara+N%2E+Sainath%2C+Brian+Kingsbury%2C+and+Bhuvana+Ramabhadran%2E+Deep+Neural+Network+Language+Models%2E+In+NAACL%2DHLT+Workshop+on+the+Future+of+Language+Modeling+for+HLT%2C+2012%2E)
3. Paul Baltescu and Phil Blunsom. Pragmatic neural language modelling in machine translation. In Proc. of ACL, 2015. [find](https://scholar.google.com/scholar?q=Paul+Baltescu+and+Phil+Blunsom%2E+Pragmatic+neural+language+modelling+in+machine+translation%2E+In+Proc%2E+of+ACL%2C+2015%2E)
4. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. [A Neural Probabilistic Language Model](https://md.research.elamur.ai/free/en/papers/bengio-2003-neural-probabilistic-language-model/). JMLR, 3:1137–1155, 2003.
5. Jacob Buckman and Graham Neubig. [Neural lattice language models](https://md.research.elamur.ai/free/en/papers/buckman-2018-neural-lattice-language-models/). TACL, 6:529–541, 2018.
6. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. [One billion word benchmark for measuring progress in statistical language modeling](https://md.research.elamur.ai/free/en/papers/chelba-2013-one-billion-word-benchmark-measuring/). Technical report, Google, 2013.
7. Wenlin Chen, David Grangier, and Michael Auli. Strategies for training large vocabulary neural language models. In Proc. of ACL, 2016. [find](https://scholar.google.com/scholar?q=Wenlin+Chen%2C+David+Grangier%2C+and+Michael+Auli%2E+Strategies+for+training+large+vocabulary+neural+language+models%2E+In+Proc%2E+of+ACL%2C+2016%2E)
8. Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. [Language modeling with gated convolutional networks](https://md.research.elamur.ai/free/en/papers/dauphin-2016-language-modeling-gated-convolutional-networks/). In Proc. of ICML, 2017.
9. Joshua Goodman. Classes for Fast Maximum Entropy Training. In Proc. of ICASSP, 2001. [find](https://scholar.google.com/scholar?q=Joshua+Goodman%2E+Classes+for+Fast+Maximum+Entropy+Training%2E+In+Proc%2E+of+ICASSP%2C+2001%2E)
10. Edouard Grave, Armand Joulin, and Nicolas Usunier. [Improving neural language models with a continuous cache](https://md.research.elamur.ai/free/en/papers/grave-2016-improving-neural-language-models-continuous/). arXiv, abs/1612.04426, 2016.
11. Edouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, and Hervé Jégou. [Efficient softmax approximation for gpus](https://md.research.elamur.ai/free/en/papers/grave-2016-efficient-softmax-approximation-gpus/). In Proc. of ICML, 2017.
12. Tail dropout Validation perplexity N/A 0.0 0.2 23.87 24.74 21.23 Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. [Deep Residual Learning for Image Recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In Proc. of CVPR, 2015.
13. Hakan Inan, Khashayar Khosravi, and Richard Socher. [Tying word vectors and word classifiers: A loss framework for language modeling](https://md.research.elamur.ai/free/en/papers/inan-2016-tying-word-vectors-word-classifiers/). arXiv, abs/1611.01462, 2016.
14. Rafal Józefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv, abs/1602.02410, 2016. [find](https://scholar.google.com/scholar?q=Rafal+Jo%CC%81zefowicz%2C+Oriol+Vinyals%2C+Mike+Schuster%2C+Noam+Shazeer%2C+and+Yonghui+Wu%2E+Exploring+the+limits+of+language+modeling%2E+arXiv%2C+abs%2F1602%2E02410%2C+2016%2E)
15. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. [Character-aware neural language models](https://md.research.elamur.ai/free/en/papers/kim-2015-character-aware-neural-language-models/). arXiv, abs/1508.06615, 2015.
16. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. [Character-aware neural language models](https://md.research.elamur.ai/free/en/papers/kim-2015-character-aware-neural-language-models/). In AAAI, pp. 2741–2749, 2016.
17. Ilya Loshchilov and Frank Hutter. abs/1608.03983, 2016. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+abs%2F1608%2E03983%2C+2016%2E)
18. SGDR: stochastic gradient descent with restarts. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. [Pointer sentinel mixture models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/). arXiv, abs/1609.07843, 2016.
19. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. An analysis of neural language modeling at multiple scales. arXiv, abs/1803.08240, 2018. [find](https://scholar.google.com/scholar?q=Stephen+Merity%2C+Nitish+Shirish+Keskar%2C+and+Richard+Socher%2E+An+analysis+of+neural+language+modeling+at+multiple+scales%2E+arXiv%2C+abs%2F1803%2E08240%2C+2018%2E)
20. Sebastian J. Mielke and Jason Eisner. Spell once, summon anywhere: A two-level open-vocabulary language model. arXiv, abs/1804.08205, 2018. — [arXiv:1804.08205](https://arxiv.org/abs/1804.08205) [find](https://scholar.google.com/scholar?q=Sebastian+J%2E+Mielke+and+Jason+Eisner%2E+Spell+once%2C+summon+anywhere%3A+A+two%2Dlevel+open%2Dvocabulary+language+model%2E+arXiv%2C+abs%2F1804%2E08205%2C+2018%2E+%E2%80%94+%5BarXiv%3A1804%2E08205%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1804%2E08205%29)
21. Tomáš Mikolov, Karafiát Martin, Lukáš Burget, Jan Cernocký, and Sanjeev Khudanpur. Recurrent Neural Network based Language Model. In Proc. of Interspeech, 2010. [find](https://scholar.google.com/scholar?q=Toma%CC%81s%CC%8C+Mikolov%2C+Karafia%CC%81t+Martin%2C+Luka%CC%81s%CC%8C+Burget%2C+Jan+Cernocky%CC%81%2C+and+Sanjeev+Khudanpur%2E+Recurrent+Neural+Network+based+Language+Model%2E+In+Proc%2E+of+Interspeech%2C+2010%2E)
22. Tomáš Mikolov, Stefan Kombrink, Lukáš Burget, Jan Cernocký, and Sanjeev Khudanpur. Extensions of Recurrent Neural Network Language Model. In Proc. of ICASSP, pp. 5528–5531, 2011. [find](https://scholar.google.com/scholar?q=Toma%CC%81s%CC%8C+Mikolov%2C+Stefan+Kombrink%2C+Luka%CC%81s%CC%8C+Burget%2C+Jan+Cernocky%CC%81%2C+and+Sanjeev+Khudanpur%2E+Extensions+of+Recurrent+Neural+Network+Language+Model%2E+In+Proc%2E+of+ICASSP%2C+pp%2E+5528%E2%80%935531%2C+2011%2E)
23. Frederic Morin and Yoshua Bengio. Hierarchical Probabilistic Neural Network Language Model. In Proc. of AISTATS, 2005. [find](https://scholar.google.com/scholar?q=Frederic+Morin+and+Yoshua+Bengio%2E+Hierarchical+Probabilistic+Neural+Network+Language+Model%2E+In+Proc%2E+of+AISTATS%2C+2005%2E)
24. Myle Ott, Michael Auli, David Grangier, and MarcAurelio Ranzato. [Analyzing uncertainty in neural machine translation](https://md.research.elamur.ai/free/en/papers/ott-2018-analyzing-uncertainty-neural-machine-translation/). In Proc. of ICML, 2018a.
25. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. [Scaling neural machine translation](https://md.research.elamur.ai/free/en/papers/ott-2018-scaling-neural-machine-translation/). In Proc. of WMT, 2018b.
26. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Proc. of ICML, 2013. [find](https://scholar.google.com/scholar?q=Razvan+Pascanu%2C+Tomas+Mikolov%2C+and+Yoshua+Bengio%2E+On+the+difficulty+of+training+recurrent+neural+networks%2E+In+Proc%2E+of+ICML%2C+2013%2E)
27. Ofir Press and Lior Wolf. [Using the output embedding to improve language models](https://md.research.elamur.ai/free/en/papers/press-2017-output-embedding-improve-language-models/). In Proc. of EACL, 2017.
28. Jack W. Rae, Chris Dyer, Peter Dayan, and Timothy P. Lillicrap. Fast parametric learning with activation memorization. arXiv, abs/1803.10049, 2018. — [arXiv:1803.10049](https://arxiv.org/abs/1803.10049) [find](https://scholar.google.com/scholar?q=Jack+W%2E+Rae%2C+Chris+Dyer%2C+Peter+Dayan%2C+and+Timothy+P%2E+Lillicrap%2E+Fast+parametric+learning+with+activation+memorization%2E+arXiv%2C+abs%2F1803%2E10049%2C+2018%2E+%E2%80%94+%5BarXiv%3A1803%2E10049%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1803%2E1004)
29. Holger Schwenk, Anthony Rousseau, and Mohammed Attik. Large, Pruned or Continuous Space Language Models on a GPU for Statistical Machine Translation. In NAACL-HLT Workshop on the Future of Language Modeling for HLT, 2012. [find](https://scholar.google.com/scholar?q=Holger+Schwenk%2C+Anthony+Rousseau%2C+and+Mohammed+Attik%2E+Large%2C+Pruned+or+Continuous+Space+Language+Models+on+a+GPU+for+Statistical+Machine+Translation%2E+In+NAACL%2DHLT+Workshop+on+the+Future+of+Language+Mo)
30. Rico Sennrich, Barry Haddow, and Alexandra Birch. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proc. of ACL, 2016.
31. Noam Shazeer, Joris Pelemans, and Ciprian Chelba. Sparse non-negative matrix language modeling. In Proc. of Interspeech, 2016. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2C+Joris+Pelemans%2C+and+Ciprian+Chelba%2E+Sparse+non%2Dnegative+matrix+language+modeling%2E+In+Proc%2E+of+Interspeech%2C+2016%2E)
32. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv, abs/1701.06538, 2017. arXiv, Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. O… [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2C+Azalia+Mirhoseini%2C+Krzysztof+Maziarz%2C+Andy+Davis%2C+Quoc+V%2E+Le%2C+Geoffrey+E%2E+Hinton%2C+and+Jeff+Dean%2E+Outrageously+large+neural+networks%3A+The+sparsely%2Dgated+mixture%2Dof%2Dexperts+layer%2E+arXiv%2C+a)
33. Ashish Vaswani, Yinggong Zhao, Victoria Fossum, and David Chiang. [Decoding with Large-scale Neural Language Models improves Translation](https://md.research.elamur.ai/free/en/papers/vaswani-2013-decoding-large-scale-neural-language/). In Proc. of EMNLP, October 2013.
34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. [Attention Is All You Need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Proc. of NIPS, 2017.

*Updated 2 November 2026*