---
title: "Neural Machine Translation of Rare Words with Subword Units — Sennrich 2016"
description: "The authors propose a different approach: segmenting rare words into subword units so that the model… 1.3 BLEU. 36 references, cited by 3,409 papers here."
authors: ["Rico Sennrich", "Barry Haddow", "Alexandra Birch"]
year: 2016
venue: "Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/P16-1162/"
doi: "10.18653/v1/p16-1162"
arxiv: "1508.07909"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/) / Neural Machine Translation of Rare Words with Subword Units

# Neural Machine Translation of Rare Words with Subword Units

**Authors:** [Rico Sennrich](https://md.research.elamur.ai/free/en/authors/rico-sennrich/), [Barry Haddow](https://md.research.elamur.ai/free/en/authors/barry-haddow/) and [Alexandra Birch](https://md.research.elamur.ai/free/en/authors/alexandra-birch/) · **Year:** 2016 · **Venue:** Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P16-1162/) · [PDF](https://aclanthology.org/P16-1162.pdf) · [DOI 10.18653/v1/p16-1162](https://doi.org/10.18653/v1/p16-1162) · [arXiv 1508.07909](https://arxiv.org/abs/1508.07909)

**Citations:** [3409](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary

Neural machine translation (NMT) systems typically have a closed vocabulary of 30–50 thousand words. As a result, they struggle with rare and unseen words such as compounds, names, and loanwords. Previously, such words were handled separately via a back-off dictionary. The authors propose a different approach: segmenting rare words into subword units so that the model itself can translate any vocabulary. They compare several segmentation methods, including character bigrams. Their best-known contribution is an adaptation of the byte pair encoding (BPE) compression algorithm. It iteratively merges the most frequent pairs of symbols into new units, yielding a fixed-size vocabulary of variable-length pieces. The system was evaluated on the WMT 15 English→German and English→Russian tasks using the Bahdanau et al. architecture. The subword models outperformed the back-off dictionary baseline by 1.1 and 1.3 BLEU, respectively. They also translate rare words better and can produce new ones. The work is important because BPE became the standard tokenization method in NMT and later in large language models. It is especially useful for morphologically rich languages such as Ukrainian.

## Key points

- The only hyperparameter of BPE is the number of merge operations. The vocabulary size equals the number of initial symbols plus the number of merges. A special end-of-word symbol makes it possible to restore the original tokenization.
- The authors manually analyzed 100 rare German tokens. Most of them can be translated piece by piece: 56 compounds, 21 names, 6 loanwords, and 5 cases of transparent affixation.
- English→Russian, newstest2015, ensemble of 8 models: the WDict baseline scored 22.8 BLEU, while BPE-J90k and C2-50k each scored 24.1. On CHR F3, WDict scored 51.0 and BPE-J90k 53.0.
- Joint BPE segmentation for both languages (BPE-J90k) segments names more consistently and therefore transliterates them better. For out-of-vocabulary (OOV) words in English→Russian, unigram F1 rose from 6.6% (WDict) to 18.3%.
- The models are robust to oversegmentation and learn to translate even when segment boundaries do not coincide with morpheme boundaries.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/)

## Cite

Cite as: Rico Sennrich, Barry Haddow and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sennrich2016neural,
  title = {{Neural Machine Translation of Rare Words with Subword Units}},
  author = {Rico Sennrich, Barry Haddow and Alexandra Birch},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1162},
  eprint = {1508.07909},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p16-1162}
}
```

- [← previous Categorical Reparameterization with Gumbel-Softmax](https://md.research.elamur.ai/free/en/papers/jang-2016-categorical-reparameterization-gumbel-softmax/)
- [next → A Diversity-Promoting Objective Function for Neural Conversation Models](https://md.research.elamur.ai/free/en/papers/li-2016-diversity-promoting-objective-function-neural/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/)

## References

36 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In Proceedings of the International Conference on Learning Representations (ICLR). [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
2. Issam Bazzi and James R. Glass. 2000. Modeling out-of-vocabulary words for robust speech recognition. In Sixth International Conference on Spoken Language Processing, ICSLP 2000 / INTERSPEECH 2000, pages 401–404, Beijing, China. [find](https://scholar.google.com/scholar?q=Issam+Bazzi+and+James+R%2E+Glass%2E+2000%2E+Modeling+out%2Dof%2Dvocabulary+words+for+robust+speech+recognition%2E+In+Sixth+International+Conference+on+Spoken+Language+Processing%2C+ICSLP+2000+%2F+INTERSPEECH+2000%2C+pa)
3. Jan A. Botha and Phil Blunsom. 2014. Compositional Morphology for Word Representations and Language Modelling. In Proceedings of the 31st International Conference on Machine Learning (ICML), Beijing, China. [find](https://scholar.google.com/scholar?q=Jan+A%2E+Botha+and+Phil+Blunsom%2E+2014%2E+Compositional+Morphology+for+Word+Representations+and+Language+Modelling%2E+In+Proceedings+of+the+31st+International+Conference+on+Machine+Learning+%28ICML%29%2C+Beijing%2C+)
4. Rohan Chitnis and John DeNero. 2015. Variable-Length Word Encodings for Neural Translation Models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Rohan+Chitnis+and+John+DeNero%2E+2015%2E+Variable%2DLength+Word+Encodings+for+Neural+Translation+Models%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E)
5. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder– Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Langu… [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2C+Bart+van+Merrienboer%2C+Caglar+Gulcehre%2C+Dzmitry+Bahdanau%2C+Fethi+Bougares%2C+Holger+Schwenk%2C+and+Yoshua+Bengio%2E+2014%2E+Learning+Phrase+Representations+using+RNN+Encoder%E2%80%93+Decoder+for+Statisti)
6. Mathias Creutz and Krista Lagus. 2002. Unsupervised Discovery of Morphemes. In Proceedings of the ACL-02 Workshop on Morphological and Phonological Learning, pages 21–30. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mathias+Creutz+and+Krista+Lagus%2E+2002%2E+Unsupervised+Discovery+of+Morphemes%2E+In+Proceedings+of+the+ACL%2D02+Workshop+on+Morphological+and+Phonological+Learning%2C+pages+21%E2%80%9330%2E+Association+for+Computational)
7. Nadir Durrani, Hassan Sajjad, Hieu Hoang, and Philipp Koehn. 2014. Integrating an Unsupervised Transliteration Model into Statistical Machine Translation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2014, pages 148–153, Gothenb… [find](https://scholar.google.com/scholar?q=Nadir+Durrani%2C+Hassan+Sajjad%2C+Hieu+Hoang%2C+and+Philipp+Koehn%2E+2014%2E+Integrating+an+Unsupervised+Transliteration+Model+into+Statistical+Machine+Translation%2E+In+Proceedings+of+the+14th+Conference+of+the+)
8. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A Simple, Fast, and Effective Reparameterization of IBM Model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648, Atlanta, Georgia.… [find](https://scholar.google.com/scholar?q=Chris+Dyer%2C+Victor+Chahuneau%2C+and+Noah+A%2E+Smith%2E+2013%2E+A+Simple%2C+Fast%2C+and+Effective+Reparameterization+of+IBM+Model+2%2E+In+Proceedings+of+the+2013+Conference+of+the+North+American+Chapter+of+the+Assoc)
9. Philip Gage. 1994. A New Algorithm for Data Compression. C Users J., 12(2):23–38, February. [find](https://scholar.google.com/scholar?q=Philip+Gage%2E+1994%2E+A+New+Algorithm+for+Data+Compression%2E+C+Users+J%2E%2C+12%282%29%3A23%E2%80%9338%2C+February%2E)
10. Barry Haddow, Matthias Huck, Alexandra Birch, Nikolay Bogoychev, and Philipp Koehn. 2015. The Edinburgh/JHU Phrase-based Machine Translation Systems for WMT 2015. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 126–133, Lisbon, Portugal. Association for Computational L… [find](https://scholar.google.com/scholar?q=Barry+Haddow%2C+Matthias+Huck%2C+Alexandra+Birch%2C+Nikolay+Bogoychev%2C+and+Philipp+Koehn%2E+2015%2E+The+Edinburgh%2FJHU+Phrase%2Dbased+Machine+Translation+Systems+for+WMT+2015%2E+In+Proceedings+of+the+Tenth+Workshop+)
11. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On Using Very Large Target Vocabulary for Neural Machine Translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Pr… [our card](https://md.research.elamur.ai/free/en/papers/jean-2015-very-large-target-vocabulary-neural/)
12. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent Continuous Translation Models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2013-recurrent-continuous-translation-models/)
13. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. 2015. Character-Aware Neural Language Models. CoRR, abs/1508.06615. [our card](https://md.research.elamur.ai/free/en/papers/kim-2015-character-aware-neural-language-models/)
14. Philipp Koehn and Kevin Knight. 2003. Empirical Methods for Compound Splitting. In EACL ’03: Proceedings of the Tenth Conference on European Chapter of the Association for Computational Linguistics, pages 187–193, Budapest, Hungary. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/koehn-2003-empirical-methods-compound-splitting/)
15. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Pro… [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Hieu+Hoang%2C+Alexandra+Birch%2C+Chris+Callison%2DBurch%2C+Marcello+Federico%2C+Nicola+Bertoldi%2C+Brooke+Cowan%2C+Wade+Shen%2C+Christine+Moran%2C+Richard+Zens%2C+Chris+Dyer%2C+Ondr%CC%8Cej+Bojar%2C+Alexandra+Const)
16. Franklin M. Liang. 1983. Word hy-phen-a-tion by com-put-er. Ph.D. thesis, Stanford University, Department of Linguistics, Stanford, CA. [find](https://scholar.google.com/scholar?q=Franklin+M%2E+Liang%2E+1983%2E+Word+hy%2Dphen%2Da%2Dtion+by+com%2Dput%2Der%2E+Ph%2ED%2E+thesis%2C+Stanford+University%2C+Department+of+Linguistics%2C+Stanford%2C+CA%2E)
17. Wang Ling, Chris Dyer, Alan W. Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015a. Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Pr… [our card](https://md.research.elamur.ai/free/en/papers/ling-2015-finding-function-form-compositional-character/)
18. Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W. Black. 2015b. Character-based Neural Machine Translation. ArXiv e-prints, November. [our card](https://md.research.elamur.ai/free/en/papers/jussa-2016-character-based-neural-machine-translation/)
19. Thang Luong, Richard Socher, and Christopher D. Manning. 2013. Better Word Representations with Recursive Neural Networks for Morphology. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, CoNLL 2013, Sofia, Bulgaria, August 8-9, 2013, pages 104– 113. [our card](https://md.research.elamur.ai/free/en/papers/luong-2013-better-word-representations-recursive-neural/)
20. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015a. Effective Approaches to Attention-based Neural Machine Translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412– 1421, Lisbon, Portugal. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-effective-approaches-attention-based-neural/)
21. Thang Luong, Ilya Sutskever, Quoc Le, Oriol Vinyals, and Wojciech Zaremba. 2015b. Addressing the Rare Word Problem in Neural Machine Translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language… [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-addressing-rare-word-problem-neural/)
22. Tomas Mikolov, Ilya Sutskever, Anoop Deoras, Hai-Son Le, Stefan Kombrink, and Jan Cernocký. 2012. Subword Language Modeling with Neural Networks. Unpublished. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Ilya+Sutskever%2C+Anoop+Deoras%2C+Hai%2DSon+Le%2C+Stefan+Kombrink%2C+and+Jan+Cernock%C3%BD%2E+2012%2E+Subword+Language+Modeling+with+Neural+Networks%2E+Unpublished%2E)
23. Graham Neubig, Taro Watanabe, Shinsuke Mori, and Tatsuya Kawahara. 2012. Machine Translation without Words through Substring Alignment. In The 50th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, July 8-14, 2012, Jeju Island, Korea - Volume 1: Long Pap… [find](https://scholar.google.com/scholar?q=Graham+Neubig%2C+Taro+Watanabe%2C+Shinsuke+Mori%2C+and+Tatsuya+Kawahara%2E+2012%2E+Machine+Translation+without+Words+through+Substring+Alignment%2E+In+The+50th+Annual+Meeting+of+the+Association+for+Computational+)
24. Sonja Nießen and Hermann Ney. 2000. Improving SMT quality with morpho-syntactic analysis. In 18th Int. Conf. on Computational Linguistics, pages 1081–1085. [find](https://scholar.google.com/scholar?q=Sonja+Nie%C3%9Fen+and+Hermann+Ney%2E+2000%2E+Improving+SMT+quality+with+morpho%2Dsyntactic+analysis%2E+In+18th+Int%2E+Conf%2E+on+Computational+Linguistics%2C+pages+1081%E2%80%931085%2E)
25. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, pages 1310–1318, Atlanta, USA. [find](https://scholar.google.com/scholar?q=Razvan+Pascanu%2C+Tomas+Mikolov%2C+and+Yoshua+Bengio%2E+2013%2E+On+the+difficulty+of+training+recurrent+neural+networks%2E+In+Proceedings+of+the+30th+International+Conference+on+Machine+Learning%2C+ICML+2013%2C+pag)
26. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/)
27. Jörg Tiedemann. 2012. Character-Based Pivot Translation for Under-Resourced Languages and Domains. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 141–151, Avignon, France. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=J%C3%B6rg+Tiedemann%2E+2012%2E+Character%2DBased+Pivot+Translation+for+Under%2DResourced+Languages+and+Domains%2E+In+Proceedings+of+the+13th+Conference+of+the+European+Chapter+of+the+Association+for+Computational+Li)
28. David Vilar, Jan-Thorsten Peter, and Hermann Ney. 2007. Can We Translate Letters? In Second Workshop on Statistical Machine Translation, pages 33– 39, Prague, Czech Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=David+Vilar%2C+Jan%2DThorsten+Peter%2C+and+Hermann+Ney%2E+2007%2E+Can+We+Translate+Letters%3F+In+Second+Workshop+on+Statistical+Machine+Translation%2C+pages+33%E2%80%93+39%2C+Prague%2C+Czech+Republic%2E+Association+for+Computati)
29. Sami Virpioja, Jaakko J. Väyrynen, Mathias Creutz, and Markus Sadeniemi. 2007. Morphology-Aware Statistical Machine Translation Based on Morphs Induced in an Unsupervised Manner. In Proceedings of the Machine Translation Summit XI, pages 491–498, Copenhagen, Denmark. [find](https://scholar.google.com/scholar?q=Sami+Virpioja%2C+Jaakko+J%2E+V%C3%A4yrynen%2C+Mathias+Creutz%2C+and+Markus+Sadeniemi%2E+2007%2E+Morphology%2DAware+Statistical+Machine+Translation+Based+on+Morphs+Induced+in+an+Unsupervised+Manner%2E+In+Proceedings+of+the)
30. Matthew D. Zeiler. 2012. ADADELTA: An Adaptive Learning Rate Method. CoRR, abs/1212.5701. [our card](https://md.research.elamur.ai/free/en/papers/zeiler-2012-adadelta-adaptive-learning-rate-method/)
31. Rico Sennrich and Barry Haddow. 2015. A Joint Dependency Model of Morphological and Syntactic Structure for Statistical Machine Translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2081–2087, Lisbon, Portugal. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Rico+Sennrich+and+Barry+Haddow%2E+2015%2E+A+Joint+Dependency+Model+of+Morphological+and+Syntactic+Structure+for+Statistical+Machine+Translation%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+)
32. Benjamin Snyder and Regina Barzilay. 2008. Unsupervised Multilingual Learning for Morphological Segmentation. In Proceedings of ACL-08: HLT, pages 737–745, Columbus, Ohio. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/snyder-2008-unsupervised-multilingual-learning-morphological-segmentation/)
33. David Stallard, Jacob Devlin, Michael Kayser, Yoong Keok Lee, and Regina Barzilay. 2012. Unsupervised Morphology Rivals Supervised Morphology for Arabic MT. In The 50th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, July 8-14, 2012, Jeju Island, Korea… [our card](https://md.research.elamur.ai/free/en/papers/stallard-2012-unsupervised-morphology-rivals-supervised-morphology/)
34. Miloš Stanojević, Amir Kamran, Philipp Koehn, and Ondřej Bojar. 2015. Results of the WMT15 Metrics Shared Task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 256– 273, Lisbon, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Milo%C5%A1+Stanojevic%CC%81%2C+Amir+Kamran%2C+Philipp+Koehn%2C+and+Ondr%CC%8Cej+Bojar%2E+2015%2E+Results+of+the+WMT15+Metrics+Shared+Task%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+pages+256%E2%80%93+27)
35. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, pages 3104–3112, Montreal, Quebec, Canada. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V%2E+Le%2E+2014%2E+Sequence+to+Sequence+Learning+with+Neural+Networks%2E+In+Advances+in+Neural+Information+Processing+Systems+27%3A+Annual+Conference+on+Neural+Informatio)
36. Jörg Tiedemann. 2009. Character-based PSMT for Closely Related Languages. In Proceedings of 13th Annual Conference of the European Association for Machine Translation (EAMT’09), pages 12–19. [find](https://scholar.google.com/scholar?q=J%C3%B6rg+Tiedemann%2E+2009%2E+Character%2Dbased+PSMT+for+Closely+Related+Languages%2E+In+Proceedings+of+13th+Annual+Conference+of+the+European+Association+for+Machine+Translation+%28EAMT%E2%80%9909%29%2C+pages+12%E2%80%9319%2E)