---
title: "Achieving Open Vocabulary Neural Machine — Luong 2016"
description: "Neural machine translation systems usually work with a fixed, limited vocabulary."
authors: ["Minh-Thang Luong", "Christopher D. Manning"]
year: 2016
venue: "Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/P16-1100/"
doi: "10.18653/v1/p16-1100"
arxiv: "1604.00788"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/luong-2016-achieving-open-vocabulary-neural-machine/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/luong-2016-achieving-open-vocabulary-neural-machine/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/) / Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models

# Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models

**Authors:** [Minh-Thang Luong](https://md.research.elamur.ai/free/en/authors/minh-thang-luong/), [Christopher D. Manning](https://md.research.elamur.ai/free/en/authors/christopher-d-manning/) · **Year:** 2016 · **Venue:** Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P16-1100/) · [PDF](https://aclanthology.org/P16-1100.pdf) · [DOI 10.18653/v1/p16-1100](https://doi.org/10.18653/v1/p16-1100) · [arXiv 1604.00788](https://arxiv.org/abs/1604.00788) · [OpenAlex](https://openalex.org/W2339995566)

**Citations:** 345 citations (openalex-snapshot · 2026-10-14) · [124](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Neural machine translation systems usually work with a fixed, limited vocabulary. Words outside it are collapsed into a single \<unk> token and later patched with dictionary lookup or copying. That approach ignores word morphology and breaks down when scripts or spellings differ across languages. Luong and Manning propose a hybrid NMT model to address this. A deep LSTM encoder-decoder with global attention handles translation of frequent words at the word level. Character-level LSTMs cover everything else. On the source side, a character LSTM builds an embedding for any rare word on the fly. On the target side, a separate character decoder spells out the actual word whenever the word-level model emits \<unk>. Both parts are trained jointly, end-to-end, so no separate unk-replacement step is needed. On WMT'15 English→Czech, a hard case because Czech is heavily inflected, the hybrid system reaches 20.7 BLEU, a new state of the art. This shows that open-vocabulary translation can be both practical and efficient.

## Key points
- Task: WMT'15 English→Czech. Example translations are drawn from newstest2015.
- Best hybrid model scores 20.7 BLEU, beating the strongest earlier word-based NMT and non-neural systems. Character components add +2.1 to +11.4 BLEU over models that already use unk replacement.
- Training combines word-level and character-level losses (J = J_w + αJ_c). Source character LSTMs start from zero states, so rare-word embeddings can be precomputed once per word type for each mini-batch.
- A Barnes-Hut-SNE visualization shows that character-built English representations group words by both form and meaning (e.g., "acceptable" vs. "unacceptable").
- Purely character-based models also beat earlier word-level NMT. Code, data and models are released at nlp.stanford.edu/projects/nmt.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

124 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 117 not yet.

- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/)
- [What do Neural Machine Translation Models Learn about Morphology?](https://md.research.elamur.ai/free/en/papers/belinkov-2017-what-do-neural-machine-translation/)
- [Universal Neural Machine Translation for Extremely Low Resource Languages](https://md.research.elamur.ai/free/en/papers/gu-2018-universal-neural-machine-translation-extremely/)
- [BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages](https://md.research.elamur.ai/free/en/papers/heinzerling-2018-bpemb-tokenization-free-pre-trained/)
- [Fully Character-Level Neural Machine Translation without Explicit Segmentation](https://md.research.elamur.ai/free/en/papers/lee-2017-fully-character-level-neural-machine/)
- [Exploring Neural Text Simplification Models](https://md.research.elamur.ai/free/en/papers/nisioi-2017-exploring-neural-text-simplification-models/)
- [How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs](https://md.research.elamur.ai/free/en/papers/sennrich-2017-how-grammatical-is-character-level/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{luong2016achieving,
  title = {{Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models}},
  author = {Luong, Minh-Thang and Manning, Christopher D.},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1100},
  eprint = {1604.00788},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p16-1100}
}
```

- [← previous Abstractive Sentence Summarization with Attentive Recurrent Neural Networks](https://md.research.elamur.ai/free/en/papers/chopra-2016-abstractive-sentence-summarization-attentive-recurrent/)
- [next → Adapting Event Embedding for Implicit Discourse Relation Recognition](https://md.research.elamur.ai/free/en/papers/pacheco-2016-adapting-event-embedding-implicit-discourse/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/)

## References

30 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
2. Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, and Yoshua Bengio. 2016. End-to-end attention-based large vocabulary speech recognition. In ICASSP. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Jan+Chorowski%2C+Dmitriy+Serdyuk%2C+Philemon+Brakel%2C+and+Yoshua+Bengio%2E+2016%2E+End%2Dto%2Dend+attention%2Dbased+large+vocabulary+speech+recognition%2E+In+ICASSP%2E)
3. Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2015. Improved transition-based parsing by modeling characters instead of words with LSTMs. In EMNLP. [find](https://scholar.google.com/scholar?q=Miguel+Ballesteros%2C+Chris+Dyer%2C+and+Noah+A%2E+Smith%2E+2015%2E+Improved+transition%2Dbased+parsing+by+modeling+characters+instead+of+words+with+LSTMs%2E+In+EMNLP%2E)
4. Ondr̆ej Bojar and Ales̆ Tamchyna. 2015. CUNI in WMT15: Chimera Strikes Again. In WMT. [find](https://scholar.google.com/scholar?q=Ondr%CC%86ej+Bojar+and+Ales%CC%86+Tamchyna%2E+2015%2E+CUNI+in+WMT15%3A+Chimera+Strikes+Again%2E+In+WMT%2E)
5. William Chan, Navdeep Jaitly, Quoc V. Le, and Oriol Vinyals. 2016. Listen, attend and spell. In ICASSP. [find](https://scholar.google.com/scholar?q=William+Chan%2C+Navdeep+Jaitly%2C+Quoc+V%2E+Le%2C+and+Oriol+Vinyals%2E+2016%2E+Listen%2C+attend+and+spell%2E+In+ICASSP%2E)
6. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP. [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2C+Bart+van+Merrienboer%2C+Caglar+Gulcehre%2C+Fethi+Bougares%2C+Holger+Schwenk%2C+and+Yoshua+Bengio%2E+2014%2E+Learning+phrase+representations+using+RNN+encoder%2Ddecoder+for+statistical+machine+transla)
7. Cícero Nogueira dos Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-of-speech tagging. In ICML. [find](https://scholar.google.com/scholar?q=C%C3%ADcero+Nogueira+dos+Santos+and+Bianca+Zadrozny%2E+2014%2E+Learning+character%2Dlevel+representations+for+part%2Dof%2Dspeech+tagging%2E+In+ICML%2E)
8. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+J%C3%BCrgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+9%288%29%3A1735%E2%80%931780%2E)
9. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015a. On using very large target vocabulary for neural machine translation. In ACL. [find](https://scholar.google.com/scholar?q=S%C3%A9bastien+Jean%2C+Kyunghyun+Cho%2C+Roland+Memisevic%2C+and+Yoshua+Bengio%2E+2015a%2E+On+using+very+large+target+vocabulary+for+neural+machine+translation%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/jean-2015-very-large-target-vocabulary-neural/)
10. Sébastien Jean, Orhan Firat, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015b. Montreal neural machine translation systems for WMT’15. In WMT. [find](https://scholar.google.com/scholar?q=S%C3%A9bastien+Jean%2C+Orhan+Firat%2C+Kyunghyun+Cho%2C+Roland+Memisevic%2C+and+Yoshua+Bengio%2E+2015b%2E+Montreal+neural+machine+translation+systems+for+WMT%E2%80%9915%2E+In+WMT%2E)
11. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. [find](https://scholar.google.com/scholar?q=Rafal+Jozefowicz%2C+Oriol+Vinyals%2C+Mike+Schuster%2C+Noam+Shazeer%2C+and+Yonghui+Wu%2E+2016%2E+Exploring+the+limits+of+language+modeling%2E)
12. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP. [find](https://scholar.google.com/scholar?q=Nal+Kalchbrenner+and+Phil+Blunsom%2E+2013%2E+Recurrent+continuous+translation+models%2E+In+EMNLP%2E)
13. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. 2016. Character-aware neural language models. In AAAI. [find](https://scholar.google.com/scholar?q=Yoon+Kim%2C+Yacine+Jernite%2C+David+Sontag%2C+and+Alexander+M%2E+Rush%2E+2016%2E+Character%2Daware+neural+language+models%2E+In+AAAI%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2015-character-aware-neural-language-models/)
14. Jiwei Li, Minh-Thang Luong, and Dan Jurafsky. 2015. A hierarchical neural autoencoder for paragraphs and documents. In ACL. [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Minh%2DThang+Luong%2C+and+Dan+Jurafsky%2E+2015%2E+A+hierarchical+neural+autoencoder+for+paragraphs+and+documents%2E+In+ACL%2E)
15. Wang Ling, Isabel Trancoso, Chris Dyer, and Alan Black. 2015b. Character-based neural machine translation. [find](https://scholar.google.com/scholar?q=Wang+Ling%2C+Isabel+Trancoso%2C+Chris+Dyer%2C+and+Alan+Black%2E+2015b%2E+Character%2Dbased+neural+machine+translation%2E) [our card](https://md.research.elamur.ai/free/en/papers/jussa-2016-character-based-neural-machine-translation/)
16. Minh-Thang Luong and Christopher D. Manning. 2015. Stanford neural machine translation systems for spoken language domain. In IWSLT. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong+and+Christopher+D%2E+Manning%2E+2015%2E+Stanford+neural+machine+translation+systems+for+spoken+language+domain%2E+In+IWSLT%2E)
17. Minh-Thang Luong, Richard Socher, and Christopher D. Manning. 2013. Better word representations with recursive neural networks for morphology. In CoNLL. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Richard+Socher%2C+and+Christopher+D%2E+Manning%2E+2013%2E+Better+word+representations+with+recursive+neural+networks+for+morphology%2E+In+CoNLL%2E) [our card](https://md.research.elamur.ai/free/en/papers/luong-2013-better-word-representations-recursive-neural/)
18. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015a. Effective approaches to attention-based neural machine translation. In EMNLP. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Hieu+Pham%2C+and+Christopher+D%2E+Manning%2E+2015a%2E+Effective+approaches+to+attention%2Dbased+neural+machine+translation%2E+In+EMNLP%2E)
19. Minh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. 2015b. Addressing the rare word problem in neural machine translation. In ACL. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Ilya+Sutskever%2C+Quoc+V%2E+Le%2C+Oriol+Vinyals%2C+and+Wojciech+Zaremba%2E+2015b%2E+Addressing+the+rare+word+problem+in+neural+machine+translation%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-addressing-rare-word-problem-neural/)
20. Kishore Papineni, Salim Roukos, Todd Ward, and Wei jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei+jing+Zhu%2E+2002%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
21. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+D%2E+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+EMNLP%2E)
22. Vu Pham, Théodore Bluche, Christopher Kermorvant, and Jérôme Louradour. 2014. Dropout improves recurrent neural networks for handwriting recognition. In ICFHR. [find](https://scholar.google.com/scholar?q=Vu+Pham%2C+Th%C3%A9odore+Bluche%2C+Christopher+Kermorvant%2C+and+J%C3%A9r%C3%B4me+Louradour%2E+2014%2E+Dropout+improves+recurrent+neural+networks+for+handwriting+recognition%2E+In+ICFHR%2E)
23. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In WMT. [find](https://scholar.google.com/scholar?q=Maja+Popovic%CC%81%2E+2015%2E+chrF%3A+character+n%2Dgram+F%2Dscore+for+automatic+MT+evaluation%2E+In+WMT%2E) [our card](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/)
24. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+In+ACL%2E)
25. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In NIPS. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V%2E+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+NIPS%2E)
26. Laurens van der Maaten. 2013. Barnes-Hut-SNE. In ICLR. [find](https://scholar.google.com/scholar?q=Laurens+van+der+Maaten%2E+2013%2E+Barnes%2DHut%2DSNE%2E+In+ICLR%2E)
27. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. abs/1409.2329. [find](https://scholar.google.com/scholar?q=Wojciech+Zaremba%2C+Ilya+Sutskever%2C+and+Oriol+Vinyals%2E+2014%2E+Recurrent+neural+network+regularization%2E+abs%2F1409%2E2329%2E) [our card](https://md.research.elamur.ai/free/en/papers/zaremba-2014-recurrent-neural-network-regularization/)
28. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In NIPS. [find](https://scholar.google.com/scholar?q=Xiang+Zhang%2C+Junbo+Zhao%2C+and+Yann+LeCun%2E+2015%2E+Character%2Dlevel+convolutional+networks+for+text+classification%2E+In+NIPS%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/)
29. Percy Liang, Ben Taskar, and Dan Klein. 2006. Alignment by agreement. In NAACL. [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Ben+Taskar%2C+and+Dan+Klein%2E+2006%2E+Alignment+by+agreement%2E+In+NAACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/liang-2006-alignment-agreement/)
30. Wang Ling, Chris Dyer, Alan W. Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luís Marujo, and Tiago Luís. 2015a. Finding function in form: Compositional character models for open vocabulary word representation. In EMNLP. [find](https://scholar.google.com/scholar?q=Wang+Ling%2C+Chris+Dyer%2C+Alan+W%2E+Black%2C+Isabel+Trancoso%2C+Ramon+Fermandez%2C+Silvio+Amir%2C+Lu%C3%ADs+Marujo%2C+and+Tiago+Lu%C3%ADs%2E+2015a%2E+Finding+function+in+form%3A+Compositional+character+models+for+open+vocabulary+wo) [our card](https://md.research.elamur.ai/free/en/papers/ling-2015-finding-function-form-compositional-character/)