---
title: "Fully Character-Level Neural Machine Translation — Lee 2017"
description: "Word- and subword-level NMT systems rely on a separate segmentation step and handle rare or unseen word forms poorly."
authors: ["Jason Lee", "Kyunghyun Cho", "Thomas Hofmann"]
year: 2017
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/Q17-1026/"
doi: "10.1162/tacl_a_00067"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lee-2017-fully-character-level-neural-machine/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lee-2017-fully-character-level-neural-machine/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/2/) / Fully Character-Level Neural Machine Translation without Explicit Segmentation

# Fully Character-Level Neural Machine Translation without Explicit Segmentation

**Authors:** [Jason Lee](https://md.research.elamur.ai/free/en/authors/jason-lee/), [Kyunghyun Cho](https://md.research.elamur.ai/free/en/authors/kyunghyun-cho/), Thomas Hofmann · **Year:** 2017 · **Venue:** Transactions of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/Q17-1026/) · [PDF](https://aclanthology.org/Q17-1026.pdf) · [DOI 10.1162/tacl\_a\_00067](https://doi.org/10.1162/tacl_a_00067) · [OpenAlex](https://openalex.org/W2531207078)

**Citations:** 394 citations (openalex-snapshot · 2026-10-14) · [151](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Word- and subword-level NMT systems rely on a separate segmentation step and handle rare or unseen word forms poorly. This is a serious weakness for morphologically rich languages such as Czech or Finnish. Lee, Cho and Hofmann propose char2char, an attentional NMT model that reads source characters and produces target characters with no tokenization at all. Character sequences are much longer than word sequences, so the encoder first shortens the input using convolution, max-pooling and highway layers and then passes it to a bidirectional GRU. This keeps training cost close to that of subword systems. The decoder also works purely on characters. On WMT'15 data, the bilingual model beats a bpe2char baseline (subword encoder, character decoder) on DE-EN and CS-EN and matches it on FI-EN and RU-EN. The authors also trained one shared character encoder on four source languages translating into English. It clearly outperforms its subword counterpart on every pair. For CS, FI and RU it even surpasses dedicated bilingual models in both BLEU and human evaluation. The work shows that explicit word boundaries are not needed and that character-level encoders make compact multilingual translation practical.

## Key points
- **Languages and data:** Source languages are German, Czech, Finnish and Russian, translated into English using WMT'15 data. In characters, sentences are about 6 times longer (DE, CS, RU) to 8 times longer (FI) than in words.
- **Encoder design:** The encoder adapts the convolution/pooling/highway stack from the character language model of Kim et al. (2015). By comparison, the earlier purely character-level baseline of Luong and Manning (2016) needed roughly 3 months to train.
- **Multilingual model:**
  - It is trained without any language identifier, yet copes with intra-sentence code-switching.
  - It overfits less on the low-resource FI-EN pair.
  - It covers four languages without increasing model size.
- **Speed (single Titan X GPU):**
  - With the same batch size, char2char trains about 35% slower than bpe2char.
  - A bilingual model takes about two weeks to train.
  - Decoding 3,000 sentences takes 71.71 minutes with char2char versus 66.63 minutes with bpe2char.
- **Code release:** Source code and pre-trained models are available at github.com/nyu-dl/dl4mt-c2c.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

151 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 139 not yet.

- [Morphological Word Embeddings for Arabic Neural Machine Translation in Low-Resource Settings](https://md.research.elamur.ai/free/en/papers/aclbulk-2018-shapiro-morphological-word-embeddings-arabic-neural/)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/)
- [Unsupervised Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/artetxe-2017-unsupervised-neural-machine-translation/)
- [Findings of the 2017 Conference on Machine Translation (WMT17)](https://md.research.elamur.ai/free/en/papers/bojar-2017-findings-2017-conference-machine-translation/)
- [Stronger Baselines for Trustable Results in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/denkowski-2017-stronger-baselines-trustable-results-neural/)
- [Learning to Parse and Translate Improves Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/eriguchi-2017-learning-parse-translate-improves-neural/)
- [Meta-Learning for Low-Resource Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/gu-2018-meta-learning-low-resource-neural/)
- [Universal Neural Machine Translation for Extremely Low Resource Languages](https://md.research.elamur.ai/free/en/papers/gu-2018-universal-neural-machine-translation-extremely/)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterfusion-non-destructive-task-composition/)
- [Abstract Syntax as Interlingua: Scaling Up the Grammatical Framework from Controlled Languages to Robust Pipelines](https://md.research.elamur.ai/free/en/papers/ranta-2020-abstract-syntax-as-interlingua-scaling/)
- [How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs](https://md.research.elamur.ai/free/en/papers/sennrich-2017-how-grammatical-is-character-level/)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://md.research.elamur.ai/free/en/papers/zhang-2020-improving-massively-multilingual-neural-machine/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{lee2017fully,
  title = {{Fully Character-Level Neural Machine Translation without Explicit Segmentation}},
  author = {Lee, Jason and Cho, Kyunghyun and Hofmann, Thomas},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2017},
  doi = {10.1162/tacl\_a\_00067},
  url = {https://doi.org/10.1162/tacl_a_00067}
}
```

- [← previous Fronting and exhaustive exclusion in Biblical Hebrew](https://md.research.elamur.ai/free/en/papers/hbo-2018-merwe-fronting-exhaustive-exclusion-biblical-hebrew/)
- [next → Gapping Constructions in Universal Dependencies v2](https://md.research.elamur.ai/free/en/papers/ud-2017-schuster-gapping-constructions-universal-dependencies-v2/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/2/)

## References

23 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+In+Proceedings+of+the+International+Conference+on+Learning+Representati) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
2. Chris Callison-Burch. 2009. Fast, cheap, and creative: Evaluating translation quality using amazon’s mechanical turk. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 286–295. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2… [find](https://scholar.google.com/scholar?q=Chris+Callison%2DBurch%2E+2009%2E+Fast%2C+cheap%2C+and+creative%3A+Evaluating+translation+quality+using+amazon%E2%80%99s+mechanical+turk%2E+In+Proceedings+of+the+2009+Conference+on+Empirical+Methods+in+Natural+Language+Pro) [our card](https://md.research.elamur.ai/free/en/papers/burch-2009-fast-cheap-creative-evaluating-translation/)
3. Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. 2016. A character-level decoder without explicit segmentation for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1693–1703. Marta R. Costa-Jussá and Josè A. R. Fonollosa… [find](https://scholar.google.com/scholar?q=Junyoung+Chung%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2016%2E+A+character%2Dlevel+decoder+without+explicit+segmentation+for+neural+machine+translation%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Associa)
4. Ferdinand de Saussure. 1916. Course in General Linguistics. [find](https://scholar.google.com/scholar?q=Ferdinand+de+Saussure%2E+1916%2E+Course+in+General+Linguistics%2E)
5. Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. 2016a. Multi-way, multilingual neural machine translation with a shared attention mechanism. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics, pages 866–875. [find](https://scholar.google.com/scholar?q=Orhan+Firat%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2016a%2E+Multi%2Dway%2C+multilingual+neural+machine+translation+with+a+shared+attention+mechanism%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+) [our card](https://md.research.elamur.ai/free/en/papers/firat-2016-multi-way-multilingual-neural-machine/)
6. Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T. Yarman Vural, and Kyunghyun Cho. 2016b. Zero-resource translation with multi-lingual neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 268–277. [find](https://scholar.google.com/scholar?q=Orhan+Firat%2C+Baskaran+Sankaran%2C+Yaser+Al%2DOnaizan%2C+Fatos+T%2E+Yarman+Vural%2C+and+Kyunghyun+Cho%2E+2016b%2E+Zero%2Dresource+translation+with+multi%2Dlingual+neural+machine+translation%2E+In+Proceedings+of+the+2016+C)
7. Dan Gillick, Cliff Brunk, Oriol Vinyals, and Amarnag Subramanya. 2015. Multilingual language processing from bytes. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics, pages 1296–1306. [find](https://scholar.google.com/scholar?q=Dan+Gillick%2C+Cliff+Brunk%2C+Oriol+Vinyals%2C+and+Amarnag+Subramanya%2E+2015%2E+Multilingual+language+processing+from+bytes%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapter+of+the+Associati)
8. Yvette Graham, Nitika Mathur, and Timothy Baldwin. 2015. Accurate evaluation of segment-level machine translation metrics. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1183–1191. [find](https://scholar.google.com/scholar?q=Yvette+Graham%2C+Nitika+Mathur%2C+and+Timothy+Baldwin%2E+2015%2E+Accurate+evaluation+of+segment%2Dlevel+machine+translation+metrics%2E+In+Proceedings+of+the+2015+Conference+of+the+North+American+Chapter+of+the+As) [our card](https://md.research.elamur.ai/free/en/papers/graham-2015-accurate-evaluation-segment-level-machine/)
9. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2017. Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering, 23(1):3–30. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735– 1780. [find](https://scholar.google.com/scholar?q=Yvette+Graham%2C+Timothy+Baldwin%2C+Alistair+Moffat%2C+and+Justin+Zobel%2E+2017%2E+Can+machine+translation+systems+be+evaluated+by+the+crowd+alone%2E+Natural+Language+Engineering%2C+23%281%29%3A3%E2%80%9330%2E+Sepp+Hochreiter+and+)
10. Ray S. Jackendoff. 1992. Semantic Structures, Volume 18. MIT press. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=Ray+S%2E+Jackendoff%2E+1992%2E+Semantic+Structures%2C+Volume+18%2E+MIT+press%2E+Se%CC%81bastien+Jean%2C+Kyunghyun+Cho%2C+Roland+Memisevic%2C+and+Yoshua+Bengio%2E+2015%2E+On+using+very+large+target+vocabulary+for+neural+machine+)
11. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. 2015. Character-aware neural language models. In Proceedings of the 30th AAAI Conference on Artificial Intelligence, pages 2741–2749. [find](https://scholar.google.com/scholar?q=Yoon+Kim%2C+Yacine+Jernite%2C+David+Sontag%2C+and+Alexander+M%2E+Rush%2E+2015%2E+Character%2Daware+neural+language+models%2E+In+Proceedings+of+the+30th+AAAI+Conference+on+Artificial+Intelligence%2C+pages+2741%E2%80%932749%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2015-character-aware-neural-language-models/)
12. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference for Learning Representations. [find](https://scholar.google.com/scholar?q=Diederik+Kingma+and+Jimmy+Ba%2E+2014%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+Proceedings+of+the+3rd+International+Conference+for+Learning+Representations%2E)
13. Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W. Black. 2015. Character-based neural machine translation. arXiv preprint arXiv:1511.04586. [find](https://scholar.google.com/scholar?q=Wang+Ling%2C+Isabel+Trancoso%2C+Chris+Dyer%2C+and+Alan+W%2E+Black%2E+2015%2E+Character%2Dbased+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1511%2E04586%2E) [our card](https://md.research.elamur.ai/free/en/papers/jussa-2016-character-based-neural-machine-translation/)
14. Minh-Thang Luong and Christopher D. Manning. 2016. Achieving open vocabulary neural machine translation with hybrid word-character models. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1054–1063. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong+and+Christopher+D%2E+Manning%2E+2016%2E+Achieving+open+vocabulary+neural+machine+translation+with+hybrid+word%2Dcharacter+models%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+) [our card](https://md.research.elamur.ai/free/en/papers/luong-2016-achieving-open-vocabulary-neural-machine/)
15. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1412–1421. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Hieu+Pham%2C+and+Christopher+D%2E+Manning%2E+2015%2E+Effective+approaches+to+attention%2Dbased+neural+machine+translation%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Comp)
16. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning, pages 1310–1318. [find](https://scholar.google.com/scholar?q=Razvan+Pascanu%2C+Tomas+Mikolov%2C+and+Yoshua+Bengio%2E+2013%2E+On+the+difficulty+of+training+recurrent+neural+networks%2E+In+Proceedings+of+the+30th+International+Conference+on+Machine+Learning%2C+pages+1310%E2%80%93131)
17. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1715–1725. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2015%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Ling)
18. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Edinburgh neural machine translation systems for WMT 16. In Proceedings of the 1st Conference on Machine Translation. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Training very deep networks. In Advances in Neural Inform… [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016%2E+Edinburgh+neural+machine+translation+systems+for+WMT+16%2E+In+Proceedings+of+the+1st+Conference+on+Machine+Translation%2E+Rupesh+Kumar+Srivastava%2C+K) [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2016-edinburgh-neural-machine-translation-systems/)
19. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, Volume 27, pages 3104–3112. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V%2E+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+Volume+27%2C+pages+3104%E2%80%933112%2E)
20. Yulia Tsvetkov, Sunayana Sitaram, Manaal Faruqui, Guillaume Lample, Patrick Littell, David Mortensen, Alan W. Black, Lori Levin, and Chris Dyer. 2016. Polyglot neural language models: A case study in cross-lingual phonetic representation learning. In Proceedings of the 2016 Conference of the North A… [find](https://scholar.google.com/scholar?q=Yulia+Tsvetkov%2C+Sunayana+Sitaram%2C+Manaal+Faruqui%2C+Guillaume+Lample%2C+Patrick+Littell%2C+David+Mortensen%2C+Alan+W%2E+Black%2C+Lori+Levin%2C+and+Chris+Dyer%2E+2016%2E+Polyglot+neural+language+models%3A+A+case+study+in+)
21. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V%2E+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+Jeff+Klingner%2C+Apurva+Shah%2C+Melvin+Johnson%2C+Xiaobing+Liu%2C+Luk)
22. Yijun Xiao and Kyunghyun Cho. 2016. Efficient character-level document classification by combining convolution and recurrent layers. arXiv preprint arXiv:1602.00367. — [arXiv:1602.00367](https://arxiv.org/abs/1602.00367) [find](https://scholar.google.com/scholar?q=Yijun+Xiao+and+Kyunghyun+Cho%2E+2016%2E+Efficient+character%2Dlevel+document+classification+by+combining+convolution+and+recurrent+layers%2E+arXiv+preprint+arXiv%3A1602%2E00367%2E+%E2%80%94+%5BarXiv%3A1602%2E00367%5D%28https%3A%2F%2Farxiv)
23. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, Volume 28, pages 649–657. [find](https://scholar.google.com/scholar?q=Xiang+Zhang%2C+Junbo+Zhao%2C+and+Yann+LeCun%2E+2015%2E+Character%2Dlevel+convolutional+networks+for+text+classification%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+Volume+28%2C+pages+649%E2%80%93657%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/)