---
title: "Revisiting Character-Based Neural Machine Translation with"
description: "The authors therefore treat character-specific designs as ways to save compute rather… 31.6 BLEU. 23 references, cited by 62 papers here."
authors: ["Colin Cherry", "George Foster", "Ankur Bapna", "Orhan Firat", "Wolfgang Macherey"]
year: 2018
venue: "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D18-1461/"
doi: "10.18653/v1/d18-1461"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/cherry-2018-revisiting-character-based-neural-machine/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cherry-2018-revisiting-character-based-neural-machine/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/) / Revisiting Character-Based Neural Machine Translation with Capacity and Compression

# Revisiting Character-Based Neural Machine Translation with Capacity and Compression

**Authors:** [Colin Cherry](https://md.research.elamur.ai/free/en/authors/colin-cherry/), [George Foster](https://md.research.elamur.ai/free/en/authors/george-foster/), [Ankur Bapna](https://md.research.elamur.ai/free/en/authors/ankur-bapna/), [Orhan Firat](https://md.research.elamur.ai/free/en/authors/orhan-firat/) and [Wolfgang Macherey](https://md.research.elamur.ai/free/en/authors/wolfgang-macherey/) · **Year:** 2018 · **Venue:** [Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2018](https://md.research.elamur.ai/free/en/venues/emnlp/2018/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D18-1461/) · [PDF](https://aclanthology.org/D18-1461.pdf) · [DOI 10.18653/v1/d18-1461](https://doi.org/10.18653/v1/d18-1461) · [OpenAlex](https://openalex.org/W2888958984)

**Citations:** 92 citations (OpenAlex, 2026-10-14) · [62](https://md.research.elamur.ai/free/en/papers/cherry-2018-revisiting-character-based-neural-machine/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Character-level neural machine translation removes the need for tokenization and BPE segmentation, along with the hyperparameters those steps require. The cost is much longer input and output sequences. Cherry et al. revisit whether special architectures are actually needed for characters. They train a standard deep BiLSTM encoder–decoder with attention, adapted from Chen et al. (2018), on both characters and BPE fragments, keeping everything else identical. With enough depth, the character-level model beats the BPE models for all but the largest training corpora, although it costs far more computation. The authors therefore treat character-specific designs as ways to save compute rather than as better models of long sequences. They compare several methods for compressing the source sequence in the encoder:
- a reimplementation of Lee et al. (2017)'s convolution-and-pooling encoder,
- a BiLSTM encoder with fixed-stride mean-pooling layers placed between recurrent layers,
- the Hierarchical Multiscale LSTM, which learns which timesteps to skip. This is its first use in NMT.

None of these methods matches the uncompressed character baseline. The paper argues that efficient character-level translation is a worthwhile open problem.

## Key points
- On WMT15 DeEn, the deep character model scores 31.6 BLEU. BPE models with vocabularies from 1k to 32k score between 29.6 and 30.5.
- The reimplementation of Lee et al. (2017) beats the originally reported scores by more than 2 BLEU, likely because of large-batch training. Making this encoder deeper brought no real gains.
- The pooled BiLSTM encoder places a stride-3 pooling layer after the second BiLSTM layer and a stride-2 layer after the third, shortening its output 6×. It trails BPE-1k by about 0.5 BLEU at similar compression.
- In small-scale tests, the HM encoder used no-flush and hierarchical modifications, a z-bias initialized to 1, and slope annealing. It reached LSTM parity (28.8 vs. 28.9 BLEU) while saving about 35% of layer-wise computation. Each HM layer kept roughly 60% of the timesteps of the layer below.
- Compared with fixed-stride pooling, HM gating compresses less but performs better; 3 HM layers beat 2. The speedups are conceptual so far, and realizing them in practice is a significant engineering challenge.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 353 works · 364,806 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 958 works · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 77 works

## Cite

Cite as: Colin Cherry, George Foster, Ankur Bapna, Orhan Firat and Wolfgang Macherey. 2018. Revisiting Character-Based Neural Machine Translation with Capacity and Compression. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{cherry2018revisiting,
  title = {{Revisiting Character-Based Neural Machine Translation with Capacity and Compression}},
  author = {Colin Cherry, George Foster, Ankur Bapna, Orhan Firat and Wolfgang Macherey},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1461},
  url = {https://doi.org/10.18653/v1/d18-1461}
}
```

- [← previous Breaking the Beam Search Curse: A Study of (Re-)Scoring Methods and Stopping Criteria for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/yang-2018-breaking-beam-search-curse-study/)
- [next → Seq2seq Dependency Parsing](https://md.research.elamur.ai/free/en/papers/li-2018-seq2seq-dependency-parsing/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/)

## References

23 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. [Layer normalization](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/). arXiv preprint arXiv:1607.06450.
2. Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2C+Nicholas+Le%CC%81onard%2C+and+Aaron+Courville%2E+2013%2E+Estimating+or+propagating+gradients+through+stochastic+neurons+for+conditional+computation%2E+arXiv+preprint+arXiv%3A1308%2E3432%2E)
3. James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. 2016. Quasi-recurrent neural networks. arXiv preprint arXiv:1611.01576. [find](https://scholar.google.com/scholar?q=James+Bradbury%2C+Stephen+Merity%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2016%2E+Quasi%2Drecurrent+neural+networks%2E+arXiv+preprint+arXiv%3A1611%2E01576%2E)
4. William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. 2016. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on, pages 4960–4964. IEEE. [find](https://scholar.google.com/scholar?q=William+Chan%2C+Navdeep+Jaitly%2C+Quoc+Le%2C+and+Oriol+Vinyals%2E+2016%2E+Listen%2C+attend+and+spell%3A+A+neural+network+for+large+vocabulary+conversational+speech+recognition%2E+In+Acoustics%2C+Speech+and+Signal+Proce)
5. Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Mike Schuster, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. 2018. [The best of both worlds: Combining recent advances in neural machine translation](https://md.research.elamur.ai/free/en/papers/chen-2018-best-both-worlds-combining-recent/). In Proceedings of the 56th Annual Me…
6. Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. 2017. Hierarchical multiscale recurrent neural networks. In International Conference on Learning Representations, Toulon, France. [find](https://scholar.google.com/scholar?q=Junyoung+Chung%2C+Sungjin+Ahn%2C+and+Yoshua+Bengio%2E+2017%2E+Hierarchical+multiscale+recurrent+neural+networks%2E+In+International+Conference+on+Learning+Representations%2C+Toulon%2C+France%2E)
7. Marta R. Costa-jussà and José A. R. Fonollosa. 2016. [Character-based neural machine translation](https://md.research.elamur.ai/free/en/papers/jussa-2016-character-based-neural-machine-translation/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 357–361, Berlin, Germany. Association for Computational Linguistics.
8. Caglar Gulcehre, Francis Dutil, Adam Trischler, and Yoshua Bengio. 2017. Plan, attend, generate: Character-level neural machine translation with planning. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 228–234, Vancouver, Canada. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Caglar+Gulcehre%2C+Francis+Dutil%2C+Adam+Trischler%2C+and+Yoshua+Bengio%2E+2017%2E+Plan%2C+attend%2C+generate%3A+Character%2Dlevel+neural+machine+translation+with+planning%2E+In+Proceedings+of+the+2nd+Workshop+on+Represe)
9. Melvin Johnson, Mike Schuster, Quoc Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernand a Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Asso… [find](https://scholar.google.com/scholar?q=Melvin+Johnson%2C+Mike+Schuster%2C+Quoc+Le%2C+Maxim+Krikun%2C+Yonghui+Wu%2C+Zhifeng+Chen%2C+Nikhil+Thorat%2C+Fernand+a+Vie%CC%81gas%2C+Martin+Wattenberg%2C+Greg+Corrado%2C+Macduff+Hughes%2C+and+Jeffrey+Dean%2E+2017%2E+Google%E2%80%99s+mult)
10. Łukasz Kaiser, Aurko Roy, Ashish Vaswani, Niki Parmar, Samy Bengio, Jakob Uszkoreit, and Noam Shazeer. 2018. Fast decoding in sequence models using discrete latent variables. arXiv preprint arXiv:1803.03382. — [arXiv:1803.03382](https://arxiv.org/abs/1803.03382) [find](https://scholar.google.com/scholar?q=%C5%81ukasz+Kaiser%2C+Aurko+Roy%2C+Ashish+Vaswani%2C+Niki+Parmar%2C+Samy+Bengio%2C+Jakob+Uszkoreit%2C+and+Noam+Shazeer%2E+2018%2E+Fast+decoding+in+sequence+models+using+discrete+latent+variables%2E+arXiv+preprint+arXiv%3A1803)
11. Nan Rosemary Ke, Konrad Zolna, Alessandro Sordoni, Zhouhan Lin, Adam Trischler, Yoshua Bengio, Joelle Pineau, Laurent Charlin, and Chris Pal. 2018. Focused hierarchical RNNs for conditional sequence processing. arXiv preprint arXiv:1806.04342. — [arXiv:1806.04342](https://arxiv.org/abs/1806.04342) [find](https://scholar.google.com/scholar?q=Nan+Rosemary+Ke%2C+Konrad+Zolna%2C+Alessandro+Sordoni%2C+Zhouhan+Lin%2C+Adam+Trischler%2C+Yoshua+Bengio%2C+Joelle+Pineau%2C+Laurent+Charlin%2C+and+Chris+Pal%2E+2018%2E+Focused+hierarchical+RNNs+for+conditional+sequence+p)
12. Diederik P Kingma and Jimmy Lei Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Lei+Ba%2E+2014%2E+Adam%3A+A+method+for+stochastic+optimization%2E+arXiv+preprint+arXiv%3A1412%2E6980%2E)
13. Taku Kudo. 2018. [Subword regularization: Improving neural network translation models with multiple subword candidates](https://md.research.elamur.ai/free/en/papers/kudo-2018-subword-regularization-improving-neural-network/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (To Appear).
14. Ákos Kádár, Marc-Alexandre Côté, Grzegorz Chrupała, and Afra Alishahi. 2018. Revisiting the hierarchical multiscale LSTM. In COLING. [find](https://scholar.google.com/scholar?q=A%CC%81kos+Ka%CC%81da%CC%81r%2C+Marc%2DAlexandre+Co%CC%82te%CC%81%2C+Grzegorz+Chrupa%C5%82a%2C+and+Afra+Alishahi%2E+2018%2E+Revisiting+the+hierarchical+multiscale+LSTM%2E+In+COLING%2E)
15. Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2017. [Fully character-level neural machine translation without explicit segmentation](https://md.research.elamur.ai/free/en/papers/lee-2017-fully-character-level-neural-machine/). Transactions of the Association for Computational Linguistics, 5:365–378.
16. Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W Black. 2015. [Character-based neural machine translation](https://md.research.elamur.ai/free/en/papers/jussa-2016-character-based-neural-machine-translation/). arXiv preprint arXiv:1511.04586.
17. Minh-Thang Luong and Christopher D. Manning. 2016. [Achieving open vocabulary neural machine translation with hybrid word-character models](https://md.research.elamur.ai/free/en/papers/luong-2016-achieving-open-vocabulary-neural-machine/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1054–1063, Berlin, Germany. Association…
18. Matt Post. 2018. [A call for clarity in reporting BLEU scores](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/). arXiv preprint arXiv:1804.08771v1.
19. Colin Raffel, Minh-Thang Luong, Peter J. Liu, Ron J. Weiss, and Douglas Eck. 2017. Online and linear-time attention by enforcing monotonic alignments. In International Conference on Machine Learning, Sydney, Australia. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Minh%2DThang+Luong%2C+Peter+J%2E+Liu%2C+Ron+J%2E+Weiss%2C+and+Douglas+Eck%2E+2017%2E+Online+and+linear%2Dtime+attention+by+enforcing+monotonic+alignments%2E+In+International+Conference+on+Machine+Learning%2C+)
20. Rico Sennrich. 2017. [How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs](https://md.research.elamur.ai/free/en/papers/sennrich-2017-how-grammatical-is-character-level/). In EACL, pages 376–382, Valencia, Spain.
21. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Li…
22. Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. [Modeling coverage for neural machine translation](https://md.research.elamur.ai/free/en/papers/tu-2016-modeling-coverage-neural-machine-translation/). arXiv preprint arXiv:1601.04811.
23. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kaza-waand Keith Stevens, George Kurian,… [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+Jeff+Klingner%2C+Apurva+Shah%2C+Melvin+Johnson%2C+Xiaobing+Liu%2C+%C5%81uka)

*Updated 24 October 2026*