---
title: "Subword Regularization — Kudo 2018"
description: "The author proposes to exploit this ambiguity as useful noise. 23.12 BLEU. 33 references, cited by 561 papers here."
authors: ["Taku Kudo"]
year: 2018
venue: "Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/P18-1007/"
doi: "10.18653/v1/p18-1007"
arxiv: "1804.10959"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kudo-2018-subword-regularization-improving-neural-network/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kudo-2018-subword-regularization-improving-neural-network/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/) / Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

# Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

**Authors:** [Taku Kudo](https://md.research.elamur.ai/free/en/authors/taku-kudo/) · **Year:** 2018 · **Venue:** Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P18-1007/) · [PDF](https://aclanthology.org/P18-1007.pdf) · [DOI 10.18653/v1/p18-1007](https://doi.org/10.18653/v1/p18-1007) · [arXiv 1804.10959](https://arxiv.org/abs/1804.10959)

**Citations:** [561](https://md.research.elamur.ai/free/en/papers/kudo-2018-subword-regularization-improving-neural-network/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Neural machine translation (NMT) operates with a limited vocabulary, so rare words are usually split into subwords. The standard approach is BPE. It produces a single fixed segmentation for each sentence. However, with the same vocabulary a sentence can be segmented in different ways, and the model perceives these variants as entirely different inputs. The author proposes to exploit this ambiguity as useful noise. During training, the segmentation of each sentence is sampled anew each time from a probability distribution. This effectively augments the training data while leaving the network architecture unchanged. To obtain segmentation probabilities, the author introduces a new method for splitting text that relies on a unigram language model. It gradually prunes a large initial vocabulary using EM and can output multiple candidates together with their probabilities. Experiments on IWSLT15, IWSLT17, WMT14 and other corpora showed consistent BLEU improvements. Low-resource language pairs and out-of-domain texts benefit the most. The method is implemented in the open-source library SentencePiece, so it is easy to apply in other systems.

## Key points
- The subword regularization method has two hyperparameters: l, the number of candidates for sampling (e.g., 64 or ∞, i.e., the entire lattice), and α, a smoothing parameter. If α = 0, sampling becomes uniform and quality drops noticeably. Hence, the distribution provided by the language model really matters.
- WMT14 en→de: word-level model 23.12 BLEU, BPE 24.53, unigram without regularization 24.50, unigram with regularization (l = 64, α = 0.1) 25.04. This is the best result among the compared segmentations.
- The gain is largest on out-of-domain data. For WMT14 en→de patent texts, BLEU rose from 15.63 to 25.76; for IWSLT15 en→vi web texts, from 13.86 to 17.36.
- Regularization applied only on the source side or only on the target side also helps, but it works best on both sides. For IWSLT15 en→vi, BLEU rose from 25.49 to 27.68. The author believes the method can be transferred to tasks that have only an encoder or only a decoder.
- The implementation is publicly available: https://github.com/google/sentencepiece.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/)

## Cite

Cite as: Taku Kudo. 2018. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kudo2018subword,
  title = {{Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates}},
  author = {Taku Kudo},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1007},
  eprint = {1804.10959},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p18-1007}
}
```

- [← previous A Survey on Recent Advances in Named Entity Recognition from Deep Learning models](https://md.research.elamur.ai/free/en/papers/yadav-2018-survey-recent-advances-named-entity/)
- [next → Style Transfer Through Back-Translation](https://md.research.elamur.ai/free/en/papers/prabhumoye-2018-style-transfer-through-back-translation/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/)

## References

33 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Toshiaki Nakazawa, Shohei Higashiyama, Chenchen Ding, Hideya Mino, Isao Goto, Hideto Kazawa, Yusuke Oda, Graham Neubig, and Sadao Kurohashi. 2017. Overview of the 4th workshop on asian translation. In Proceedings of the 4th Workshop on Asian Translation (WAT2017). pages 1–54. [find](https://scholar.google.com/scholar?q=Toshiaki+Nakazawa%2C+Shohei+Higashiyama%2C+Chenchen+Ding%2C+Hideya+Mino%2C+Isao+Goto%2C+Hideto+Kazawa%2C+Yusuke+Oda%2C+Graham+Neubig%2C+and+Sadao+Kurohashi%2E+2017%2E+Overview+of+the+4th+workshop+on+asian+translation%2E+In)
2. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2017. Unsupervised neural machine translation. arXive preprint arXiv:1710.11041 . [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2017-unsupervised-neural-machine-translation/)
3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 . [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
4. Ge Nong, Sen Zhang, and Wai Hong Chan. 2009. Linear suffix array construction by almost pure induced-sorting. In Proc. of DCC. [find](https://scholar.google.com/scholar?q=Ge+Nong%2C+Sen+Zhang%2C+and+Wai+Hong+Chan%2E+2009%2E+Linear+suffix+array+construction+by+almost+pure+induced%2Dsorting%2E+In+Proc%2E+of+DCC%2E)
5. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proc. of ACL. [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
6. Yonatan Belinkov and Yonatan Bisk. 2017. Synthetic and natural noise both break neural machine translation. arXive preprint arXiv:1711.02173 . [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov+and+Yonatan+Bisk%2E+2017%2E+Synthetic+and+natural+noise+both+break+neural+machine+translation%2E+arXive+preprint+arXiv%3A1711%2E02173+%2E)
7. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=Alexander+M+Rush%2C+Sumit+Chopra%2C+and+Jason+Weston%2E+2015%2E+A+neural+attention+model+for+abstractive+sentence+summarization%2E+In+Proc%2E+of+EMNLP%2E)
8. William Chan, Yu Zhang, Quoc Le, and Navdeep Jaitly. 2016. Latent sequence decompositions. arXiv preprint arXiv:1610.03035 . — [arXiv:1610.03035](https://arxiv.org/abs/1610.03035) [find](https://scholar.google.com/scholar?q=William+Chan%2C+Yu+Zhang%2C+Quoc+Le%2C+and+Navdeep+Jaitly%2E+2016%2E+Latent+sequence+decompositions%2E+arXiv+preprint+arXiv%3A1610%2E03035+%2E+%E2%80%94+%5BarXiv%3A1610%2E03035%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1610%2E03035%29)
9. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In Proc. of ICASSP. [find](https://scholar.google.com/scholar?q=Mike+Schuster+and+Kaisuke+Nakajima%2E+2012%2E+Japanese+and+korean+voice+search%2E+In+Proc%2E+of+ICASSP%2E)
10. Rohan Chitnis and John DeNero. 2015. Variable-length word encodings for neural translation models. In Proc. of EMNLP. pages 2088–2093. [find](https://scholar.google.com/scholar?q=Rohan+Chitnis+and+John+DeNero%2E+2015%2E+Variable%2Dlength+word+encodings+for+neural+translation+models%2E+In+Proc%2E+of+EMNLP%2E+pages+2088%E2%80%932093%2E)
11. Steven L Scott. 2002. Bayesian methods for hidden markov models: Recursive computing in the 21st century. Journal of the American Statistical Association . [find](https://scholar.google.com/scholar?q=Steven+L+Scott%2E+2002%2E+Bayesian+methods+for+hidden+markov+models%3A+Recursive+computing+in+the+21st+century%2E+Journal+of+the+American+Statistical+Association+%2E)
12. Michael Denkowski and Graham Neubig. 2017. Stronger baselines for trustable results in neural machine translation. Proc. of Workshop on Neural Machine Translation . [our card](https://md.research.elamur.ai/free/en/papers/denkowski-2017-stronger-baselines-trustable-results-neural/)
13. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proc. of ACL. [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/)
14. Philip Gage. 1994. A new algorithm for data compression. C Users J. 12(2):23–38. [find](https://scholar.google.com/scholar?q=Philip+Gage%2E+1994%2E+A+new+algorithm+for+data+compression%2E+C+Users+J%2E+12%282%29%3A23%E2%80%9338%2E)
15. Matthias Sperber, Graham Neubig, Jan Niehues, and Alex Waibel. 2017. Neural lattice-to-sequence models for uncertain inputs. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=Matthias+Sperber%2C+Graham+Neubig%2C+Jan+Niehues%2C+and+Alex+Waibel%2E+2017%2E+Neural+lattice%2Dto%2Dsequence+models+for+uncertain+inputs%2E+In+Proc%2E+of+EMNLP%2E)
16. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. arXiv preprint arXiv:1705.03122 . [our card](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/)
17. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. JMLR 15(1). [find](https://scholar.google.com/scholar?q=Nitish+Srivastava%2C+Geoffrey+E+Hinton%2C+Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Ruslan+Salakhutdinov%2E+2014%2E+Dropout%3A+a+simple+way+to+prevent+neural+networks+from+overfitting%2E+JMLR+15%281%29%2E)
18. Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In Proc. of ICLR. [our card](https://md.research.elamur.ai/free/en/papers/goodfellow-2014-explaining-harnessing-adversarial-examples/)
19. Jinsong Su, Zhixing Tan, De yi Xiong, Rongrong Ji, Xiaodong Shi, and Yang Liu. 2017. Lattice-based recurrent neural network encoders for neural machine translation. In AAAI. pages 3302–3308. [find](https://scholar.google.com/scholar?q=Jinsong+Su%2C+Zhixing+Tan%2C+De+yi+Xiong%2C+Rongrong+Ji%2C+Xiaodong+Shi%2C+and+Yang+Liu%2E+2017%2E+Lattice%2Dbased+recurrent+neural+network+encoders+for+neural+machine+translation%2E+In+AAAI%2E+pages+3302%E2%80%933308%2E)
20. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proc. of ACL. [our card](https://md.research.elamur.ai/free/en/papers/iyyer-2015-deep-unordered-composition-rivals-syntactic/)
21. Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. Proc. of ACL . [find](https://scholar.google.com/scholar?q=Kai+Sheng+Tai%2C+Richard+Socher%2C+and+Christopher+D+Manning%2E+2015%2E+Improved+semantic+representations+from+tree%2Dstructured+long+short%2Dterm+memory+networks%2E+Proc%2E+of+ACL+%2E)
22. Diederik P Kingma and Jimmy Ba Adam. 2014. A method for stochastic optimization. arXiv preprint arXiv:1412.6980 . [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba+Adam%2E+2014%2E+A+method+for+stochastic+optimization%2E+arXiv+preprint+arXiv%3A1412%2E6980+%2E)
23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. arXive preprint arXiv:1706.03762 . [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
24. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proc. of EMNLP. [our card](https://md.research.elamur.ai/free/en/papers/koehn-2004-statistical-significance-tests-machine-translation/)
25. Guillaume Lample, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2017. Unsupervised machine translation using monolingual corpora only. arXive preprint arXiv:1711.00043 . [our card](https://md.research.elamur.ai/free/en/papers/lample-2017-unsupervised-machine-translation-using-monolingual/)
26. Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and composing robust features with denoising autoencoders. In Proc. of ICML. [find](https://scholar.google.com/scholar?q=Pascal+Vincent%2C+Hugo+Larochelle%2C+Yoshua+Bengio%2C+and+Pierre%2DAntoine+Manzagol%2E+2008%2E+Extracting+and+composing+robust+features+with+denoising+autoencoders%2E+In+Proc%2E+of+ICML%2E)
27. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In Proc of EMNLP. [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-effective-approaches-attention-based-neural/)
28. Oriol Vinyals and Quoc V. Le. 2015. A neural conversational model. In ICML Deep Learning Workshop. [our card](https://md.research.elamur.ai/free/en/papers/vinyals-2015-neural-conversational-model/)
29. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Computer Vision and Pattern Recognition. [our card](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/)
30. Masaaki Nagata. 1994. A stochastic japanese morphological analyzer using a forward-dp backward-a* nbest search algorithm. In Proc. of COLING. [find](https://scholar.google.com/scholar?q=Masaaki+Nagata%2E+1994%2E+A+stochastic+japanese+morphological+analyzer+using+a+forward%2Ddp+backward%2Da%2A+nbest+search+algorithm%2E+In+Proc%2E+of+COLING%2E)
31. Andrew Viterbi. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE transactions on Information Theory 13(2):260–269. [find](https://scholar.google.com/scholar?q=Andrew+Viterbi%2E+1967%2E+Error+bounds+for+convolutional+codes+and+an+asymptotically+optimum+decoding+algorithm%2E+IEEE+transactions+on+Information+Theory+13%282%29%3A260%E2%80%93269%2E)
32. Yonghui Wu, Mike Schuster, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 . [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+et+al%2E+2016%2E+Google%E2%80%99s+neural+machine+translation+system%3A+Bridging+the+gap+between+human+and+machine+translation%2E+arXiv+preprint+arXiv%3A1609%2E08144+%2E)
33. Ziang Xie, Sida I. Wang, Jiwei Li, Daniel Lévy, Aiming Nie, Dan Jurafsky, and Andrew Y. Ng. 2017. Data noising as smoothing in neural network language models. In Proc. of ICLR. [find](https://scholar.google.com/scholar?q=Ziang+Xie%2C+Sida+I%2E+Wang%2C+Jiwei+Li%2C+Daniel+Le%CC%81vy%2C+Aiming+Nie%2C+Dan+Jurafsky%2C+and+Andrew+Y%2E+Ng%2E+2017%2E+Data+noising+as+smoothing+in+neural+network+language+models%2E+In+Proc%2E+of+ICLR%2E)