---
title: "Non-Autoregressive Machine Translation with Latent Alignments"
description: "The paper addresses two common weaknesses of non-autoregressive machine… 25.7 BLEU. 36 references, cited by 70 papers here."
authors: ["Chitwan Saharia", "William Chan", "Saurabh Saxena", "Mohammad Norouzi"]
year: 2020
venue: "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)"
source: "https://aclanthology.org/2020.emnlp-main.83/"
doi: "10.18653/v1/2020.emnlp-main.83"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/saharia-2020-non-autoregressive-machine-translation-latent/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/saharia-2020-non-autoregressive-machine-translation-latent/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) / Non-Autoregressive Machine Translation with Latent Alignments

# Non-Autoregressive Machine Translation with Latent Alignments

**Authors:** [Chitwan Saharia](https://md.research.elamur.ai/free/en/authors/chitwan-saharia/), [William Chan](https://md.research.elamur.ai/free/en/authors/william-chan/), [Saurabh Saxena](https://md.research.elamur.ai/free/en/authors/saurabh-saxena/) and [Mohammad Norouzi](https://md.research.elamur.ai/free/en/authors/mohammad-norouzi/) · **Year:** 2020 · **Venue:** [Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2020](https://md.research.elamur.ai/free/en/venues/emnlp/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.emnlp-main.83/) · [PDF](https://aclanthology.org/2020.emnlp-main.83.pdf) · [DOI 10.18653/v1/2020.emnlp-main.83](https://doi.org/10.18653/v1/2020.emnlp-main.83) · [OpenAlex](https://openalex.org/W3016440711)

**Citations:** 106 citations (OpenAlex, 2026-10-14) · [70](https://md.research.elamur.ai/free/en/papers/saharia-2020-non-autoregressive-machine-translation-latent/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper addresses two common weaknesses of non-autoregressive machine translation: duplicated tokens caused by conditionally independent predictions, and the need to predict the output length in advance and often re-score several length candidates. The authors adopt latent alignment models from speech recognition, namely CTC and Imputer. These models place blank tokens between target tokens to form a monotonic alignment and use dynamic programming to sum over all possible alignments during training. The architecture is a self-attention stack over an upsampled source with no cross-attention. Imputer adds conditioning on a partially masked alignment from the previous step, so it can refine its output over a fixed number of steps.

The models are trained on distilled data and evaluated on WMT'14 En↔De and WMT'16 En↔Ro. Single-step CTC reached state-of-the-art for one-step decoding. Imputer with 4 steps matched or slightly exceeded an autoregressive Transformer. Neither model needs length prediction or autoregressive re-scoring, which shows that simple alignment-based models can make parallel translation competitive.

## Key points
- On WMT'14 En→De, single-step CTC scores 25.7 BLEU according to the abstract (the contributions list gives 25.8). Imputer reaches 27.5, 28.0 and 28.2 BLEU with 2, 4 and 8 steps, compared with 27.8 for the autoregressive Transformer baseline.
- Repeated tokens on WMT'14 En→De drop to 0.17% for CTC, compared with 16.72% for Mask-Predict and 1.41% for AXE CMLM.
- Speed-ups over greedy autoregressive decoding are about ×18.6 for single-step decoding and ×9.2, ×5.9 and ×3.9 for 2, 4 and 8 Imputer steps.
- Distillation is critical: CTC trained on original data scores 15.6 BLEU, while CTC trained on distilled data scores 25.4. The benefit shrinks as more decoding iterations are used.
- With fully sequential decoding, Imputer reaches 28.3 BLEU, which is 0.5 above its base Transformer teacher.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 353 works · 364,806 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 958 works

## Cite

Cite as: Chitwan Saharia, William Chan, Saurabh Saxena and Mohammad Norouzi. 2020. Non-Autoregressive Machine Translation with Latent Alignments. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{saharia2020non,
  title = {{Non-Autoregressive Machine Translation with Latent Alignments}},
  author = {Chitwan Saharia, William Chan, Saurabh Saxena and Mohammad Norouzi},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.83},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.83}
}
```

- [← previous Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information](https://md.research.elamur.ai/free/en/papers/lin-2020-pre-training-multilingual-neural-machine/)
- [next → Translation Artifacts in Cross-lingual Transfer Learning](https://md.research.elamur.ai/free/en/papers/artetxe-2020-translation-artifacts-cross-lingual-transfer/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/)

## References

36 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Harris Chan, Jamie Kiros, and William Chan. 2019a. Multilingual KERMIT: Its Not Easy Being Generative. In NeurIPS: Workshop on Perception as Generative Reasoning. [find](https://scholar.google.com/scholar?q=Harris+Chan%2C+Jamie+Kiros%2C+and+William+Chan%2E+2019a%2E+Multilingual+KERMIT%3A+Its+Not+Easy+Being+Generative%2E+In+NeurIPS%3A+Workshop+on+Perception+as+Generative+Reasoning%2E)
2. William Chan, Nikita Kitaev, Kelvin Guu, Mitchell Stern, and Jakob Uszkoreit. 2019b. KERMIT: Generative Insertion-Based Modeling for Sequences. In arXiv. [find](https://scholar.google.com/scholar?q=William+Chan%2C+Nikita+Kitaev%2C+Kelvin+Guu%2C+Mitchell+Stern%2C+and+Jakob+Uszkoreit%2E+2019b%2E+KERMIT%3A+Generative+Insertion%2DBased+Modeling+for+Sequences%2E+In+arXiv%2E)
3. Jungo Kasai, James Cross, Marjan Ghazvininejad, and Jiatao Gu. 2020. Parallel Machine Translation with Disentangled Context Transformer. arXiv preprint arXiv:2001.05136. — [arXiv:2001.05136](https://arxiv.org/abs/2001.05136) [find](https://scholar.google.com/scholar?q=Jungo+Kasai%2C+James+Cross%2C+Marjan+Ghazvininejad%2C+and+Jiatao+Gu%2E+2020%2E+Parallel+Machine+Translation+with+Disentangled+Context+Transformer%2E+arXiv+preprint+arXiv%3A2001%2E05136%2E+%E2%80%94+%5BarXiv%3A2001%2E05136%5D%28https%3A%2F%2Fa)
4. Diederik Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR. [find](https://scholar.google.com/scholar?q=Diederik+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+Method+for+Stochastic+Optimization%2E+In+ICLR%2E)
5. William Chan, Chitwan Saharia, Geoffrey Hinton, Mohammad Norouzi, and Navdeep Jaitly. 2020. Imputer: Sequence modelling via imputation and dynamic programming. In arXiv. [find](https://scholar.google.com/scholar?q=William+Chan%2C+Chitwan+Saharia%2C+Geoffrey+Hinton%2C+Mohammad+Norouzi%2C+and+Navdeep+Jaitly%2E+2020%2E+Imputer%3A+Sequence+modelling+via+imputation+and+dynamic+programming%2E+In+arXiv%2E)
6. Taku Kudo and John Richardson. 2018. Sentence-Piece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In EMNLP. [find](https://scholar.google.com/scholar?q=Taku+Kudo+and+John+Richardson%2E+2018%2E+Sentence%2DPiece%3A+A+simple+and+language+independent+subword+tokenizer+and+detokenizer+for+Neural+Text+Processing%2E+In+EMNLP%2E)
7. William Chan, Mitchell Stern, Jamie Kiros, and Jakob Uszkoreit. 2019c. An Empirical Study of Generation Order for Machine Translation. In arXiv. [find](https://scholar.google.com/scholar?q=William+Chan%2C+Mitchell+Stern%2C+Jamie+Kiros%2C+and+Jakob+Uszkoreit%2E+2019c%2E+An+Empirical+Study+of+Generation+Order+for+Machine+Translation%2E+In+arXiv%2E)
8. Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. [Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement](https://md.research.elamur.ai/free/en/papers/lee-2018-deterministic-non-autoregressive-neural-sequence/). In EMNLP.
9. Chris Dyer, Victor Chahuneau, and Noah Smith. 2013. A Simple, Fast, and Effective Reparameterization of IBM Model 2. In NAACL. [find](https://scholar.google.com/scholar?q=Chris+Dyer%2C+Victor+Chahuneau%2C+and+Noah+Smith%2E+2013%2E+A+Simple%2C+Fast%2C+and+Effective+Reparameterization+of+IBM+Model+2%2E+In+NAACL%2E)
10. Marjan Ghazvininejad, Vladimir Karpukhin, Luke Zettlemoyer, and Omer Levy. 2020a. Aligned Cross Entropy for Non-Autoregressive Machine Translation. In arXiv. [find](https://scholar.google.com/scholar?q=Marjan+Ghazvininejad%2C+Vladimir+Karpukhin%2C+Luke+Zettlemoyer%2C+and+Omer+Levy%2E+2020a%2E+Aligned+Cross+Entropy+for+Non%2DAutoregressive+Machine+Translation%2E+In+arXiv%2E)
11. Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. [Mask-Predict: Parallel Decoding of Conditional Masked Language Models](https://md.research.elamur.ai/free/en/papers/ghazvininejad-2019-mask-predict-parallel-decoding-conditional/). In EMNLP.
12. Marjan Ghazvininejad, Omer Levy, and Luke Zettlemoyer. 2020b. Semi-Autoregressive Training Improves Mask-Predict Decoding. In arXiv. [find](https://scholar.google.com/scholar?q=Marjan+Ghazvininejad%2C+Omer+Levy%2C+and+Luke+Zettlemoyer%2E+2020b%2E+Semi%2DAutoregressive+Training+Improves+Mask%2DPredict+Decoding%2E+In+arXiv%2E)
13. Alex Graves, Santiago Fernandez, Faustino Gomez, and Jurgen Schmidhuber. 2006. Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks. In ICML. [find](https://scholar.google.com/scholar?q=Alex+Graves%2C+Santiago+Fernandez%2C+Faustino+Gomez%2C+and+Jurgen+Schmidhuber%2E+2006%2E+Connectionist+Temporal+Classification%3A+Labelling+Unsegmented+Sequence+Data+with+Recurrent+Neural+Networks%2E+In+ICML%2E)
14. Alex Graves and Navdeep Jaitly. 2014. Towards End-to-End Speech Recognition with Recurrent Neural Networks. In ICML. [find](https://scholar.google.com/scholar?q=Alex+Graves+and+Navdeep+Jaitly%2E+2014%2E+Towards+End%2Dto%2DEnd+Speech+Recognition+with+Recurrent+Neural+Networks%2E+In+ICML%2E)
15. Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech Recognition with Deep Recurrent Neural Networks. In ICASSP. [find](https://scholar.google.com/scholar?q=Alex+Graves%2C+Abdel%2Drahman+Mohamed%2C+and+Geoffrey+Hinton%2E+2013%2E+Speech+Recognition+with+Deep+Recurrent+Neural+Networks%2E+In+ICASSP%2E)
16. Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. 2018. [Non-Autoregressive Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/gu-2017-non-autoregressive-neural-machine-translation/). In ICLR.
17. Jiatao Gu, Changhan Wang, and Jake Zhao. 2019. Levenshtein Transformer. In NeurIPS. [find](https://scholar.google.com/scholar?q=Jiatao+Gu%2C+Changhan+Wang%2C+and+Jake+Zhao%2E+2019%2E+Levenshtein+Transformer%2E+In+NeurIPS%2E)
18. Junliang Guo, Linli Xu, and Enhong Chen. 2020. Jointly masked sequence-to-sequence model for non-autoregressive neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 376– 385, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Junliang+Guo%2C+Linli+Xu%2C+and+Enhong+Chen%2E+2020%2E+Jointly+masked+sequence%2Dto%2Dsequence+model+for+non%2Dautoregressive+neural+machine+translation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association)
19. Lala Li and William Chan. 2019. Big Bidirectional Insertion Representations for Documents. In EMNLP: Workshop of Neural Generation and Translation. [find](https://scholar.google.com/scholar?q=Lala+Li+and+William+Chan%2E+2019%2E+Big+Bidirectional+Insertion+Representations+for+Documents%2E+In+EMNLP%3A+Workshop+of+Neural+Generation+and+Translation%2E)
20. Zhuohan Li, Zi Lin, Di He, Fei Tian, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2019. Hint-Based Training for Non-Autoregressive Machine Translation . In EMNLP. [find](https://scholar.google.com/scholar?q=Zhuohan+Li%2C+Zi+Lin%2C+Di+He%2C+Fei+Tian%2C+Tao+Qin%2C+Liwei+Wang%2C+and+Tie%2DYan+Liu%2E+2019%2E+Hint%2DBased+Training+for+Non%2DAutoregressive+Machine+Translation+%2E+In+EMNLP%2E)
21. Jindrich Libovicky and Jindrich Helcl. 2018. [End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification](https://md.research.elamur.ai/free/en/papers/libovicky-2018-end-end-non-autoregressive-neural/). In EMNLP.
22. Jinglin Liu, Yi Ren, Xu Tan, Chen Zhang, Tao Qin, Zhou Zhao, and Tie-Yan Liu. 2020. Task-level curriculum learning for non-autoregressive neural machine translation. arXiv preprint arXiv:2007.08772. — [arXiv:2007.08772](https://arxiv.org/abs/2007.08772) [find](https://scholar.google.com/scholar?q=Jinglin+Liu%2C+Yi+Ren%2C+Xu+Tan%2C+Chen+Zhang%2C+Tao+Qin%2C+Zhou+Zhao%2C+and+Tie%2DYan+Liu%2E+2020%2E+Task%2Dlevel+curriculum+learning+for+non%2Dautoregressive+neural+machine+translation%2E+arXiv+preprint+arXiv%3A2007%2E08772%2E+%E2%80%94)
23. Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. 2019. [FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow](https://md.research.elamur.ai/free/en/papers/ma-2019-flowseq-non-autoregressive-conditional-sequence/). In EMNLP.
24. Christopher D Manning, Christopher D Manning, and Hinrich Schütze. 1999. Foundations of statistical natural language processing. MIT press. [find](https://scholar.google.com/scholar?q=Christopher+D+Manning%2C+Christopher+D+Manning%2C+and+Hinrich+Schu%CC%88tze%2E+1999%2E+Foundations+of+statistical+natural+language+processing%2E+MIT+press%2E)
25. Graham Neubig, Zi-Yi Dou, Junjie Hu, Paul Michel, Danish Pruthi, Xinyi Wang, and John Wieting. 2019. [compare-mt: A Tool for Holistic Comparison of Language Generation Systems](https://md.research.elamur.ai/free/en/papers/neubig-2019-compare-mt-tool-holistic-comparison/). CoRR.
26. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [BLEU: a Method for Automatic Evaluation of Machine Translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In ACL.
27. Laura Ruis, Mitchell Stern, Julia Proskurnia, and William Chan. 2019. Insertion-Deletion Transformer. In EMNLP: Workshop of Neural Generation and Translation. [find](https://scholar.google.com/scholar?q=Laura+Ruis%2C+Mitchell+Stern%2C+Julia+Proskurnia%2C+and+William+Chan%2E+2019%2E+Insertion%2DDeletion+Transformer%2E+In+EMNLP%3A+Workshop+of+Neural+Generation+and+Translation%2E)
28. Chenze Shao, Jinchao Zhang, Yang Feng, Fandong Meng, and Jie Zhou. 2020. Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation. In AAAI. [find](https://scholar.google.com/scholar?q=Chenze+Shao%2C+Jinchao+Zhang%2C+Yang+Feng%2C+Fandong+Meng%2C+and+Jie+Zhou%2E+2020%2E+Minimizing+the+Bag%2Dof%2DNgrams+Difference+for+Non%2DAutoregressive+Neural+Machine+Translation%2E+In+AAAI%2E)
29. Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. 2019. Insertion Transformer: Flexible Sequence Generation via Insertion Operations. In ICML. [find](https://scholar.google.com/scholar?q=Mitchell+Stern%2C+William+Chan%2C+Jamie+Kiros%2C+and+Jakob+Uszkoreit%2E+2019%2E+Insertion+Transformer%3A+Flexible+Sequence+Generation+via+Insertion+Operations%2E+In+ICML%2E)
30. Zhiqing Sun, Zhuohan Li, Haoqing Wang, Di He, Zi Lin, and Zhihong Deng. 2019. Fast Structured Decoding for Sequence Models. In NeurIPS. [find](https://scholar.google.com/scholar?q=Zhiqing+Sun%2C+Zhuohan+Li%2C+Haoqing+Wang%2C+Di+He%2C+Zi+Lin%2C+and+Zhihong+Deng%2E+2019%2E+Fast+Structured+Decoding+for+Sequence+Models%2E+In+NeurIPS%2E)
31. Zhiqing Sun and Yiming Yang. 2020. An EM Approach to Non-autoregressive Conditional Sequence Generation. arXiv preprint arXiv:2006.16378. — [arXiv:2006.16378](https://arxiv.org/abs/2006.16378) [find](https://scholar.google.com/scholar?q=Zhiqing+Sun+and+Yiming+Yang%2E+2020%2E+An+EM+Approach+to+Non%2Dautoregressive+Conditional+Sequence+Generation%2E+arXiv+preprint+arXiv%3A2006%2E16378%2E+%E2%80%94+%5BarXiv%3A2006%2E16378%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2006%2E16378%29)
32. Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. [Tensor2Tensor for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/vaswani-2018-tensor2tensor-neural-machine-translation/). In AMTA.
33. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention Is All You Need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In NIPS.
34. Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. 2019. Non-Autoregressive Machine Translation with Auxiliary Regularization. In AAAI. [find](https://scholar.google.com/scholar?q=Yiren+Wang%2C+Fei+Tian%2C+Di+He%2C+Tao+Qin%2C+ChengXiang+Zhai%2C+and+Tie%2DYan+Liu%2E+2019%2E+Non%2DAutoregressive+Machine+Translation+with+Auxiliary+Regularization%2E+In+AAAI%2E)
35. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, ukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Ni… [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V%2E+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+Jeff+Klingner%2C+Apurva+Shah%2C+Melvin+Johnson%2C+Xiaobing+Liu%2C+uka)
36. Chunting Zhou, Graham Neubig, and Jiatao Gu. 2020. Understanding Knowledge Distillation in Non-autoregressive Machine Translation. In ICLR. [find](https://scholar.google.com/scholar?q=Chunting+Zhou%2C+Graham+Neubig%2C+and+Jiatao+Gu%2E+2020%2E+Understanding+Knowledge+Distillation+in+Non%2Dautoregressive+Machine+Translation%2E+In+ICLR%2E)

*Updated 24 October 2026*