---
title: "Is MAP Decoding All You Need? The Inadequacy — Eikema 2020"
description: "Neural machine translation (NMT) has been criticised for several failures."
authors: ["Bryan Eikema", "Wilker Aziz"]
year: 2020
venue: "Proceedings of the 28th International Conference on Computational Linguistics"
source: "https://aclanthology.org/2020.coling-main.398/"
doi: "10.18653/v1/2020.coling-main.398"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/eikema-2020-is-map-decoding-all-you/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/eikema-2020-is-map-decoding-all-you/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/) / Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation

# Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation

**Authors:** Bryan Eikema, [Wilker Aziz](https://md.research.elamur.ai/free/en/authors/wilker-aziz/) · **Year:** 2020 · **Venue:** Proceedings of the 28th International Conference on Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.coling-main.398/) · [PDF](https://aclanthology.org/2020.coling-main.398.pdf) · [DOI 10.18653/v1/2020.coling-main.398](https://doi.org/10.18653/v1/2020.coling-main.398) · [OpenAlex](https://openalex.org/W3027871355)

**Citations:** 60 citations (openalex-snapshot · 2026-10-14) · [105](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Neural machine translation (NMT) has been criticised for several failures. Beam search outputs tend to be too short, wider beams often lower quality, and the exact highest-probability output is frequently the empty string. Earlier work mostly blamed the locally normalised model or maximum likelihood estimation (MLE). Eikema and Aziz argue that the main culprit is the decision rule: maximum a posteriori (MAP) decoding, which looks for the mode. They analyse MLE-trained NMT models as probability distributions, using unbiased ancestral sampling across several language pairs, training-data sizes and test domains. The learnt distributions match data statistics such as length and lexical properties fairly well, while beam search pulls its outputs away from them. Beam outputs and the empty string are rare events under the model, and the mode carries so little probability mass that it is close to arbitrary. Samples are of reasonable quality. A sampling-based approximation of minimum Bayes risk (MBR) decoding performs competitively, especially when the model is uncertain. The work redirects model criticism toward the distribution as a whole and motivates decoding methods that use all of it.

## Key points
- The paper examines three known pathologies: length bias, the beam search curse, and what the authors call the inadequacy of the mode. The empty-mode finding comes from Stahlberg and Byrne (2019).
- The authors argue that approximate MAP decoding worsens exposure bias and adds non-admissible heuristic search bias. Ancestral sampling avoids the latter and, if the model fits well, introduces no distribution shift.
- MAP is presented as a special case of MBR with an exact-match utility. The MBR approximation uses model samples and a metric that is natively sentence-level, because smoothing sentence-level BLEU would bias its statistics.
- Earlier MBR attempts in NMT built the candidate set from beam search and needed very large beams. The authors see this as further evidence that high-scoring translations are unrepresentative.
- Even small sets of samples contain translations better than the beam search output, measured by oracle selection. This agrees with concurrent work by Naskar et al. (2020), whose reranking of 100 samples gained up to 3 BLEU over beam search. The paper appeared at COLING 2020 (pages 4506–4520).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

105 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 94 not yet.

- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://md.research.elamur.ai/free/en/papers/alves-2024-tower-open-multilingual-large-language/)
- [Quality-Aware Decoding for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/fernandes-2022-quality-aware-decoding-neural-machine/)
- [MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methods](https://md.research.elamur.ai/free/en/papers/finkelstein-2023-mbr-qe-finetuning-training-time/)
- [High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics](https://md.research.elamur.ai/free/en/papers/freitag-2022-high-quality-rather-than-high/)
- [Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent](https://md.research.elamur.ai/free/en/papers/freitag-2023-results-wmt23-metrics-shared-task/)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://md.research.elamur.ai/free/en/papers/li-2022-contrastive-decoding-open-ended-text/)
- [Locally Typical Sampling](https://md.research.elamur.ai/free/en/papers/meister-2023-locally-typical-sampling/)
- [Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics](https://md.research.elamur.ai/free/en/papers/perrella-2024-beyond-correlation-interpretable-evaluation-machine/)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://md.research.elamur.ai/free/en/papers/pillutla-2021-mauve-measuring-gap-between-neural/)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/)
- [Early-Exit and Instant Confidence Translation Quality Estimation](https://md.research.elamur.ai/free/en/papers/zouhar-2026-early-exit-instant-confidence-translation/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{eikema2020is,
  title = {{Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation}},
  author = {Eikema, Bryan and Aziz, Wilker},
  booktitle = {Proceedings of the 28th International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.398},
  url = {https://doi.org/10.18653/v1/2020.coling-main.398}
}
```

- [← previous Introducing various Semantic Models for Amharic: Experimentation and Evaluation with multiple Tasks and Datasets](https://md.research.elamur.ai/free/en/papers/yimam-2020-introducing-various-semantic-models-amharic/)
- [next → Is this sentence valid? An Arabic Dataset for Commonsense Validation](https://md.research.elamur.ai/free/en/papers/tawalbeh-2020-this-sentence-valid-arabic-dataset/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/)

## References

53 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Globally normalized transition-based neural networks. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape… [find](https://scholar.google.com/scholar?q=Daniel+Andor%2C+Chris+Alberti%2C+David+Weiss%2C+Aliaksei+Severyn%2C+Alessandro+Presta%2C+Kuzman+Ganchev%2C+Slav+Petrov%2C+and+Michael+Collins%2E+2016%2E+Globally+normalized+transition%2Dbased+neural+networks%2E+In+Proceedi) [our card](https://md.research.elamur.ai/free/en/papers/andor-2016-globally-normalized-transition-based-neural/)
2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In ICLR, 2015, San Diego, USA. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+Machine+Translation+by+Jointly+Learning+to+Align+and+Translate%2E+In+ICLR%2C+2015%2C+San+Diego%2C+USA%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
3. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 1171–1179. Curran… [find](https://scholar.google.com/scholar?q=Samy+Bengio%2C+Oriol+Vinyals%2C+Navdeep+Jaitly%2C+and+Noam+Shazeer%2E+2015%2E+Scheduled+sampling+for+sequence+prediction+with+recurrent+neural+networks%2E+In+C%2E+Cortes%2C+N%2E+D%2E+Lawrence%2C+D%2E+D%2E+Lee%2C+M%2E+Sugiyama%2C+and)
4. Peter J Bickel and Kjell A Doksum. 1977. Mathematical statistics: basic ideas and selected topics. Holden-Day Inc., Oakland, CA, USA. [find](https://scholar.google.com/scholar?q=Peter+J+Bickel+and+Kjell+A+Doksum%2E+1977%2E+Mathematical+statistics%3A+basic+ideas+and+selected+topics%2E+Holden%2DDay+Inc%2E%2C+Oakland%2C+CA%2C+USA%2E)
5. Frédéric Blain, Lucia Specia, and Pranava Madhyastha. 2017. Exploring hypotheses spaces in neural machine translation. Asia-Pacific Association for Machine Translation (AAMT), editor, Machine Translation Summit XVI. Nagoya, Japan. [find](https://scholar.google.com/scholar?q=Fre%CC%81de%CC%81ric+Blain%2C+Lucia+Specia%2C+and+Pranava+Madhyastha%2E+2017%2E+Exploring+hypotheses+spaces+in+neural+machine+translation%2E+Asia%2DPacific+Association+for+Machine+Translation+%28AAMT%29%2C+editor%2C+Machine+Transl)
6. David M Blei. 2014. Build, compute, critique, repeat: Data analysis with latent variable models. Annual Review of Statistics and Its Application, 1:203–232. [find](https://scholar.google.com/scholar?q=David+M+Blei%2E+2014%2E+Build%2C+compute%2C+critique%2C+repeat%3A+Data+analysis+with+latent+variable+models%2E+Annual+Review+of+Statistics+and+Its+Application%2C+1%3A203%E2%80%93232%2E)
7. Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (WMT18). In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 272–303, Belgium, Brussels, O… [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Christian+Federmann%2C+Mark+Fishel%2C+Yvette+Graham%2C+Barry+Haddow%2C+Philipp+Koehn%2C+and+Christof+Monz%2E+2018%2E+Findings+of+the+2018+conference+on+machine+translation+%28WMT18%29%2E+In+Proceedings+of+) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2018-findings-2018-conference-machine-translation/)
8. Sebastian Borgeaud and Guy Emerson. 2020. Leveraging sentence similarity in natural language generation: Improving beam search using range voting. In Proceedings of the Fourth Workshop on Neural Generation and Translation, pages 97–109, Online, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sebastian+Borgeaud+and+Guy+Emerson%2E+2020%2E+Leveraging+sentence+similarity+in+natural+language+generation%3A+Improving+beam+search+using+range+voting%2E+In+Proceedings+of+the+Fourth+Workshop+on+Neural+Gener)
9. Léon Bottou and Yann L. Cun. 2004. Large scale online learning. In S. Thrun, L. K. Saul, and B. Schölkopf, editors, Advances in Neural Information Processing Systems 16, pages 217–224. MIT Press. [find](https://scholar.google.com/scholar?q=Le%CC%81on+Bottou+and+Yann+L%2E+Cun%2E+2004%2E+Large+scale+online+learning%2E+In+S%2E+Thrun%2C+L%2E+K%2E+Saul%2C+and+B%2E+Scho%CC%88lkopf%2C+editors%2C+Advances+in+Neural+Information+Processing+Systems+16%2C+pages+217%E2%80%93224%2E+MIT+Press%2E)
10. Bottou. 1991. Une approche théorique de lapprentissage connexioniste; applications à la reconnaissance de la parole. [find](https://scholar.google.com/scholar?q=Bottou%2E+1991%2E+Une+approche+the%CC%81orique+de+lapprentissage+connexioniste%3B+applications+a%CC%80+la+reconnaissance+de+la+parole%2E)
11. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014a. On the properties of neural machine translation: Encoder–decoder approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103–111, Doha, Qatar, October.… [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2C+Bart+van+Merrie%CC%88nboer%2C+Dzmitry+Bahdanau%2C+and+Yoshua+Bengio%2E+2014a%2E+On+the+properties+of+neural+machine+translation%3A+Encoder%E2%80%93decoder+approaches%2E+In+Proceedings+of+SSST%2D8%2C+Eighth+Workshop) [our card](https://md.research.elamur.ai/free/en/papers/cho-2014-properties-neural-machine-translation-encoder/)
12. Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014b. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Lang… [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2C+Bart+van+Merrie%CC%88nboer%2C+Caglar+Gulcehre%2C+Dzmitry+Bahdanau%2C+Fethi+Bougares%2C+Holger+Schwenk%2C+and+Yoshua+Bengio%2E+2014b%2E+Learning+phrase+representations+using+RNN+encoder%E2%80%93decoder+for+statist)
13. Eldan Cohen and J. Christopher Beck. 2019. Empirical analysis of beam search performance degradation in neural sequence models. In ICML. [find](https://scholar.google.com/scholar?q=Eldan+Cohen+and+J%2E+Christopher+Beck%2E+2019%2E+Empirical+analysis+of+beam+search+performance+degradation+in+neural+sequence+models%2E+In+ICML%2E)
14. Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems. In Proceedings of WMT, 2011, pages 85–91, Edinburgh, Scotland, July. [find](https://scholar.google.com/scholar?q=Michael+Denkowski+and+Alon+Lavie%2E+2011%2E+Meteor+1%2E3%3A+Automatic+metric+for+reliable+optimization+and+evaluation+of+machine+translation+systems%2E+In+Proceedings+of+WMT%2C+2011%2C+pages+85%E2%80%9391%2C+Edinburgh%2C+Scotl) [our card](https://md.research.elamur.ai/free/en/papers/denkowski-2011-meteor-1-3-automatic-metric/)
15. Bryan Eikema and Wilker Aziz. 2019. Auto-encoding variational neural machine translation. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 124–141, Florence, Italy, August. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bryan+Eikema+and+Wilker+Aziz%2E+2019%2E+Auto%2Dencoding+variational+neural+machine+translation%2E+In+Proceedings+of+the+4th+Workshop+on+Representation+Learning+for+NLP+%28RepL4NLP%2D2019%29%2C+pages+124%E2%80%93141%2C+Florence)
16. Andrew Gelman, John B. Carlin, Hal S. Stern, and Donald B. Rubin. 2013. Bayesian Data Analysis. Chapman and Hall/CRC, 3rd ed. edition. [find](https://scholar.google.com/scholar?q=Andrew+Gelman%2C+John+B%2E+Carlin%2C+Hal+S%2E+Stern%2C+and+Donald+B%2E+Rubin%2E+2013%2E+Bayesian+Data+Analysis%2E+Chapman+and+Hall%2FCRC%2C+3rd+ed%2E+edition%2E)
17. Vaibhava Goel and William J. Byrne. 2000. Minimum bayes-risk automatic speech recognition. Comput. Speech Lang., 14(2):115–135. [find](https://scholar.google.com/scholar?q=Vaibhava+Goel+and+William+J%2E+Byrne%2E+2000%2E+Minimum+bayes%2Drisk+automatic+speech+recognition%2E+Comput%2E+Speech+Lang%2E%2C+14%282%29%3A115%E2%80%93135%2E)
18. Jiatao Gu, Graham Neubig, Kyunghyun Cho, and Victor O.K. Li. 2017. Learning to translate in real-time with neural machine translation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1053–1062, Valencia, Spa… [find](https://scholar.google.com/scholar?q=Jiatao+Gu%2C+Graham+Neubig%2C+Kyunghyun+Cho%2C+and+Victor+O%2EK%2E+Li%2E+2017%2E+Learning+to+translate+in+real%2Dtime+with+neural+machine+translation%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+) [our card](https://md.research.elamur.ai/free/en/papers/gu-2017-learning-translate-real-time-neural/)
19. Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019. The FLORES evaluation datasets for low-resource machine translation: Nepali–English and Sinhala–English. In Proceedings of the 2019 Conference on Empirical Metho… [find](https://scholar.google.com/scholar?q=Francisco+Guzma%CC%81n%2C+Peng%2DJen+Chen%2C+Myle+Ott%2C+Juan+Pino%2C+Guillaume+Lample%2C+Philipp+Koehn%2C+Vishrav+Chaudhary%2C+and+Marc%E2%80%99Aurelio+Ranzato%2E+2019%2E+The+FLORES+evaluation+datasets+for+low%2Dresource+machine+trans) [our card](https://md.research.elamur.ai/free/en/papers/guzman-2019-flores-evaluation-datasets-low-resource/)
20. P. E. Hart, N. J. Nilsson, and B. Raphael. 1968. A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics, 4(2):100–107. [find](https://scholar.google.com/scholar?q=P%2E+E%2E+Hart%2C+N%2E+J%2E+Nilsson%2C+and+B%2E+Raphael%2E+1968%2E+A+formal+basis+for+the+heuristic+determination+of+minimum+cost+paths%2E+IEEE+Transactions+on+Systems+Science+and+Cybernetics%2C+4%282%29%3A100%E2%80%93107%2E)
21. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ari+Holtzman%2C+Jan+Buys%2C+Li+Du%2C+Maxwell+Forbes%2C+and+Yejin+Choi%2E+2020%2E+The+curious+case+of+neural+text+degeneration%2E+In+International+Conference+on+Learning+Representations%2E)
22. Liang Huang, Kai Zhao, and Mingbo Ma. 2017. When to finish? optimal beam search for neural text generation (modulo beam size). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2134–2139, Copenhagen, Denmark, September. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Liang+Huang%2C+Kai+Zhao%2C+and+Mingbo+Ma%2E+2017%2E+When+to+finish%3F+optimal+beam+search+for+neural+text+generation+%28modulo+beam+size%29%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Lan)
23. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28–39, Vancouver, August. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Philipp+Koehn+and+Rebecca+Knowles%2E+2017%2E+Six+challenges+for+neural+machine+translation%2E+In+Proceedings+of+the+First+Workshop+on+Neural+Machine+Translation%2C+pages+28%E2%80%9339%2C+Vancouver%2C+August%2E+Association+)
24. Shankar Kumar and William Byrne. 2004. Minimum Bayes-risk decoding for statistical machine translation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 169–176, Boston, Massachusetts, USA… [find](https://scholar.google.com/scholar?q=Shankar+Kumar+and+William+Byrne%2E+2004%2E+Minimum+Bayes%2Drisk+decoding+for+statistical+machine+translation%2E+In+Proceedings+of+the+Human+Language+Technology+Conference+of+the+North+American+Chapter+of+the+) [our card](https://md.research.elamur.ai/free/en/papers/kumar-2004-minimum-bayes-risk-decoding-statistical/)
25. Aviral Kumar and Sunita Sarawagi. 2019. Calibration of encoder decoder models for neural machine translation. arXiv preprint arXiv:1903.00802. [find](https://scholar.google.com/scholar?q=Aviral+Kumar+and+Sunita+Sarawagi%2E+2019%2E+Calibration+of+encoder+decoder+models+for+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1903%2E00802%2E)
26. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML 01, page 282289, San Francisco, CA, USA. Morgan Kaufmann… [find](https://scholar.google.com/scholar?q=John+D%2E+Lafferty%2C+Andrew+McCallum%2C+and+Fernando+C%2E+N%2E+Pereira%2E+2001%2E+Conditional+random+fields%3A+Probabilistic+models+for+segmenting+and+labeling+sequence+data%2E+In+Proceedings+of+the+Eighteenth+Interna)
27. Qingsong Ma, Ondřej Bojar, and Yvette Graham. 2018. Results of the WMT18 metrics shared task: Both characters and embeddings achieve good performance. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 671–688, Belgium, Brussels, October. Association for Comput… [find](https://scholar.google.com/scholar?q=Qingsong+Ma%2C+Ondr%CC%8Cej+Bojar%2C+and+Yvette+Graham%2E+2018%2E+Results+of+the+WMT18+metrics+shared+task%3A+Both+characters+and+embeddings+achieve+good+performance%2E+In+Proceedings+of+the+Third+Conference+on+Machin) [our card](https://md.research.elamur.ai/free/en/papers/ma-2018-results-wmt18-metrics-shared-task/)
28. David J. C. MacKay. 2003. Information Theory, Inference & Learning Algorithms. Cambridge University Press. [find](https://scholar.google.com/scholar?q=David+J%2E+C%2E+MacKay%2E+2003%2E+Information+Theory%2C+Inference+%26+Learning+Algorithms%2E+Cambridge+University+Press%2E)
29. Kenton Murray and David Chiang. 2018. Correcting length bias in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 212–223, Brussels, Belgium, October. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kenton+Murray+and+David+Chiang%2E+2018%2E+Correcting+length+bias+in+neural+machine+translation%2E+In+Proceedings+of+the+Third+Conference+on+Machine+Translation%3A+Research+Papers%2C+pages+212%E2%80%93223%2C+Brussels%2C+Bel)
30. Subhajit Naskar, Amirmohammad Rooshenas, Simeng Sun, Mohit Iyyer, and Andrew McCallum. 2020. Energy-based reranking: Improving neural machine translation using energy-based models. CoRR, abs/2009.13267. [find](https://scholar.google.com/scholar?q=Subhajit+Naskar%2C+Amirmohammad+Rooshenas%2C+Simeng+Sun%2C+Mohit+Iyyer%2C+and+Andrew+McCallum%2E+2020%2E+Energy%2Dbased+reranking%3A+Improving+neural+machine+translation+using+energy%2Dbased+models%2E+CoRR%2C+abs%2F2009%2E1326)
31. Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages… [find](https://scholar.google.com/scholar?q=Myle+Ott%2C+Michael+Auli%2C+David+Grangier%2C+and+Marc%E2%80%99Aurelio+Ranzato%2E+2018%2E+Analyzing+uncertainty+in+neural+machine+translation%2E+In+Jennifer+Dy+and+Andreas+Krause%2C+editors%2C+Proceedings+of+the+35th+Interna) [our card](https://md.research.elamur.ai/free/en/papers/ott-2018-analyzing-uncertainty-neural-machine-translation/)
32. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of ACL, 2002, pages 311–318, Philadelphia, USA, July. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+BLEU%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+ACL%2C+2002%2C+pages+311%E2%80%93318%2C+Philadelphia%2C+USA%2C+July%2E) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
33. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. [find](https://scholar.google.com/scholar?q=Marc%E2%80%99Aurelio+Ranzato%2C+Sumit+Chopra%2C+Michael+Auli%2C+and+Wojciech+Zaremba%2E+2016%2E+Sequence+level+training+with+recurrent+neural+networks%2E+In+4th+International+Conference+on+Learning+Representations%2C+ICLR+) [our card](https://md.research.elamur.ai/free/en/papers/ranzato-2015-sequence-level-training-recurrent-neural/)
34. Herbert Robbins and Sutton Monro. 1951. A stochastic approximation method. Ann. Math. Statist., 22(3):400– 407. [find](https://scholar.google.com/scholar?q=Herbert+Robbins+and+Sutton+Monro%2E+1951%2E+A+stochastic+approximation+method%2E+Ann%2E+Math%2E+Statist%2E%2C+22%283%29%3A400%E2%80%93+407%2E)
35. Harshil Shah and David Barber. 2018. Generative neural machine translation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 1346–1355. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Harshil+Shah+and+David+Barber%2E+2018%2E+Generative+neural+machine+translation%2E+In+S%2E+Bengio%2C+H%2E+Wallach%2C+H%2E+Larochelle%2C+K%2E+Grauman%2C+N%2E+Cesa%2DBianchi%2C+and+R%2E+Garnett%2C+editors%2C+Advances+in+Neural+Informatio)
36. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1683–1692, Berlin, Germany, August. As… [find](https://scholar.google.com/scholar?q=Shiqi+Shen%2C+Yong+Cheng%2C+Zhongjun+He%2C+Wei+He%2C+Hua+Wu%2C+Maosong+Sun%2C+and+Yang+Liu%2E+2016%2E+Minimum+risk+training+for+neural+machine+translation%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association) [our card](https://md.research.elamur.ai/free/en/papers/shen-2015-minimum-risk-training-neural-machine/)
37. Raphael Shu and Hideki Nakayama. 2017. Later-stage minimum bayes-risk decoding for neural machine translation. CoRR, abs/1704.03169. [find](https://scholar.google.com/scholar?q=Raphael+Shu+and+Hideki+Nakayama%2E+2017%2E+Later%2Dstage+minimum+bayes%2Drisk+decoding+for+neural+machine+translation%2E+CoRR%2C+abs%2F1704%2E03169%2E)
38. Raphael Shu and Hideki Nakayama. 2018. Improving beam search by removing monotonic constraint for neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 339–344, Melbourne, Australia, July. Association fo… [find](https://scholar.google.com/scholar?q=Raphael+Shu+and+Hideki+Nakayama%2E+2018%2E+Improving+beam+search+by+removing+monotonic+constraint+for+neural+machine+translation%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computati)
39. Noah A. Smith. 2011. Linguistic Structure Prediction. Morgan and Claypool. [find](https://scholar.google.com/scholar?q=Noah+A%2E+Smith%2E+2011%2E+Linguistic+Structure+Prediction%2E+Morgan+and+Claypool%2E)
40. Pavel Sountsov and Sunita Sarawagi. 2016. Length bias in encoder decoder models and a case for global conditioning. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1516–1525, Austin, Texas, November. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pavel+Sountsov+and+Sunita+Sarawagi%2E+2016%2E+Length+bias+in+encoder+decoder+models+and+a+case+for+global+conditioning%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Proce)
41. Felix Stahlberg and Bill Byrne. 2019. On NMT search errors and model errors: Cat got your tongue? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3347–3353, Hong… [find](https://scholar.google.com/scholar?q=Felix+Stahlberg+and+Bill+Byrne%2E+2019%2E+On+NMT+search+errors+and+model+errors%3A+Cat+got+your+tongue%3F+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th+) [our card](https://md.research.elamur.ai/free/en/papers/stahlberg-2019-nmt-search-errors-model-errors/)
42. Felix Stahlberg, Adrià de Gispert, Eva Hasler, and Bill Byrne. 2017. Neural machine translation by minimising the Bayes-risk with respect to syntactic translation lattices. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Shor… [find](https://scholar.google.com/scholar?q=Felix+Stahlberg%2C+Adria%CC%80+de+Gispert%2C+Eva+Hasler%2C+and+Bill+Byrne%2E+2017%2E+Neural+machine+translation+by+minimising+the+Bayes%2Drisk+with+respect+to+syntactic+translation+lattices%2E+In+Proceedings+of+the+15th)
43. Miloš Stanojević and Mark Steedman. 2020. Max-Margin Incremental CCG Parsing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Milos%CC%8C+Stanojevic%CC%81+and+Mark+Steedman%2E+2020%2E+Max%2DMargin+Incremental+CCG+Parsing%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers%29%2E+Asso)
44. Mitchell Stern, Daniel Fried, and Dan Klein. 2017. Effective inference for generative neural parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1695–1700, Copenhagen, Denmark, September. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mitchell+Stern%2C+Daniel+Fried%2C+and+Dan+Klein%2E+2017%2E+Effective+inference+for+generative+neural+parsing%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+1)
45. Ilya Sutskever, Oriol Vinyals, and Quoc V. V Le. 2014. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N.D. Lawrence, and K.Q. Weinberger, editors, NIPS, 2014, pages 3104–3112. Montreal, Canada. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V%2E+V+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Z%2E+Ghahramani%2C+M%2E+Welling%2C+C%2E+Cortes%2C+N%2ED%2E+Lawrence%2C+and+K%2EQ%2E+Weinberger%2C+editors%2C+NIPS%2C+2)
46. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826. [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Vincent+Vanhoucke%2C+Sergey+Ioffe%2C+Jon+Shlens%2C+and+Zbigniew+Wojna%2E+2016%2E+Rethinking+the+inception+architecture+for+computer+vision%2E+In+Proceedings+of+the+IEEE+conference+on+computer+v)
47. Roy Tromble, Shankar Kumar, Franz Och, and Wolfgang Macherey. 2008. Lattice Minimum Bayes-Risk decoding for statistical machine translation. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 620–629, Honolulu, Hawaii, October. Association for Computatio… [find](https://scholar.google.com/scholar?q=Roy+Tromble%2C+Shankar+Kumar%2C+Franz+Och%2C+and+Wolfgang+Macherey%2E+2008%2E+Lattice+Minimum+Bayes%2DRisk+decoding+for+statistical+machine+translation%2E+In+Proceedings+of+the+2008+Conference+on+Empirical+Methods+)
48. Vladimir Vapnik. 1998. Statistical learning theory Wiley. New York, 1:624. [find](https://scholar.google.com/scholar?q=Vladimir+Vapnik%2E+1998%2E+Statistical+learning+theory+Wiley%2E+New+York%2C+1%3A624%2E)
49. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, pages 6000–6010. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+NIPS%2C+pages+6000%E2%80%936010%2E)
50. Sam Wiseman and Alexander M. Rush. 2016. Sequence-to-sequence learning as beam-search optimization. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1296–1306, Austin, Texas, November. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sam+Wiseman+and+Alexander+M%2E+Rush%2E+2016%2E+Sequence%2Dto%2Dsequence+learning+as+beam%2Dsearch+optimization%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+129)
51. Yilin Yang, Liang Huang, and Mingbo Ma. 2018. Breaking the beam search curse: A study of (re-)scoring methods and stopping criteria for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3054–3059, Brussels, Belgium, October-N… [find](https://scholar.google.com/scholar?q=Yilin+Yang%2C+Liang+Huang%2C+and+Mingbo+Ma%2E+2018%2E+Breaking+the+beam+search+curse%3A+A+study+of+%28re%2D%29scoring+methods+and+stopping+criteria+for+neural+machine+translation%2E+In+Proceedings+of+the+2018+Conferenc)
52. Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. 2019. Bridging the gap between training and inference for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4334–4343, Florence, Italy, July. Association for Computati… [find](https://scholar.google.com/scholar?q=Wen+Zhang%2C+Yang+Feng%2C+Fandong+Meng%2C+Di+You%2C+and+Qun+Liu%2E+2019%2E+Bridging+the+gap+between+training+and+inference+for+neural+machine+translation%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Associat) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2019-bridging-gap-between-training-inference/)
53. Hugh Zhang, Daniel Duckworth, Daphne Ippolito, and Arvind Neelakantan. 2020. Trading off diversity and quality in natural language generation. CoRR, abs/2004.10450. [find](https://scholar.google.com/scholar?q=Hugh+Zhang%2C+Daniel+Duckworth%2C+Daphne+Ippolito%2C+and+Arvind+Neelakantan%2E+2020%2E+Trading+off+diversity+and+quality+in+natural+language+generation%2E+CoRR%2C+abs%2F2004%2E10450%2E)