---
title: "Looking for a Needle in a Haystack — Guerreiro 2023"
description: "Hallucinations in NMT are rare but severe: translations whose content drifts away from the source."
authors: ["Nuno M. Guerreiro", "Elena Voita", "André Martins"]
year: 2023
venue: "Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics"
source: "https://aclanthology.org/2023.eacl-main.75/"
doi: "10.18653/v1/2023.eacl-main.75"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/guerreiro-2023-looking-needle-haystack-comprehensive-study/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/guerreiro-2023-looking-needle-haystack-comprehensive-study/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translation

# Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translation

**Authors:** [Nuno M. Guerreiro](https://md.research.elamur.ai/free/en/authors/nuno-m-guerreiro/), [Elena Voita](https://md.research.elamur.ai/free/en/authors/elena-voita/), [André Martins](https://md.research.elamur.ai/free/en/authors/andre-martins/) · **Year:** 2023 · **Venue:** Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.eacl-main.75/) · [PDF](https://aclanthology.org/2023.eacl-main.75.pdf) · [DOI 10.18653/v1/2023.eacl-main.75](https://doi.org/10.18653/v1/2023.eacl-main.75) · [OpenAlex](https://openalex.org/W4386566643)

**Citations:** 60 citations (openalex-snapshot · 2026-10-14) · [87](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Hallucinations in NMT are rare but severe: translations whose content drifts away from the source. Earlier studies mostly made them more frequent artificially, through noisy data, perturbed inputs or domain shift. They also used inconsistent definitions and seldom checked whether their detectors actually work. Guerreiro, Voita and Martins instead study a natural setting with in-domain data and no injected noise, where the task is to find a tiny "needle" among many ordinary outputs (examples are German→English). They introduce a taxonomy that separates oscillatory, strongly detached and fully detached hallucinations from other critical errors. Using this taxonomy, they collect a human-annotated corpus of flagged translations. They then compare earlier heuristics, quality-estimation filters and newly proposed uncertainty-based detectors. Plain sequence log-probability turns out to be the strongest detector, matching even reference-based COMET. Building on this, they propose DeHallucinator, a test-time fix that resamples and reranks flagged outputs and substantially cuts the hallucination rate. The released data and code give a reliable basis for evaluating and preventing hallucinations.

## Key points
- The authors release a corpus of 3,415 structured annotations covering hallucination types and other critical errors, and they also make public the model, the data it was trained on and their code (github.com/deep-spin/hallucinations-in-nmt).
- Detectors from earlier work perform poorly when high recall is needed. These include attention-based criteria (Attn-to-EOS, Attn-ign-SRC) and binary heuristics such as TNG and RT; TNG underperforms even on the oscillatory cases it was designed to catch.
- Seq-Logprob, the length-normalised log-probability of the output, is the best detector overall and performs on par with reference-based COMET.
- The reference-free COMET-QE does not penalise fully detached hallucinations enough, although it does well on oscillatory ones.
- DeHallucinator works in three steps:
  - It flags translations with Seq-Logprob.
  - It generates 10 MC-dropout hypotheses for each flagged translation.
  - It picks the best hypothesis using COMET-QE, which proved a better reranker than Seq-Logprob.
- Tested on 200 flagged translations, DeHallucinator raised the share of correct outputs from 33% to 85% and reduced hallucinations roughly threefold.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

87 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 80 not yet.

- [Findings of the WMT 2023 Shared Task on Quality Estimation](https://md.research.elamur.ai/free/en/papers/blain-2023-findings-wmt-2023-shared-task/)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://md.research.elamur.ai/free/en/papers/guerreiro-2024-xcomet-transparent-machine-translation-evaluation/)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/)
- [When Flores Bloomz Wrong: Cross-Direction Contamination in Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/tan-2026-when-flores-bloomz-wrong-cross/)
- [Document-Level Machine Translation with Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2023-document-level-machine-translation-large/)
- [Hallucination is Inevitable: An Innate Limitation of Large Language Models](https://md.research.elamur.ai/free/en/papers/xu-2024-hallucination-inevitable-innate-limitation-large/)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2023-siren-s-song-ai-ocean/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{guerreiro2023looking,
  title = {{Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translation}},
  author = {Guerreiro, Nuno M. and Voita, Elena and Martins, André},
  booktitle = {Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.eacl-main.75},
  url = {https://doi.org/10.18653/v1/2023.eacl-main.75}
}
```

- [← previous LooGLE: Can Long-Context Language Models Understand Long Contexts?](https://md.research.elamur.ai/free/en/papers/li-2023-loogle-long-context-language-models/)
- [next → Lost in the Middle: How Language Models Use Long Contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

60 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ondřej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina España-Bonet, Angela Fan, Christian Federmann, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christoph… [find](https://scholar.google.com/scholar?q=Farhad+Akhbardeh%2C+Arkady+Arkhangorodsky%2C+Magdalena+Biesialska%2C+Ondr%CC%8Cej+Bojar%2C+Rajen+Chatterjee%2C+Vishrav+Chaudhary%2C+Marta+R%2E+Costa%2Djussa%2C+Cristina+Espa%C3%B1a%2DBonet%2C+Angela+Fan%2C+Christian+Federmann%2C+Markus+) [our card](https://md.research.elamur.ai/free/en/papers/akhbardeh-2021-findings-2021-conference-machine-translation/)
2. Chantal Amrhein and Rico Sennrich. 2022. Identifying weaknesses in machine translation metrics through minimum bayes risk decoding: A case study for comet. [find](https://scholar.google.com/scholar?q=Chantal+Amrhein+and+Rico+Sennrich%2E+2022%2E+Identifying+weaknesses+in+machine+translation+metrics+through+minimum+bayes+risk+decoding%3A+A+case+study+for+comet%2E)
3. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Ass… [find](https://scholar.google.com/scholar?q=Satanjeev+Banerjee+and+Alon+Lavie%2E+2005%2E+METEOR%3A+An+automatic+metric+for+MT+evaluation+with+improved+correlation+with+human+judgments%2E+In+Proceedings+of+the+ACL+Workshop+on+Intrinsic+and+Extrinsic+Eva) [our card](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/)
4. Jasmijn Bastings and Katja Filippova. 2020. The elephant in the interpretability room: Why use attention as explanation when we have saliency methods? In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 149–155, Online. Association for Comput… [find](https://scholar.google.com/scholar?q=Jasmijn+Bastings+and+Katja+Filippova%2E+2020%2E+The+elephant+in+the+interpretability+room%3A+Why+use+attention+as+explanation+when+we+have+saliency+methods%3F+In+Proceedings+of+the+Third+BlackboxNLP+Workshop+)
5. Alexandre Berard, Ioan Calapodescu, and Claude Roux. 2019. Naver labs Europe’s systems for the WMT19 machine translation robustness task. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 526– 532, Florence, Italy. Association for Computation… [find](https://scholar.google.com/scholar?q=Alexandre+Berard%2C+Ioan+Calapodescu%2C+and+Claude+Roux%2E+2019%2E+Naver+labs+Europe%E2%80%99s+systems+for+the+WMT19+machine+translation+robustness+task%2E+In+Proceedings+of+the+Fourth+Conference+on+Machine+Translation)
6. Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (WMT18). In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 272–303, Belgium, Brussels. A… [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Christian+Federmann%2C+Mark+Fishel%2C+Yvette+Graham%2C+Barry+Haddow%2C+Philipp+Koehn%2C+and+Christof+Monz%2E+2018%2E+Findings+of+the+2018+conference+on+machine+translation+%28WMT18%29%2E+In+Proceedings+of+) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2018-findings-2018-conference-machine-translation/)
7. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. [find](https://scholar.google.com/scholar?q=Jacob+Cohen%2E+1960%2E+A+coefficient+of+agreement+for+nominal+scales%2E+Educational+and+Psychological+Measurement%2C+20%281%29%3A37%E2%80%9346%2E)
8. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Unsupervised+cross%2Dlingu)
9. Patrick Fernandes, António Farinhas, Ricardo Rei, José De Souza, Perez Ogayo, Graham Neubig, and Andre Martins. 2022. Quality-aware decoding for neural machine translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan… [find](https://scholar.google.com/scholar?q=Patrick+Fernandes%2C+Ant%C3%B3nio+Farinhas%2C+Ricardo+Rei%2C+Jos%C3%A9+De+Souza%2C+Perez+Ogayo%2C+Graham+Neubig%2C+and+Andre+Martins%2E+2022%2E+Quality%2Daware+decoding+for+neural+machine+translation%2E+In+Proceedings+of+the+2022+) [our card](https://md.research.elamur.ai/free/en/papers/fernandes-2022-quality-aware-decoding-neural-machine/)
10. Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8:539–555. [find](https://scholar.google.com/scholar?q=Marina+Fomicheva%2C+Shuo+Sun%2C+Lisa+Yankovskaya%2C+Fr%C3%A9d%C3%A9ric+Blain%2C+Francisco+Guzm%C3%A1n%2C+Mark+Fishel%2C+Nikolaos+Aletras%2C+Vishrav+Chaudhary%2C+and+Lucia+Specia%2E+2020%2E+Unsupervised+quality+estimation+for+neural+mac) [our card](https://md.research.elamur.ai/free/en/papers/fomicheva-2020-unsupervised-quality-estimation-neural-machine/)
11. Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. 2022. High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics. Transactions of the Association for Computational Linguistics, 10:811–825. [find](https://scholar.google.com/scholar?q=Markus+Freitag%2C+David+Grangier%2C+Qijun+Tan%2C+and+Bowen+Liang%2E+2022%2E+High+Quality+Rather+than+High+Model+Probability%3A+Minimum+Bayes+Risk+Decoding+with+Neural+Metrics%2E+Transactions+of+the+Association+for+) [our card](https://md.research.elamur.ai/free/en/papers/freitag-2022-high-quality-rather-than-high/)
12. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021. Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain. In Proceedings of the Sixth Conference on Machine Transl… [find](https://scholar.google.com/scholar?q=Markus+Freitag%2C+Ricardo+Rei%2C+Nitika+Mathur%2C+Chi%2Dkiu+Lo%2C+Craig+Stewart%2C+George+Foster%2C+Alon+Lavie%2C+and+Ondr%CC%8Cej+Bojar%2E+2021%2E+Results+of+the+WMT21+metrics+shared+task%3A+Evaluating+metrics+with+expert%2Dbase) [our card](https://md.research.elamur.ai/free/en/papers/freitag-2021-results-wmt21-metrics-shared-task/)
13. Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1050–1059, New York, New York, USA. PMLR. [find](https://scholar.google.com/scholar?q=Yarin+Gal+and+Zoubin+Ghahramani%2E+2016%2E+Dropout+as+a+bayesian+approximation%3A+Representing+model+uncertainty+in+deep+learning%2E+In+Proceedings+of+The+33rd+International+Conference+on+Machine+Learning%2C+vo)
14. Junjie Hu, Hiroaki Hayashi, Kyunghyun Cho, and Graham Neubig. 2022. DEEP: DEnoising entity pre-training for neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1753–1766, Dublin, Ireland. Association fo… [find](https://scholar.google.com/scholar?q=Junjie+Hu%2C+Hiroaki+Hayashi%2C+Kyunghyun+Cho%2C+and+Graham+Neubig%2E+2022%2E+DEEP%3A+DEnoising+entity+pre%2Dtraining+for+neural+machine+translation%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for)
15. Sarthak Jain and Byron C. Wallace. 2019. Attention is not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543–3556, Minneapolis, Minnesota. Associa… [find](https://scholar.google.com/scholar?q=Sarthak+Jain+and+Byron+C%2E+Wallace%2E+2019%2E+Attention+is+not+Explanation%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+Langu) [our card](https://md.research.elamur.ai/free/en/papers/jain-2019-attention-is-not-explanation/)
16. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022. Survey of hallucination in natural language generation. [find](https://scholar.google.com/scholar?q=Ziwei+Ji%2C+Nayeon+Lee%2C+Rita+Frieske%2C+Tiezheng+Yu%2C+Dan+Su%2C+Yan+Xu%2C+Etsuko+Ishii%2C+Yejin+Bang%2C+Andrea+Madotto%2C+and+Pascale+Fung%2E+2022%2E+Survey+of+hallucination+in+natural+language+generation%2E) [our card](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/)
17. Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. [find](https://scholar.google.com/scholar?q=Tom+Kocmi%2C+Christian+Federmann%2C+Roman+Grundkiewicz%2C+Marcin+Junczys%2DDowmunt%2C+Hitokazu+Matsushita%2C+and+Arul+Menezes%2E+2021%2E+To+ship+or+not+to+ship%3A+An+extensive+evaluation+of+automatic+metrics+for+machin)
18. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28–39, Vancouver. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Philipp+Koehn+and+Rebecca+Knowles%2E+2017%2E+Six+challenges+for+neural+machine+translation%2E+In+Proceedings+of+the+First+Workshop+on+Neural+Machine+Translation%2C+pages+28%E2%80%9339%2C+Vancouver%2E+Association+for+Comp)
19. Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro… [find](https://scholar.google.com/scholar?q=Julia+Kreutzer%2C+Isaac+Caswell%2C+Lisa+Wang%2C+Ahsan+Wahab%2C+Daan+van+Esch%2C+Nasanbayar+Ulzii%2DOrshikh%2C+Allahsera+Tapo%2C+Nishant+Subramani%2C+Artem+Sokolov%2C+Claytone+Sikasote%2C+Monang+Setyawan%2C+Supheakmungkol+Sar) [our card](https://md.research.elamur.ai/free/en/papers/kreutzer-2021-quality-glance-audit-web-crawled/)
20. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ… [find](https://scholar.google.com/scholar?q=Taku+Kudo+and+John+Richardson%2E+2018%2E+SentencePiece%3A+A+simple+and+language+independent+subword+tokenizer+and+detokenizer+for+neural+text+processing%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+M) [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)
21. Aviral Kumar and Sunita Sarawagi. 2019. Calibration of encoder decoder models for neural machine translation. CoRR, abs/1903.00802. [find](https://scholar.google.com/scholar?q=Aviral+Kumar+and+Sunita+Sarawagi%2E+2019%2E+Calibration+of+encoder+decoder+models+for+neural+machine+translation%2E+CoRR%2C+abs%2F1903%2E00802%2E)
22. Ann Lee, Michael Auli, and Marc’Aurelio Ranzato. 2021. Discriminative reranking for neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers… [find](https://scholar.google.com/scholar?q=Ann+Lee%2C+Michael+Auli%2C+and+Marc%E2%80%99Aurelio+Ranzato%2E+2021%2E+Discriminative+reranking+for+neural+machine+translation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguisti)
23. Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo. 2018. Hallucinations in neural machine translation. [find](https://scholar.google.com/scholar?q=Katherine+Lee%2C+Orhan+Firat%2C+Ashish+Agarwal%2C+Clara+Fannjiang%2C+and+David+Sussillo%2E+2018%2E+Hallucinations+in+neural+machine+translation%2E)
24. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),… [find](https://scholar.google.com/scholar?q=Katherine+Lee%2C+Daphne+Ippolito%2C+Andrew+Nystrom%2C+Chiyuan+Zhang%2C+Douglas+Eck%2C+Chris+Callison%2DBurch%2C+and+Nicholas+Carlini%2E+2022%2E+Deduplicating+training+data+makes+language+models+better%2E+In+Proceedings+o)
25. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the… [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Yinhan+Liu%2C+Naman+Goyal%2C+Marjan+Ghazvininejad%2C+Abdelrahman+Mohamed%2C+Omer+Levy%2C+Veselin+Stoyanov%2C+and+Luke+Zettlemoyer%2E+2020%2E+BART%3A+Denoising+sequence%2Dto%2Dsequence+pre%2Dtraining+for+natural+l) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
26. Panpan Li, Mengxiang Wang, and Jian Wang. 2021. Named entity translation method based on machine translation lexicon. Neural Comput. Appl., 33(9):3977–3985. [find](https://scholar.google.com/scholar?q=Panpan+Li%2C+Mengxiang+Wang%2C+and+Jian+Wang%2E+2021%2E+Named+entity+translation+method+based+on+machine+translation+lexicon%2E+Neural+Comput%2E+Appl%2E%2C+33%289%29%3A3977%E2%80%933985%2E)
27. Arle Lommel, Aljoscha Burchardt, and Hans Uszkoreit. 2014. Multidimensional quality metrics (mqm): A framework for declaring and describing translation quality metrics. Tradumàtica: tecnologies de la traducció, 0:455–463. [find](https://scholar.google.com/scholar?q=Arle+Lommel%2C+Aljoscha+Burchardt%2C+and+Hans+Uszkoreit%2E+2014%2E+Multidimensional+quality+metrics+%28mqm%29%3A+A+framework+for+declaring+and+describing+translation+quality+metrics%2E+Tradum%C3%A0tica%3A+tecnologies+de+la+)
28. Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020. Results of the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 688–725, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nitika+Mathur%2C+Johnny+Wei%2C+Markus+Freitag%2C+Qingsong+Ma%2C+and+Ondr%CC%8Cej+Bojar%2E+2020%2E+Results+of+the+WMT20+metrics+shared+task%2E+In+Proceedings+of+the+Fifth+Conference+on+Machine+Translation%2C+pages+688%E2%80%93725%2C) [our card](https://md.research.elamur.ai/free/en/papers/mathur-2020-results-wmt20-metrics-shared-task/)
29. Mathias Müller, Annette Rios, and Rico Sennrich. 2020. Domain robustness in neural machine translation. In Proceedings of the 14th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 151–164, Virtual. Association for Machine Translation in the Amer… [find](https://scholar.google.com/scholar?q=Mathias+M%C3%BCller%2C+Annette+Rios%2C+and+Rico+Sennrich%2E+2020%2E+Domain+robustness+in+neural+machine+translation%2E+In+Proceedings+of+the+14th+Conference+of+the+Association+for+Machine+Translation+in+the+Americas) [our card](https://md.research.elamur.ai/free/en/papers/muller-2020-domain-robustness-neural-machine-translation/)
30. Mathias Müller and Rico Sennrich. 2021. Understanding the properties of minimum Bayes risk decoding in neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vo… [find](https://scholar.google.com/scholar?q=Mathias+M%C3%BCller+and+Rico+Sennrich%2E+2021%2E+Understanding+the+properties+of+minimum+Bayes+risk+decoding+in+neural+machine+translation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Comp)
31. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstr… [find](https://scholar.google.com/scholar?q=Myle+Ott%2C+Sergey+Edunov%2C+Alexei+Baevski%2C+Angela+Fan%2C+Sam+Gross%2C+Nathan+Ng%2C+David+Grangier%2C+and+Michael+Auli%2E+2019%2E+fairseq%3A+A+fast%2C+extensible+toolkit+for+sequence+modeling%2E+In+Proceedings+of+the+2019) [our card](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/)
32. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Matt+Post%2E+2018%2E+A+call+for+clarity+in+reporting+BLEU+scores%2E+In+Proceedings+of+the+Third+Conference+on+Machine+Translation%3A+Research+Papers%2C+pages+186%E2%80%93+191%2C+Brussels%2C+Belgium%2E+Association+for+Computa) [our card](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/)
33. Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, and Zachary C. Lipton. 2020. Learning to deceive with attention-based explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4782– 4793, Online. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Danish+Pruthi%2C+Mansi+Gupta%2C+Bhuwan+Dhingra%2C+Graham+Neubig%2C+and+Zachary+C%2E+Lipton%2E+2020%2E+Learning+to+deceive+with+attention%2Dbased+explanations%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Associat)
34. Gema Ramírez-Sánchez, Jaume Zaragoza-Bernabeu, Marta Bañón, and Sergio Ortiz-Rojas. 2020. Bifixer and bicleaner: two open-source tools to clean your parallel data. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 291–298, Lisboa, Portugal. Europ… [find](https://scholar.google.com/scholar?q=Gema+Ram%C3%ADrez%2DS%C3%A1nchez%2C+Jaume+Zaragoza%2DBernabeu%2C+Marta+Ba%C3%B1%C3%B3n%2C+and+Sergio+Ortiz%2DRojas%2E+2020%2E+Bifixer+and+bicleaner%3A+two+open%2Dsource+tools+to+clean+your+parallel+data%2E+In+Proceedings+of+the+22nd+Annual+Co) [our card](https://md.research.elamur.ai/free/en/papers/sanchez-2020-bifixer-bicleaner-two-open-source/)
35. Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt. 2021. The curious case of hallucinations in neural machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1172–1183, Online.… [find](https://scholar.google.com/scholar?q=Vikas+Raunak%2C+Arul+Menezes%2C+and+Marcin+Junczys%2DDowmunt%2E+2021%2E+The+curious+case+of+hallucinations+in+neural+machine+translation%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+t) [our card](https://md.research.elamur.ai/free/en/papers/raunak-2021-curious-case-hallucinations-neural-machine/)
36. Vikas Raunak, Matt Post, and Arul Menezes. 2022. Salted: A framework for salient long-tail translation error detection. [find](https://scholar.google.com/scholar?q=Vikas+Raunak%2C+Matt+Post%2C+and+Arul+Menezes%2E+2022%2E+Salted%3A+A+framework+for+salient+long%2Dtail+translation+error+detection%2E)
37. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020a. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Craig+Stewart%2C+Ana+C+Farinha%2C+and+Alon+Lavie%2E+2020a%2E+COMET%3A+A+neural+framework+for+MT+evaluation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing) [our card](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/)
38. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020b. Unbabel’s participation in the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 911–920, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Craig+Stewart%2C+Ana+C+Farinha%2C+and+Alon+Lavie%2E+2020b%2E+Unbabel%E2%80%99s+participation+in+the+WMT20+metrics+shared+task%2E+In+Proceedings+of+the+Fifth+Conference+on+Machine+Translation%2C+pages+911%E2%80%93920) [our card](https://md.research.elamur.ai/free/en/papers/rei-2020-unbabel-s-participation-wmt20-metrics/)
39. Víctor M. Sánchez-Cartagena, Marta Bañón, Sergio Ortiz-Rojas, and Gema Ramírez-Sánchez. 2018. Prompsit’s submission to wmt 2018 parallel corpus filtering shared task. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, Brussels, Belgium. Association for Compu… [find](https://scholar.google.com/scholar?q=V%C3%ADctor+M%2E+S%C3%A1nchez%2DCartagena%2C+Marta+Ba%C3%B1%C3%B3n%2C+Sergio+Ortiz%2DRojas%2C+and+Gema+Ram%C3%ADrez%2DS%C3%A1nchez%2E+2018%2E+Prompsit%E2%80%99s+submission+to+wmt+2018+parallel+corpus+filtering+shared+task%2E+In+Proceedings+of+the+Third+Confe) [our card](https://md.research.elamur.ai/free/en/papers/cartagena-2018-prompsit-s-submission-wmt-2018/)
40. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Ling)
41. Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2931–2951, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sofia+Serrano+and+Noah+A%2E+Smith%2E+2019%2E+Is+attention+interpretable%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+2931%E2%80%932951%2C+Florence%2C+Italy%2E+Associa) [our card](https://md.research.elamur.ai/free/en/papers/serrano-2019-is-attention-interpretable/)
42. Libin Shen, Anoop Sarkar, and Franz Josef Och. 2004. Discriminative reranking for machine translation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 177–184, Boston, Massachusetts, USA.… [find](https://scholar.google.com/scholar?q=Libin+Shen%2C+Anoop+Sarkar%2C+and+Franz+Josef+Och%2E+2004%2E+Discriminative+reranking+for+machine+translation%2E+In+Proceedings+of+the+Human+Language+Technology+Conference+of+the+North+American+Chapter+of+the+A)
43. Felix Stahlberg and Bill Byrne. 2019. On NMT search errors and model errors: Cat got your tongue? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3356– 3362, Hong… [find](https://scholar.google.com/scholar?q=Felix+Stahlberg+and+Bill+Byrne%2E+2019%2E+On+NMT+search+errors+and+model+errors%3A+Cat+got+your+tongue%3F+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th+) [our card](https://md.research.elamur.ai/free/en/papers/stahlberg-2019-nmt-search-errors-model-errors/)
44. Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. Context-aware neural machine translation learns anaphora resolution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1264–1274, Melbourne, Australia. Associati… [find](https://scholar.google.com/scholar?q=Elena+Voita%2C+Pavel+Serdyukov%2C+Rico+Sennrich%2C+and+Ivan+Titov%2E+2018%2E+Context%2Daware+neural+machine+translation+learns+anaphora+resolution%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for) [our card](https://md.research.elamur.ai/free/en/papers/voita-2018-context-aware-neural-machine-translation/)
45. Katsuhito Sudoh, Kosuke Takahashi, and Satoshi Nakamura. 2021. Is this translation error critical?: Classification-based human and automatic machine translation evaluation focusing on critical errors. In Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval), pages 46–55, Online. A… [find](https://scholar.google.com/scholar?q=Katsuhito+Sudoh%2C+Kosuke+Takahashi%2C+and+Satoshi+Nakamura%2E+2021%2E+Is+this+translation+error+critical%3F%3A+Classification%2Dbased+human+and+automatic+machine+translation+evaluation+focusing+on+critical+errors%2E)
46. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797–5808, Florence, I… [find](https://scholar.google.com/scholar?q=Elena+Voita%2C+David+Talbot%2C+Fedor+Moiseev%2C+Rico+Sennrich%2C+and+Ivan+Titov%2E+2019%2E+Analyzing+multi%2Dhead+self%2Dattention%3A+Specialized+heads+do+the+heavy+lifting%2C+the+rest+can+be+pruned%2E+In+Proceedings+of+th) [our card](https://md.research.elamur.ai/free/en/papers/voita-2019-analyzing-multi-head-self-attention/)
47. Kosuke Takahashi, Yoichi Ishibashi, Katsuhito Sudoh, and Satoshi Nakamura. 2021. Multilingual machine translation evaluation metrics fine-tuned on pseudo-negative examples for wmt 2021 metrics task. In Proceedings of the Sixth Conference on Machine Translation, pages 1049–1052, Online. Association f… [find](https://scholar.google.com/scholar?q=Kosuke+Takahashi%2C+Yoichi+Ishibashi%2C+Katsuhito+Sudoh%2C+and+Satoshi+Nakamura%2E+2021%2E+Multilingual+machine+translation+evaluation+metrics+fine%2Dtuned+on+pseudo%2Dnegative+examples+for+wmt+2021+metrics+task%2E+I)
48. Chaojun Wang and Rico Sennrich. 2020. On exposure bias, hallucination and domain shift in neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3544–3552, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chaojun+Wang+and+Rico+Sennrich%2E+2020%2E+On+exposure+bias%2C+hallucination+and+domain+shift+in+neural+machine+translation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Lin) [our card](https://md.research.elamur.ai/free/en/papers/wang-2020-exposure-bias-hallucination-domain-shift/)
49. Chau Tran, Shruti Bhosale, James Cross, Philipp Koehn, Sergey Edunov, and Angela Fan. 2021. Facebook AI’s WMT21 news translation task submission. In Proceedings of the Sixth Conference on Machine Translation, pages 205–215, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chau+Tran%2C+Shruti+Bhosale%2C+James+Cross%2C+Philipp+Koehn%2C+Sergey+Edunov%2C+and+Angela+Fan%2E+2021%2E+Facebook+AI%E2%80%99s+WMT21+news+translation+task+submission%2E+In+Proceedings+of+the+Sixth+Conference+on+Machine+Tran) [our card](https://md.research.elamur.ai/free/en/papers/tran-2021-facebook-ai-s-wmt21-news/)
50. Arata Ugawa, Akihiro Tamura, Takashi Ninomiya, Hiroya Takamura, and Manabu Okumura. 2018. Neural machine translation incorporating named entity. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3240–3250, Santa Fe, New Mexico, USA. Association for Computational… [find](https://scholar.google.com/scholar?q=Arata+Ugawa%2C+Akihiro+Tamura%2C+Takashi+Ninomiya%2C+Hiroya+Takamura%2C+and+Manabu+Okumura%2E+2018%2E+Neural+machine+translation+incorporating+named+entity%2E+In+Proceedings+of+the+27th+International+Conference+on+)
51. Shikhar Vashishth, Shyam Upadhyay, Gaurav Singh Tomar, and Manaal Faruqui. 2019. Attention interpretability across nlp tasks. [find](https://scholar.google.com/scholar?q=Shikhar+Vashishth%2C+Shyam+Upadhyay%2C+Gaurav+Singh+Tomar%2C+and+Manaal+Faruqui%2E+2019%2E+Attention+interpretability+across+nlp+tasks%2E)
52. Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11–20, Hong Kong, China. Association for… [find](https://scholar.google.com/scholar?q=Sarah+Wiegreffe+and+Yuval+Pinter%2E+2019%2E+Attention+is+not+not+explanation%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th+International+Joint+Conf) [our card](https://md.research.elamur.ai/free/en/papers/wiegreffe-2019-attention-is-not-not-explanation/)
53. Jianhao Yan, Fandong Meng, and Jie Zhou. 2022. Probing causes of hallucinations in neural machine translations. [find](https://scholar.google.com/scholar?q=Jianhao+Yan%2C+Fandong+Meng%2C+and+Jie+Zhou%2E+2022%2E+Probing+causes+of+hallucinations+in+neural+machine+translations%2E)
54. Chrysoula Zerva, Daan van Stigt, Ricardo Rei, Ana C Farinha, Pedro Ramos, José G. C. de Souza, Taisiya Glushkova, Miguel Vera, Fabio Kepler, and André F. T. Martins. 2021. [find](https://scholar.google.com/scholar?q=Chrysoula+Zerva%2C+Daan+van+Stigt%2C+Ricardo+Rei%2C+Ana+C+Farinha%2C+Pedro+Ramos%2C+Jos%C3%A9+G%2E+C%2E+de+Souza%2C+Taisiya+Glushkova%2C+Miguel+Vera%2C+Fabio+Kepler%2C+and+Andr%C3%A9+F%2E+T%2E+Martins%2E+2021%2E)
55. IST-unbabel 2021 submission for the quality estimation shared task. In Proceedings of the Sixth Conference on Machine Translation, pages 961–972, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=IST%2Dunbabel+2021+submission+for+the+quality+estimation+shared+task%2E+In+Proceedings+of+the+Sixth+Conference+on+Machine+Translation%2C+pages+961%E2%80%93972%2C+Online%2E+Association+for+Computational+Linguistics%2E)
56. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81+ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Advances+in+Neural+Information+Process)
57. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+In+International+Conference+on+Learning+Representations%2E)
58. David Vilar, Jia Xu, Luis Fernando D’Haro, and Hermann Ney. 2006. Error analysis of statistical machine translation output. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Genoa, Italy. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=David+Vilar%2C+Jia+Xu%2C+Luis+Fernando+D%E2%80%99Haro%2C+and+Hermann+Ney%2E+2006%2E+Error+analysis+of+statistical+machine+translation+output%2E+In+Proceedings+of+the+Fifth+International+Conference+on+Language+Resources+a) [our card](https://md.research.elamur.ai/free/en/papers/vilar-2006-error-analysis-statistical-machine-translation/)
59. Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzmán, Luke Zettlemoyer, and Marjan Ghazvininejad. 2021. Detecting hallucinated content in conditional neural sequence generation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1393–1404, Online. As… [find](https://scholar.google.com/scholar?q=Chunting+Zhou%2C+Graham+Neubig%2C+Jiatao+Gu%2C+Mona+Diab%2C+Francisco+Guzm%C3%A1n%2C+Luke+Zettlemoyer%2C+and+Marjan+Ghazvininejad%2E+2021%2E+Detecting+hallucinated+content+in+conditional+neural+sequence+generation%2E+In+Fin) [our card](https://md.research.elamur.ai/free/en/papers/zhou-2021-detecting-hallucinated-content-conditional-neural/)
60. Elena Voita, Rico Sennrich, and Ivan Titov. 2021. Analyzing the source and target contributions to predictions in neural machine translation. In Proceedings [find](https://scholar.google.com/scholar?q=Elena+Voita%2C+Rico+Sennrich%2C+and+Ivan+Titov%2E+2021%2E+Analyzing+the+source+and+target+contributions+to+predictions+in+neural+machine+translation%2E+In+Proceedings)