---
title: "Assessing the Role of Context in Chat — Agrawal 2024"
description: "Most automatic MT metrics were built and validated on well-formed text such as news."
authors: ["Sweta Agrawal", "Amin Farajian", "Patrick Fernandes", "Ricardo Rei", "André F. T. Martins"]
year: 2024
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/2024.tacl-1.69/"
doi: "10.1162/tacl_a_00700"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/agrawal-2024-assessing-role-context-chat-translation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/agrawal-2024-assessing-role-context-chat-translation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/) / Assessing the Role of Context in Chat Translation Evaluation: Is Context Helpful and Under What Conditions?

# Assessing the Role of Context in Chat Translation Evaluation: Is Context Helpful and Under What Conditions?

**Authors:** [Sweta Agrawal](https://md.research.elamur.ai/free/en/authors/sweta-agrawal/), [Amin Farajian](https://md.research.elamur.ai/free/en/authors/amin-farajian/), [Patrick Fernandes](https://md.research.elamur.ai/free/en/authors/patrick-fernandes/), [Ricardo Rei](https://md.research.elamur.ai/free/en/authors/ricardo-rei/), [André F. T. Martins](https://md.research.elamur.ai/free/en/authors/andre-f-t-martins/) · **Year:** 2024 · **Venue:** Transactions of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.tacl-1.69/) · [PDF](https://aclanthology.org/2024.tacl-1.69.pdf) · [DOI 10.1162/tacl\_a\_00700](https://doi.org/10.1162/tacl_a_00700) · [OpenAlex](https://openalex.org/W4403051469)

**Citations:** 1 citations (openalex-snapshot · 2026-10-14) · 7 in our library, of the 140673 papers whose bibliographies we read

## Summary
Most automatic MT metrics were built and validated on well-formed text such as news. It is unclear how well they judge machine-translated chats, which are short, informal and depend heavily on earlier turns. The authors first compare MQM error annotations for chat and news in WMT22 English–German. Errors in chat turn out to be rarer and of a different kind: fluency problems dominate, while news shows more accuracy errors. They then meta-evaluate lexical, embedding-based, neural and LLM-based metrics on MQM data from the WMT 2022 Chat Shared Task, which covers real customer-support dialogues. Correlation with human scores is measured at the turn level and the dialogue level, both on all segments and on imperfect ones only. Next, they feed conversational context from both speakers into COMET-style metrics. Finally, they introduce CONTEXT-MQM, a GPT-4 prompt that includes bilingual context. Context brings little benefit when a reference exists or when translating into English. It does help reference-free scoring of out-of-English translations, especially for short or ambiguous turns. This matters for live quality estimation in chat, where references are never available.

## Key points
- **Data:** The study uses MQM annotations from the WMT22 Chat Shared Task for EN–DE, EN–FR and EN–PT in both directions. Agents write in English and customers in the other language. Depending on the direction, 46.7–73.6% of turns contain no errors.
- **Chat vs. news:** In WMT22 EN–DE (7,120 chat vs. 4,800 news pairs), 57.8% of chat translations have perfect MQM scores, compared with 46.4% for news. This corresponds to roughly 21% fewer errors in chat.
- **Best metrics:** Reference-based COMET-22 correlates best with human judgments (Spearman). Reference-free metrics lag behind, most clearly in out-of-English directions.
- **Effect of context:** Context does not improve COMET-22. It does raise the correlation of COMET-20-QE for out-of-English evaluation, provided the context is correct and complete. For dialogue-level reference-free evaluation, SLIDE is the recommended option.
- **CONTEXT-MQM:** This GPT-4-based, reference-free metric beat COMET-22 on imperfect translations in a small preliminary study. Code is available at github.com/sweta20/chat-qe.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{agrawal2024assessing,
  title = {{Assessing the Role of Context in Chat Translation Evaluation: Is Context Helpful and Under What Conditions?}},
  author = {Agrawal, Sweta and Farajian, Amin and Fernandes, Patrick and Rei, Ricardo and Martins, André F. T.},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.1162/tacl\_a\_00700},
  url = {https://doi.org/10.1162/tacl_a_00700}
}
```

- [← previous Assessing the Cross-linguistic Utility of Abstract Meaning Representation](https://md.research.elamur.ai/free/en/papers/wein-2024-assessing-cross-linguistic-utility-abstract/)
- [next → At the Crossroad of Cuneiform and NLP: Challenges for Fine-grained Part-of-speech Tagging](https://md.research.elamur.ai/free/en/papers/coling-2024-smidt-crossroad-cuneiform-nlp-challenges-fine/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/)

## References

62 references, 35 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv: 2303.08774. [↗](https://arxiv.org/abs/2303.08774) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
2. Frederic Blain, Chrysoula Zerva, Ricardo Ribeiro, Nuno M. Guerreiro, Diptesh Kanojia, José G. C. de Souza, Beatriz Silva, Tânia Vaz, Yan Jingxuan, Fatemeh Azadi, Constantin Orasan, and André Martins. 2023. Findings of the WMT 2023 shared task on quality estimation. In Proceedings of the Eighth Conference on Machine Translation, pages 629–653, Singapore. Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.wmt-1.52 [↗](https://aclanthology.org/2023.wmt-1.52/) [our card](https://md.research.elamur.ai/free/en/papers/blain-2023-findings-wmt-2023-shared-task/)
3. Ondřej Bojar, Yvette Graham, and Amir Kamran. 2017. Results of the WMT17 metrics shared task. In Proceedings of the Second Conference on Machine Translation, pages 489–513, Copenhagen, Denmark. Association for Computational Linguistics. https://doi.org/10.18653/v1/W17-4755 [↗](https://doi.org/10.18653/v1/w17-4755) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2017-results-wmt17-metrics-shared-task/)
4. Ondřej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016. Results of the WMT16 metrics shared task. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 199–231, Berlin, Germany. Association for Computational Linguistics. https://doi.org/10 .18653/v1/W16-2302 [↗](https://aclanthology.org/W16-2302/) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2016-results-wmt16-metrics-shared-task/)
5. Boxing Chen and Colin Cherry. 2014. A systematic comparison of smoothing techniques for sentence-level BLEU. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 362–367, Baltimore, Maryland, USA. Association for Computational Linguistics. https://doi.org/10.3115/v1/W14-3346 [↗](https://aclanthology.org/W14-3346/) [our card](https://md.research.elamur.ai/free/en/papers/chen-2014-systematic-comparison-smoothing-techniques-sentence/)
6. Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021. InfoXLM: An information-theoretic framework for cross-lingual language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3576–3588, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.naacl-main.280 Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451. https://doi.org/10 .18653/v1/2020.acl-main.747 [↗](https://doi.org/10.18653/v1/2021.naacl-main.280) [our card](https://md.research.elamur.ai/free/en/papers/chi-2020-infoxlm-information-theoretic-framework-cross/)
7. Daniel Deutsch, George Foster, and Markus Freitag. 2023. Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12914–12929, Singapore. Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.emnlp-main.798 [↗](https://aclanthology.org/2023.emnlp-main.798/)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423 [↗](https://doi.org/10.18653/v1/n19-1423)
9. Ana C. Farinha, M. Amin Farajian, Marianna Buchicchio, Patrick Fernandes, José G. C. de Souza, Helena Moniz, and André F. T. Martins. 2022. Findings of the WMT 2022 shared task on chat translation. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 724–743, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ana+C%2E+Farinha%2C+M%2E+Amin+Farajian%2C+Marianna+Buchicchio%2C+Patrick+Fernandes%2C+Jose%CC%81+G%2E+C%2E+de+Souza%2C+Helena+Moniz%2C+and+Andre%CC%81+F%2E+T%2E+Martins%2E+2022%2E+Findings+of+the+WMT+2022+shared+task+on+chat+translation%2E+)
10. Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. 2023a. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation. In Proceedings of the Eighth Conference on Machine Translation, pages 1066–1083, Singapore. Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.wmt-1.100 [↗](https://doi.org/10.18653/v1/2023.wmt-1.100) [our card](https://md.research.elamur.ai/free/en/papers/fernandes-2023-devil-is-errors-leveraging-large/)
11. Patrick Fernandes, Kayo Yin, Emmy Liu, André Martins, and Graham Neubig. 2023b. When does translation require context? A data-driven, multilingual exploration. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 606–626, Toronto, Canada. Association for Computational Linguistics. https://doi.org/10.18653/v1/2023 .acl-long.36 [↗](https://doi.org/10.18653/v1/2023)
12. Patrick Fernandes, Kayo Yin, Graham Neubig, and André F. T. Martins. 2021. Measuring and increasing context usage in context-aware machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6467–6478, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.acl-long.505 [↗](https://doi.org/10.18653/v1/2021.acl-long.505)
13. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022. Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 46–68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Markus+Freitag%2C+Ricardo+Rei%2C+Nitika+Mathur%2C+Chi%2Dkiu+Lo%2C+Craig+Stewart%2C+Eleftherios+Avramidis%2C+Tom+Kocmi%2C+George+Foster%2C+Alon+Lavie%2C+and+Andre%CC%81+F%2E+T%2E+Martins%2E+2022%2E+Results+of+WMT22+metrics+shared+task) [our card](https://md.research.elamur.ai/free/en/papers/freitag-2022-results-wmt22-metrics-shared-task/)
14. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021. Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain. In Proceedings of the Sixth Conference on Machine Translation, pages 733–774, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Markus+Freitag%2C+Ricardo+Rei%2C+Nitika+Mathur%2C+Chi%2Dkiu+Lo%2C+Craig+Stewart%2C+George+Foster%2C+Alon+Lavie%2C+and+Ondr%CC%8Cej+Bojar%2E+2021%2E+Results+of+the+WMT21+metrics+shared+task%3A+Evaluating+metrics+with+expert%2Dbase) [our card](https://md.research.elamur.ai/free/en/papers/freitag-2021-results-wmt21-metrics-shared-task/)
15. Marina Fomicheva, Shuo Sun, Erick Fonseca, Chrysoula Zerva, Frédéric Blain, Vishrav Chaudhary, Francisco Guzmán, Nina Lopatina, Lucia Specia, and André F. T. Martins. 2022. MLQE-PE: A multilingual quality estimation and post-editing dataset. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4963–4974, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Marina+Fomicheva%2C+Shuo+Sun%2C+Erick+Fonseca%2C+Chrysoula+Zerva%2C+Fre%CC%81de%CC%81ric+Blain%2C+Vishrav+Chaudhary%2C+Francisco+Guzma%CC%81n%2C+Nina+Lopatina%2C+Lucia+Specia%2C+and+Andre%CC%81+F%2E+T%2E+Martins%2E+2022%2E+MLQE%2DPE%3A+A+multilingual) [our card](https://md.research.elamur.ai/free/en/papers/fomicheva-2022-mlqe-pe-multilingual-quality-estimation/)
16. Ge Gao, Bin Xu, David C. Hau, Zheng Yao, Dan Cosley, and Susan R. Fussell. 2015. Two is better than one: Improving multilingual collaboration by giving two machine translation outputs. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, CSCW ’15, pages 852–863, New York, NY, USA. Association for Computing Machinery. https://doi.org/10 .1145/2675133.2675197 [↗](https://doi.org/10)
17. Erick Fonseca, Lisa Yankovskaya, André F. T. Martins, Mark Fishel, and Christian Federmann. 2019. Findings of the WMT 2019 shared tasks on quality estimation. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 1–10, Florence, Italy. Association for Computational Linguistics. https://doi.org/10.18653/v1/W19-5401 [↗](https://doi.org/10.18653/v1/w19-5401) [our card](https://md.research.elamur.ai/free/en/papers/fonseca-2019-findings-wmt-2019-shared-tasks/)
18. Madalena Gonçalves, Marianna Buchicchio, Craig Stewart, Helena Moniz, and Alon Lavie. 2022. Agent and user-generated content and its impact on customer support MT. In Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, pages 201–210, Ghent, Belgium. European Association for Machine Translation. [find](https://scholar.google.com/scholar?q=Madalena+Gonc%CC%A7alves%2C+Marianna+Buchicchio%2C+Craig+Stewart%2C+Helena+Moniz%2C+and+Alon+Lavie%2E+2022%2E+Agent+and+user%2Dgenerated+content+and+its+impact+on+customer+support+MT%2E+In+Proceedings+of+the+23rd+Annual+C)
19. Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023. Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent. In Proceedings of the Eighth Conference on Machine Translation, pages 578–628, Singapore. Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.wmt-1.51 [↗](https://doi.org/10.18653/v1/2023.wmt-1.51) [our card](https://md.research.elamur.ai/free/en/papers/freitag-2023-results-wmt23-metrics-shared-task/)
20. Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. 2023. xcomet: Transparent machine translation evaluation through fine-grained error detection. arXiv preprint arXiv: 2310.10482. [↗](https://arxiv.org/abs/2310.10482) [our card](https://md.research.elamur.ai/free/en/papers/guerreiro-2024-xcomet-transparent-machine-translation-evaluation/)
21. Lifeng Han, Alan Smeaton, and Gareth Jones. 2021. Translation quality assessment: A brief survey on manual and automatic methods. In Proceedings for the First Workshop on Modelling Translation: Translatology in the Digital Age, pages 15–33, online. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2105.03311)
22. Sébastien Jean, Stanislas Lauly, Orhan Firat, and Kyunghyun Cho. 2017. Does neural machine translation benefit from larger context? ArXiv, abs/1704.05135. [↗](https://arxiv.org/abs/1704.05135)
23. Yuchen Jiang, Tianyu Liu, Shuming Ma, Dongdong Zhang, Jian Yang, Haoyang Huang, Rico Sennrich, Ryan Cotterell, Mrinmaya Sachan, and Ming Zhou. 2022. BlonDe: An automatic evaluation metric for document-level machine translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1550–1565, Seattle, United States. Association for Computational Linguistics. https://doi.org/10.18653/v1/2022 .naacl-main.111 [↗](https://doi.org/10.18653/v1/2022) [our card](https://md.research.elamur.ai/free/en/papers/jiang-2022-blonde-automatic-evaluation-metric-document/)
24. Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. 2023. MetricX-23: The Google submission to the WMT 2023 metrics shared task. In Proceedings of the Eighth Conference on Machine Translation, pages 756–767, Singapore. Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.wmt-1.63 [↗](https://aclanthology.org/2023.wmt-1.63/) [our card](https://md.research.elamur.ai/free/en/papers/juraska-2023-metricx-23-google-submission-wmt/)
25. Tom Kocmi and Christian Federmann. 2023a. GEMBA-MQM: Detecting translation quality error spans with GPT-4. In Proceedings of the Eighth Conference on Machine Translation, pages 768–775, Singapore. Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.wmt-1.64 [↗](https://doi.org/10.18653/v1/2023.wmt-1.64) [our card](https://md.research.elamur.ai/free/en/papers/kocmi-2023-gemba-mqm-detecting-translation-quality/)
26. Tom Kocmi and Christian Federmann. 2023b. Large language models are state-of-the-art evaluators of translation quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193–203, Tampere, Finland. European Association for Machine Translation. [↗](https://doi.org/10.48550/arxiv.2302.14520)
27. Julia Kreutzer, Shigehiko Schamoni, and Stefan Riezler. 2015. QUality estimation from ScraTCH (QUETCH): Deep learning for word-level translation quality estimation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 316–322, Lisbon, Portugal. Association for Computational Linguistics. https://doi.org/10.18653/v1/W15-3037 [↗](https://aclanthology.org/W15-3037/)
28. Yunmeng Li, Jun Suzuki, Makoto Morishita, Kaori Abe, Ryoko Tokuhisa, Ana Brassard, and Kentaro Inui. 2022. Chat translation error detection for assisting cross-lingual communications. In Proceedings of the 3rd Workshop on Evaluation and Comparison of NLP Systems, pages 88–95, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yunmeng+Li%2C+Jun+Suzuki%2C+Makoto+Morishita%2C+Kaori+Abe%2C+Ryoko+Tokuhisa%2C+Ana+Brassard%2C+and+Kentaro+Inui%2E+2022%2E+Chat+translation+error+detection+for+assisting+cross%2Dlingual+communications%2E+In+Proceedings+o)
29. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. https://doi.org/10.1162/tacl_a_00343 [↗](https://doi.org/10.1162/tacl_a_00343)
30. Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014. Multidimensional quality metrics (mqm): A framework for declaring and describing translation quality metrics. Tradu-màtica, (12):0455–463. https://doi.org/10.5565/rev/tradumatica.77 [↗](https://doi.org/10.5565/rev/tradumatica.77)
31. Qingyu Lu, Baopu Qiu, Liang Ding, Liping Xie, and Dacheng Tao. 2023. Error analysis prompting enables human-like translation evaluation in large language models: A case study on chatgpt. arXiv preprint arXiv:2303.13809. [↗](https://arxiv.org/abs/2303.13809) [our card](https://md.research.elamur.ai/free/en/papers/lu-2024-error-analysis-prompting-enables-human/)
32. Qingsong Ma, Ondřej Bojar, and Yvette Graham. 2018. Results of the WMT18 metrics shared task: Both characters and embeddings achieve good performance. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 671–688, Belgium, Brussels. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Qingsong+Ma%2C+Ondr%CC%8Cej+Bojar%2C+and+Yvette+Graham%2E+2018%2E+Results+of+the+WMT18+metrics+shared+task%3A+Both+characters+and+embeddings+achieve+good+performance%2E+In+Proceedings+of+the+Third+Conference+on+Machin) [our card](https://md.research.elamur.ai/free/en/papers/ma-2018-results-wmt18-metrics-shared-task/)
33. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62–90, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Qingsong+Ma%2C+Johnny+Wei%2C+Ondr%CC%8Cej+Bojar%2C+and+Yvette+Graham%2E+2019%2E+Results+of+the+WMT19+metrics+shared+task%3A+Segment%2Dlevel+and+strong+MT+systems+pose+big+challenges%2E+In+Proceedings+of+the+Fourth+Confere) [our card](https://md.research.elamur.ai/free/en/papers/ma-2019-results-wmt19-metrics-shared-task/)
34. Matouš Macháček and Ondřej Bojar. 2014. Results of the WMT14 metrics shared task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 293–301, Baltimore, Maryland, USA. Association for Computational Linguistics. https://doi.org/10.3115/v1/W14-3336 [↗](https://doi.org/10.3115/v1/w14-3336) [our card](https://md.research.elamur.ai/free/en/papers/machacek-2014-results-wmt14-metrics-shared-task/)
35. André F. T. Martins, Joao Graca, Paulo Dimas, Helena Moniz, and Graham Neubig. 2020. Project MAIA: Multilingual AI agent assistant. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 495–496, Lisboa, Portugal. European Association for Machine Translation. [find](https://scholar.google.com/scholar?q=Andre%CC%81+F%2E+T%2E+Martins%2C+Joao+Graca%2C+Paulo+Dimas%2C+Helena+Moniz%2C+and+Graham+Neubig%2E+2020%2E+Project+MAIA%3A+Multilingual+AI+agent+assistant%2E+In+Proceedings+of+the+22nd+Annual+Conference+of+the+European+Associ)
36. Sameen Maruf, André F. T. Martins, and Gholamreza Haffari. 2018. Contextual neural model for translating bilingual multi-speaker conversations. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 101–112, Brussels, Belgium. Association for Computational Linguistics. https://doi.org/10.18653/v1/W18 -6311 [↗](https://doi.org/10.18653/v1/w18)
37. Sameen Maruf, André F. T. Martins, and Gholamreza Haffari. 2019. Selective attention for context-aware neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3092–3102, Minneapolis, Minnesota. Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1313 [↗](https://aclanthology.org/N19-1313/)
38. Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020. Results of the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 688–725, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nitika+Mathur%2C+Johnny+Wei%2C+Markus+Freitag%2C+Qingsong+Ma%2C+and+Ondr%CC%8Cej+Bojar%2E+2020%2E+Results+of+the+WMT20+metrics+shared+task%2E+In+Proceedings+of+the+Fifth+Conference+on+Machine+Translation%2C+pages+688%E2%80%93725%2C) [our card](https://md.research.elamur.ai/free/en/papers/mathur-2020-results-wmt20-metrics-shared-task/)
39. Miguel Menezes, M. Amin Farajian, Helena Moniz, and João Varelas Graça. 2023. A context-aware annotation framework for customer support live chat machine translation. In Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track, pages 286–297, Macau SAR, China. Asia-Pacific Association for Machine Translation. [find](https://scholar.google.com/scholar?q=Miguel+Menezes%2C+M%2E+Amin+Farajian%2C+Helena+Moniz%2C+and+Joa%CC%83o+Varelas+Grac%CC%A7a%2E+2023%2E+A+context%2Daware+annotation+framework+for+customer+support+live+chat+machine+translation%2E+In+Proceedings+of+Machine+Trans)
40. Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. LLM evaluators recognize and favor their own generations. arXiv preprint arXiv:2404.13076. [↗](https://arxiv.org/abs/2404.13076)
41. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318. https://doi.org/10.3115/1073083.1073135 [↗](https://doi.org/10.3115/1073083.1073135) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
42. Maja Popović. 2015. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395. https://doi.org/10.18653/v1/W15 -3049 [↗](https://doi.org/10.18653/v1/w15) [our card](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/)
43. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium. Association for Computational Linguistics. https://doi.org/10.18653/v1/W18 -6319 [↗](https://doi.org/10.18653/v1/w18) [our card](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/)
44. Vikas Raunak, Tom Kocmi, and Matt Post. 2023. Evaluating metrics for document-context evaluation in machine translation. In Proceedings of the Eighth Conference on Machine Translation, pages 812–814, Singapore. Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.wmt-1.68 [↗](https://doi.org/10.18653/v1/2023.wmt-1.68)
45. Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C. Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022a. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Jose%CC%81+G%2E+C%2E+de+Souza%2C+Duarte+Alves%2C+Chrysoula+Zerva%2C+Ana+C%2E+Farinha%2C+Taisiya+Glushkova%2C+Alon+Lavie%2C+Luisa+Coheur%2C+and+Andre%CC%81+F%2E+T%2E+Martins%2E+2022a%2E+COMET%2D22%3A+Unbabel%2DIST+2022+submission+fo) [our card](https://md.research.elamur.ai/free/en/papers/rei-2022-comet-22-unbabel-ist-2022/)
46. Ricardo Rei, Craig Stewart, Ana C. Farinha, and Alon Lavie. 2020a. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Craig+Stewart%2C+Ana+C%2E+Farinha%2C+and+Alon+Lavie%2E+2020a%2E+COMET%3A+A+neural+framework+for+MT+evaluation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processin) [our card](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/)
47. Ricardo Rei, Craig Stewart, Ana C. Farinha, and Alon Lavie. 2020b. Unbabel’s participation in the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 911–920, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Craig+Stewart%2C+Ana+C%2E+Farinha%2C+and+Alon+Lavie%2E+2020b%2E+Unbabel%E2%80%99s+participation+in+the+WMT20+metrics+shared+task%2E+In+Proceedings+of+the+Fifth+Conference+on+Machine+Translation%2C+pages+911%E2%80%9392) [our card](https://md.research.elamur.ai/free/en/papers/rei-2020-unbabel-s-participation-wmt20-metrics/)
48. Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C. Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022b. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2209.06243) [our card](https://md.research.elamur.ai/free/en/papers/rei-2022-cometkiwi-ist-unbabel-2022-submission/)
49. Samantha Robertson and Mark Dı́az. 2022. Understanding and being understood: User strategies for identifying and recovering from mistranslations in machine translation-mediated chat. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 2223–2238. https://doi.org/10.1145/3531146.3534638 [↗](https://doi.org/10.1145/3531146.3534638)
50. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. Bleurt: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892. https://doi.org/10.18653/v1/2020 .acl-main.704 [↗](https://doi.org/10.18653/v1/2020)
51. Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pages 223–231, Cambridge, Massachusetts, USA. Association for Machine Translation in the Americas. [find](https://scholar.google.com/scholar?q=Matthew+Snover%2C+Bonnie+Dorr%2C+Rich+Schwartz%2C+Linnea+Micciulla%2C+and+John+Makhoul%2E+2006%2E+A+study+of+translation+edit+rate+with+targeted+human+annotation%2E+In+Proceedings+of+the+7th+Conference+of+the+Assoc) [our card](https://md.research.elamur.ai/free/en/papers/snover-2006-study-translation-edit-rate-targeted/)
52. Lucia Specia, Frédéric Blain, Marina Fomicheva, Chrysoula Zerva, Zhenhao Li, Vishrav Chaudhary, and André F. T. Martins. 2021. Findings of the WMT 2021 shared task on quality estimation. In Proceedings of the Sixth Conference on Machine Translation, pages 684–725, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Lucia+Specia%2C+Fre%CC%81de%CC%81ric+Blain%2C+Marina+Fomicheva%2C+Chrysoula+Zerva%2C+Zhenhao+Li%2C+Vishrav+Chaudhary%2C+and+Andre%CC%81+F%2E+T%2E+Martins%2E+2021%2E+Findings+of+the+WMT+2021+shared+task+on+quality+estimation%2E+In+Proceed) [our card](https://md.research.elamur.ai/free/en/papers/specia-2021-findings-wmt-2021-shared-task/)
53. Miloš Stanojević, Amir Kamran, Philipp Koehn, and Ondřej Bojar. 2015. Results of the WMT15 metrics shared task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 256–273, Lisbon, Portugal. Association for Computational Linguistics. https://doi.org/10.18653/v1/W15 -3031 [↗](https://doi.org/10.18653/v1/w15)
54. Jörg Tiedemann and Yves Scherrer. 2017. Neural machine translation with extended context. In DiscoMT@EMNLP. https://doi.org/10.18653/v1/W17-4811 [↗](https://doi.org/10.18653/v1/w17-4811) [our card](https://md.research.elamur.ai/free/en/papers/tiedemann-2017-neural-machine-translation-extended-context/)
55. Giorgos Vernikos, Brian Thompson, Prashant Mathur, and Marcello Federico. 2022. Embarrassingly easy document-level MT metrics: How to convert any pretrained metric into a document-level metric. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 118–128, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Giorgos+Vernikos%2C+Brian+Thompson%2C+Prashant+Mathur%2C+and+Marcello+Federico%2E+2022%2E+Embarrassingly+easy+document%2Dlevel+MT+metrics%3A+How+to+convert+any+pretrained+metric+into+a+document%2Dlevel+metric%2E+In+Pro) [our card](https://md.research.elamur.ai/free/en/papers/vernikos-2022-embarrassingly-easy-document-level-mt/)
56. Elena Voita, Rico Sennrich, and Ivan Titov. 2019. When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1198–1212, Florence, Italy. Association for Computational Linguistics. https://doi.org/10.18653/v1/P19-1116 [↗](https://doi.org/10.18653/v1/p19-1116) [our card](https://md.research.elamur.ai/free/en/papers/voita-2019-when-good-translation-is-wrong/)
57. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2021.naacl-main.41)
58. Naomi Yamashita, Rieko Inaba, Hideaki Kuzuoka, and Toru Ishida. 2009. Difficulties in establishing common ground in multiparty groups using machine translation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 679–688. https://doi.org/10.1145/1518701 .1518807 [↗](https://doi.org/10.1145/1518701)
59. Jerrold H. Zar. 2005. Spearman rank correlation. Encyclopedia of Biostatistics, 7. https://doi.org/10.1002/0470011815.b2a15150 [↗](https://doi.org/10.1002/0470011815.b2a15150)
60. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with BERT. In International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.1904.09675)
61. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 563–578, Hong Kong, China. Association for Computational Linguistics. https://doi.org/10 .18653/v1/D19-1053 [↗](https://aclanthology.org/D19-1053/) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2019-moverscore-text-generation-evaluating-contextualized/)
62. Vilém Zouhar, Shuoyang Ding, Anna Currey, Tatyana Badeka, Jenyuan Wang, and Brian Thompson. 2024. Fine-tuned machine translation metrics struggle in unseen domains. [↗](https://doi.org/10.48550/arxiv.2402.18747) [our card](https://md.research.elamur.ai/free/en/papers/zouhar-2024-fine-tuned-machine-translation-metrics/)