---
title: "Hindsight Quality Prediction Experiments in — Marmonier 2026"
description: "The authors examine whether established ways of anticipating MT quality still work now that LLMs are part of translation pipelines."
authors: ["Malik Marmonier", "Benoît Sagot", "Rachel Bawden"]
year: 2026
venue: "Proceedings of the Fifteenth Language Resources and Evaluation Conference"
source: "https://aclanthology.org/2026.lrec-1.688/"
doi: "10.63317/24puen8nstzh"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/marmonier-2026-hindsight-quality-prediction-experiments-multi/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/marmonier-2026-hindsight-quality-prediction-experiments-multi/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2026](https://md.research.elamur.ai/free/papers/2026/page/3/) / Hindsight Quality Prediction Experiments in Multi-Candidate Human-Post-Edited Machine Translation

# Hindsight Quality Prediction Experiments in Multi-Candidate Human-Post-Edited Machine Translation

**Authors:** Malik Marmonier, [Benoît Sagot](https://md.research.elamur.ai/free/authors/benoit-sagot/), [Rachel Bawden](https://md.research.elamur.ai/free/authors/rachel-bawden/) · **Year:** 2026 · **Venue:** Proceedings of the Fifteenth Language Resources and Evaluation Conference · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.lrec-1.688/) · [PDF](https://aclanthology.org/2026.lrec-1.688.pdf) · [DOI 10.63317/24puen8nstzh](https://doi.org/10.63317/24puen8nstzh) · [OpenAlex](https://openalex.org/W7160084174)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14)

## Summary
The authors examine whether established ways of anticipating MT quality still work now that LLMs are part of translation pipelines. They consider two kinds of prediction: estimating source-segment difficulty before translation, and reference-free quality estimation (QE) of candidate outputs. The study reuses data from a real post-editing project that built the OLDI Seed Corpus's French portion. Each English Wikipedia segment has nine French hypotheses, four from sentence-level NMT systems and five from prompted LLMs (Llama-4-Scout variants and DeepSeek-R1). All hypotheses are scored against one final human post-edited reference using TER (a post-editing effort proxy) and COMET (a human judgment proxy). Predictors are rated by their Kendall's τ with these scores.

The usefulness of difficulty features depends strongly on the target metric: length and neural predictors track COMET but not TER. QE scores align better with NMT outputs than with LLM outputs. Document-level LLMs degrade measurably for later segments, but the effect is tiny. The overall message is that quality-prediction tools calibrated on older NMT need rechecking for LLM-based translation.

## Key points
- **Dataset:** 6,193 English source segments from 203 core Wikipedia articles. The analysis uses 6,028 segments after dropping 165 that DeepSeek-R1 refused to translate. The data and code ("Hindex", GitHub) are released under CC-BY SA 4.0.
- **Systems compared:**
  - NMT: OPUS-MT, NLLB-3.3B, NLLB-600M-Distilled, MADLAD-400-3B.
  - LLMs: Llama 4 Scout (109B) in segment-level and document-level settings, the latter with and without guidelines and with Wikipedia context, plus DeepSeek-R1 (671B).
  - DeepSeek-R1 scored clearly best (TER 12.44, COMET 0.9219). OPUS-MT and distilled NLLB scored lowest.
- **Source-side predictors:** twelve features in total.
  - Nine readability and linguistic measures: Dale-Chall, SMOG, Flesch, Coleman-Liau, FK grade, length, word rarity, polysemy and tree height.
  - Neural predictors: MT surprisal and Sentinel DA/MQM scores.
- **Candidate-side QE:** COMET_QE and MetricX_QE correlate more strongly with gold quality for NMT outputs than for LLM outputs. Against ChrF, for example, average COMET_QE τ is about 0.22 for NMT versus 0.16 for LLMs.
- **Positional bias:** the bias in doc2doc LLMs is statistically significant but very weak (e.g., τ ≈ −0.04 for DeepSeek-R1 with ChrF). Supplementary ChrF results mirror the TER patterns.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{marmonier2026hindsight,
  title = {{Hindsight Quality Prediction Experiments in Multi-Candidate Human-Post-Edited Machine Translation}},
  author = {Marmonier, Malik and Sagot, Benoît and Bawden, Rachel},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/24puen8nstzh},
  url = {https://doi.org/10.63317/24puen8nstzh}
}
```

- [← previous Hebrew Diacritics Restoration using Visual Representation](https://md.research.elamur.ai/free/papers/aclbulk-2026-elboher-hebrew-diacritics-restoration-visual-representation/)
- [next → HistoriQA-ThirdRepublic: Multi-Hop Question Answering Corpus for Historical Research, Parliamentary Debates from the French Third Republic (1870-1940)](https://md.research.elamur.ai/free/papers/pellet-2026-historiqa-thirdrepublic-multi-hop-question/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/papers/2026/page/3/)

## References

41 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015), San Diego, CA, USA. Oral presentation. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+In+Proceedings+of+the+3rd+International+Conference+on+Learning+Represen) [our card](https://md.research.elamur.ai/free/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
2. Steven Bird and Edward Loper. 2004. NLTK: The natural language toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages 214–217, Barcelona, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Steven+Bird+and+Edward+Loper%2E+2004%2E+NLTK%3A+The+natural+language+toolkit%2E+In+Proceedings+of+the+ACL+Interactive+Poster+and+Demonstration+Sessions%2C+pages+214%E2%80%93217%2C+Barcelona%2C+Spain%2E+Association+for+Comput)
3. John Blatz, Erin Fitzgerald, George Foster, Simona Gandrabur, Cyril Goutte, Alex Kulesza, Alberto Sanchis, and Nicola Ueffing. 2004. Confidence estimation for machine translation. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pages 315–321, Geneva, Sw… [find](https://scholar.google.com/scholar?q=John+Blatz%2C+Erin+Fitzgerald%2C+George+Foster%2C+Simona+Gandrabur%2C+Cyril+Goutte%2C+Alex+Kulesza%2C+Alberto+Sanchis%2C+and+Nicola+Ueffing%2E+2004%2E+Confidence+estimation+for+machine+translation%2E+In+COLING+2004%3A+Proc)
4. Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2012. Findings of the 2012 Workshop on Statistical Machine Translation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 10–51, Montréal, Canada. Association for Computation… [find](https://scholar.google.com/scholar?q=Chris+Callison%2DBurch%2C+Philipp+Koehn%2C+Christof+Monz%2C+Matt+Post%2C+Radu+Soricut%2C+and+Lucia+Specia%2E+2012%2E+Findings+of+the+2012+Workshop+on+Statistical+Machine+Translation%2E+In+Proceedings+of+the+Seventh+Wor) [our card](https://md.research.elamur.ai/free/papers/burch-2012-findings-2012-workshop-statistical-machine/)
5. Aaron Chatterji, Thomas Cunningham, David J. Deming, Zoë Hitzig, Christopher Ong, Carl Yan Shan, and Kevin Wadman. 2025. How People Use ChatGPT. Working Paper 34255, National Bureau of Economic Research. [find](https://scholar.google.com/scholar?q=Aaron+Chatterji%2C+Thomas+Cunningham%2C+David+J%2E+Deming%2C+Zo%C3%AB+Hitzig%2C+Christopher+Ong%2C+Carl+Yan+Shan%2C+and+Kevin+Wadman%2E+2025%2E+How+People+Use+ChatGPT%2E+Working+Paper+34255%2C+National+Bureau+of+Economic+Resear)
6. Meri Coleman and Ta Lin Liau. 1975. A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60(2):283–284. [find](https://scholar.google.com/scholar?q=Meri+Coleman+and+Ta+Lin+Liau%2E+1975%2E+A+computer+readability+formula+designed+for+machine+scoring%2E+Journal+of+Applied+Psychology%2C+60%282%29%3A283%E2%80%93284%2E)
7. David Dale, Laurie V. Burchell, Jean Maillard, Idris Abdulmumin, Antonios Anastasopoulos, Isaac Caswell, and Philipp Koehn. 2025. Findings of the WMT 2025 Shared Task of the Open Language Data Initiative. In Proceedings of the Tenth Conference on Machine Translation (WMT 2025), pages 495–502, Suzhou… [find](https://scholar.google.com/scholar?q=David+Dale%2C+Laurie+V%2E+Burchell%2C+Jean+Maillard%2C+Idris+Abdulmumin%2C+Antonios+Anastasopoulos%2C+Isaac+Caswell%2C+and+Philipp+Koehn%2E+2025%2E+Findings+of+the+WMT+2025+Shared+Task+of+the+Open+Language+Data+Initiat)
8. Edgar Dale and Jeanne S. Chall. 1948. A formula for predicting readability: Instructions. Educational Research Bulletin, pages 37–54. [find](https://scholar.google.com/scholar?q=Edgar+Dale+and+Jeanne+S%2E+Chall%2E+1948%2E+A+formula+for+predicting+readability%3A+Instructions%2E+Educational+Research+Bulletin%2C+pages+37%E2%80%9354%2E)
9. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chengg… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Daya+Guo%2C+Dejian+Yang%2C+Haowei+Zhang%2C+Junxiao+Song%2C+Ruoyu+Zhang%2C+Runxin+Xu%2C+Qihao+Zhu%2C+Shirong+Ma%2C+Peiyi+Wang%2C+Xiao+Bi%2C+Xiaokang+Zhang%2C+Xingkai+Yu%2C+Yu+Wu%2C+Z%2E+F%2E+Wu%2C+Zhibin+Gou%2C+Zhihong+Sha) [our card](https://md.research.elamur.ai/free/papers/deepseek-2025-ai-deepseek-r1-incentivizing-reasoning-capability/)
10. Francesco Fernicola, Silvia Bernardini, Federico Garcea, Adriano Ferraresi, and Alberto Barrón-Cedeño. 2023. Return to the source: Assessing machine translation suitability. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 79– 89, Tampere, Finla… [find](https://scholar.google.com/scholar?q=Francesco+Fernicola%2C+Silvia+Bernardini%2C+Federico+Garcea%2C+Adriano+Ferraresi%2C+and+Alberto+Barr%C3%B3n%2DCede%C3%B1o%2E+2023%2E+Return+to+the+source%3A+Assessing+machine+translation+suitability%2E+In+Proceedings+of+the+24th)
11. Rudolph Flesch. 1948. A new readability yardstick. Journal of Applied Psychology, 32(3):221–233. [find](https://scholar.google.com/scholar?q=Rudolph+Flesch%2E+1948%2E+A+new+readability+yardstick%2E+Journal+of+Applied+Psychology%2C+32%283%29%3A221%E2%80%93233%2E)
12. Simona Gandrabur and George Foster. 2003. Confidence estimation for translation prediction. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 95–102. [find](https://scholar.google.com/scholar?q=Simona+Gandrabur+and+George+Foster%2E+2003%2E+Confidence+estimation+for+translation+prediction%2E+In+Proceedings+of+the+Seventh+Conference+on+Natural+Language+Learning+at+HLT%2DNAACL+2003%2C+pages+95%E2%80%93102%2E)
13. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous Measurement Scales in Human Evaluation of Machine Translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. Association for Computational… [find](https://scholar.google.com/scholar?q=Yvette+Graham%2C+Timothy+Baldwin%2C+Alistair+Moffat%2C+and+Justin+Zobel%2E+2013%2E+Continuous+Measurement+Scales+in+Human+Evaluation+of+Machine+Translation%2E+In+Proceedings+of+the+7th+Linguistic+Annotation+Works)
14. Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. 2024. MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task. In Proceedings of the Ninth Conference on Machine Translation, pages 492–504, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Juraj+Juraska%2C+Daniel+Deutsch%2C+Mara+Finkelstein%2C+and+Markus+Freitag%2E+2024%2E+MetricX%2D24%3A+The+Google+Submission+to+the+WMT+2024+Metrics+Shared+Task%2E+In+Proceedings+of+the+Ninth+Conference+on+Machine+Tran) [our card](https://md.research.elamur.ai/free/papers/juraska-2024-metricx-24-google-submission-wmt/)
15. J. Peter Kincaid, Robert P. Fishburne Jr, Richard L. Rogers, and Brad S. Chissom. 1975. Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel. Technical report, Naval Technical Training Command Millington TN Resear… [find](https://scholar.google.com/scholar?q=J%2E+Peter+Kincaid%2C+Robert+P%2E+Fishburne+Jr%2C+Richard+L%2E+Rogers%2C+and+Brad+S%2E+Chissom%2E+1975%2E+Derivation+of+New+Readability+Formulas+%28Automated+Readability+Index%2C+Fog+Count+and+Flesch+Reading+Ease+Formula%29+)
16. Zheng Wei Lim, Ekaterina Vylomova, Charles Kemp, and Trevor Cohn. 2024. Predicting human translation difficulty with neural machine translation. Transactions of the Association for Computational Linguistics, 12:1479–1496. [find](https://scholar.google.com/scholar?q=Zheng+Wei+Lim%2C+Ekaterina+Vylomova%2C+Charles+Kemp%2C+and+Trevor+Cohn%2E+2024%2E+Predicting+human+translation+difficulty+with+neural+machine+translation%2E+Transactions+of+the+Association+for+Computational+Lingu)
17. Arle Richard Lommel, Aljoscha Burchardt, and Hans Uszkoreit. 2013. Multidimensional quality metrics: a flexible system for assessing translation quality. In Proceedings of Translating and the Computer 35, London, UK. Aslib. [find](https://scholar.google.com/scholar?q=Arle+Richard+Lommel%2C+Aljoscha+Burchardt%2C+and+Hans+Uszkoreit%2E+2013%2E+Multidimensional+quality+metrics%3A+a+flexible+system+for+assessing+translation+quality%2E+In+Proceedings+of+Translating+and+the+Computer) [our card](https://md.research.elamur.ai/free/papers/lommel-2013-multidimensional-quality-metrics-flexible-system/)
18. Jean Maillard, Cynthia Gao, Elahe Kalbassi, Kaushik Ram Sadagopan, Vedanuj Goswami, Philipp Koehn, Angela Fan, and Francisco Guzman. 2023. Small Data, Big Impact: Leveraging Minimal Data for Effective Machine Translation. In Proceedings of the 61st Annual Meeting of the Association for Computational… [find](https://scholar.google.com/scholar?q=Jean+Maillard%2C+Cynthia+Gao%2C+Elahe+Kalbassi%2C+Kaushik+Ram+Sadagopan%2C+Vedanuj+Goswami%2C+Philipp+Koehn%2C+Angela+Fan%2C+and+Francisco+Guzman%2E+2023%2E+Small+Data%2C+Big+Impact%3A+Leveraging+Minimal+Data+for+Effective)
19. Sameen Maruf, Fahimeh Saleh, and Gholamreza Haffari. 2021. A survey on document-level neural machine translation: Methods and evaluation. ACM Computing Surveys, 54(2). [find](https://scholar.google.com/scholar?q=Sameen+Maruf%2C+Fahimeh+Saleh%2C+and+Gholamreza+Haffari%2E+2021%2E+A+survey+on+document%2Dlevel+neural+machine+translation%3A+Methods+and+evaluation%2E+ACM+Computing+Surveys%2C+54%282%29%2E)
20. G. Harry McLaughlin. 1969. Smog grading-a new readability formula. Journal of Reading, 12(8):639–646. [find](https://scholar.google.com/scholar?q=G%2E+Harry+McLaughlin%2E+1969%2E+Smog+grading%2Da+new+readability+formula%2E+Journal+of+Reading%2C+12%288%29%3A639%E2%80%93646%2E)
21. Meta AI. 2025. The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation. [find](https://scholar.google.com/scholar?q=Meta+AI%2E+2025%2E+The+Llama+4+herd%3A+The+beginning+of+a+new+era+of+natively+multimodal+AI+innovation%2E)
22. George A. Miller. 1994. WordNet: A lexical database for English. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994. [find](https://scholar.google.com/scholar?q=George+A%2E+Miller%2E+1994%2E+WordNet%3A+A+lexical+database+for+English%2E+In+Human+Language+Technology%3A+Proceedings+of+a+Workshop+held+at+Plainsboro%2C+New+Jersey%2C+March+8%2D11%2C+1994%2E) [our card](https://md.research.elamur.ai/free/papers/lex-1994-miller-wordnet/)
23. Abhijit Mishra, Pushpak Bhattacharyya, and Michael Carl. 2013. Automatically predicting sentence translation difficulty. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 346–351, Sofia, Bulgaria. Association for Computational… [find](https://scholar.google.com/scholar?q=Abhijit+Mishra%2C+Pushpak+Bhattacharyya%2C+and+Michael+Carl%2E+2013%2E+Automatically+predicting+sentence+translation+difficulty%2E+In+Proceedings+of+the+51st+Annual+Meeting+of+the+Association+for+Computational+)
24. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hof… [find](https://scholar.google.com/scholar?q=NLLB+Team%2C+Marta+R%2E+Costa%2Djuss%C3%A0%2C+James+Cross%2C+Onur+%C3%87elebi%2C+Maha+Elbayad%2C+Kenneth+Heafield%2C+Kevin+Heffernan%2C+Elahe+Kalbassi%2C+Janice+Lam%2C+Daniel+Licht%2C+Jean+Maillard%2C+Anna+Sun%2C+Skyler+Wang%2C+Guillaume+We)
25. Joseph Olive. 2005. Global autonomous language exploitation (GALE). DARPA/IPTO Proposer Information Pamphlet. [find](https://scholar.google.com/scholar?q=Joseph+Olive%2E+2005%2E+Global+autonomous+language+exploitation+%28GALE%29%2E+DARPA%2FIPTO+Proposer+Information+Pamphlet%2E)
26. Ziqian Peng, Rachel Bawden, and François Yvon. 2024. À propos des difficultés de traduire automatiquement de longs documents. In 35èmes Journées d’Études sur la Parole (JEP 2024), volume 1 : articles longs et prises de position, pages 2–21, Toulouse, France. ATALA & AFPC. [find](https://scholar.google.com/scholar?q=Ziqian+Peng%2C+Rachel+Bawden%2C+and+Fran%C3%A7ois+Yvon%2E+2024%2E+%C3%80+propos+des+difficult%C3%A9s+de+traduire+automatiquement+de+longs+documents%2E+In+35%C3%A8mes+Journ%C3%A9es+d%E2%80%99%C3%89tudes+sur+la+Parole+%28JEP+2024%29%2C+volume+1+%3A+articles+)
27. Ziqian Peng, Rachel Bawden, and François Yvon. 2025. Investigating length issues in document-level machine translation. In Proceedings of Machine Translation Summit XX: Volume 1, pages 4–23, Geneva, Switzerland. European Association for Machine Translation. [find](https://scholar.google.com/scholar?q=Ziqian+Peng%2C+Rachel+Bawden%2C+and+Fran%C3%A7ois+Yvon%2E+2025%2E+Investigating+length+issues+in+document%2Dlevel+machine+translation%2E+In+Proceedings+of+Machine+Translation+Summit+XX%3A+Volume+1%2C+pages+4%E2%80%9323%2C+Geneva%2C+S)
28. Maja Popović. 2015. chrF: character n-gram Fscore for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Maja+Popovi%C4%87%2E+2015%2E+chrF%3A+character+n%2Dgram+Fscore+for+automatic+MT+evaluation%2E+In+Proceedings+of+the+Tenth+Workshop+on+Statistical+Machine+Translation%2C+pages+392%E2%80%93395%2C+Lisbon%2C+Portugal%2E+Association+for)
29. Lorenzo Proietti, Stefano Perrella, Vilém Zouhar, Roberto Navigli, and Tom Kocmi. 2025. Estimating machine translation difficulty. [find](https://scholar.google.com/scholar?q=Lorenzo+Proietti%2C+Stefano+Perrella%2C+Vil%C3%A9m+Zouhar%2C+Roberto+Navigli%2C+and+Tom+Kocmi%2E+2025%2E+Estimating+machine+translation+difficulty%2E)
30. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Craig+Stewart%2C+Ana+C+Farinha%2C+and+Alon+Lavie%2E+2020%2E+COMET%3A+A+neural+framework+for+MT+evaluation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing+) [our card](https://md.research.elamur.ai/free/papers/rei-2020-comet-neural-framework-mt-evaluation/)
31. Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Marcos+Treviso%2C+Nuno+M%2E+Guerreiro%2C+Chrysoula+Zerva%2C+Ana+C+Farinha%2C+Christine+Maroti%2C+Jos%C3%A9+G%2E+C%2E+de+Souza%2C+Taisiya+Glushkova%2C+Duarte+Alves%2C+Luisa+Coheur%2C+Alon+Lavie%2C+and+Andr%C3%A9+F%2E+T%2E+Martin)
32. CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=CometKiwi%3A+IST%2DUnbabel+2022+Submission+for+the+Quality+Estimation+Shared+Task%2E+In+Proceedings+of+the+Seventh+Conference+on+Machine+Translation+%28WMT%29%2C+pages+634%E2%80%93645%2C+Abu+Dhabi%2C+United+Arab+Emirates+%28Hy) [our card](https://md.research.elamur.ai/free/papers/rei-2022-cometkiwi-ist-unbabel-2022-submission/)
33. Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pages 223–231, Cambridge, Massachu… [find](https://scholar.google.com/scholar?q=Matthew+Snover%2C+Bonnie+Dorr%2C+Rich+Schwartz%2C+Linnea+Micciulla%2C+and+John+Makhoul%2E+2006%2E+A+Study+of+Translation+Edit+Rate+with+Targeted+Human+Annotation%2E+In+Proceedings+of+the+7th+Conference+of+the+Assoc) [our card](https://md.research.elamur.ai/free/papers/snover-2006-study-translation-edit-rate-targeted/)
34. Robyn Speer. 2022. rspeer/wordfreq: v3.0. Zenodo, Version v3.0.2. [find](https://scholar.google.com/scholar?q=Robyn+Speer%2E+2022%2E+rspeer%2Fwordfreq%3A+v3%2E0%2E+Zenodo%2C+Version+v3%2E0%2E2%2E)
35. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, NEURIPS’14, page 3104–3112, Cambridge, MA, USA. MIT Press. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V%2E+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Proceedings+of+the+28th+International+Conference+on+Neural+Information+Processing+Systems+%2D+)
36. Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT – Building open translation services for the World. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 479–480, Lisboa, Portugal. European Association for Machine Translation. [find](https://scholar.google.com/scholar?q=J%C3%B6rg+Tiedemann+and+Santhosh+Thottingal%2E+2020%2E+OPUS%2DMT+%E2%80%93+Building+open+translation+services+for+the+World%2E+In+Proceedings+of+the+22nd+Annual+Conference+of+the+European+Association+for+Machine+Translati)
37. Nicola Ueffing, Klaus Macherey, and Hermann Ney. 2003. Confidence measures for statistical machine translation. In Proceedings of Machine Translation Summit IX: Papers, New Orleans, USA. [find](https://scholar.google.com/scholar?q=Nicola+Ueffing%2C+Klaus+Macherey%2C+and+Hermann+Ney%2E+2003%2E+Confidence+measures+for+statistical+machine+translation%2E+In+Proceedings+of+Machine+Translation+Summit+IX%3A+Papers%2C+New+Orleans%2C+USA%2E)
38. Zhang Zhuocheng, Shuhao Gu, Min Zhang, and Yang Feng. 2023. Addressing the length bias challenge in document-level neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11545–11556, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zhang+Zhuocheng%2C+Shuhao+Gu%2C+Min+Zhang%2C+and+Yang+Feng%2E+2023%2E+Addressing+the+length+bias+challenge+in+document%2Dlevel+neural+machine+translation%2E+In+Findings+of+the+Association+for+Computational+Linguist)
39. Vilém Zouhar, Pinzhen Chen, Tsz Kin Lam, Nikita Moghe, and Barry Haddow. 2024. Pitfalls and outlooks in using COMET. In Proceedings of the Ninth Conference on Machine Translation, pages 1272–1288, Miami, Florida, USA. Association for Computational Linguistics. 10. [find](https://scholar.google.com/scholar?q=Vil%C3%A9m+Zouhar%2C+Pinzhen+Chen%2C+Tsz+Kin+Lam%2C+Nikita+Moghe%2C+and+Barry+Haddow%2E+2024%2E+Pitfalls+and+outlooks+in+using+COMET%2E+In+Proceedings+of+the+Ninth+Conference+on+Machine+Translation%2C+pages+1272%E2%80%931288%2C+Mia)
40. Language Resource References Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Christopher A. Choquette-Choo, Katherine Lee, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. 2023. MADLAD-400: A Multilingual And Document-Level Large Audited Dataset. [find](https://scholar.google.com/scholar?q=Language+Resource+References+Sneha+Kudugunta%2C+Isaac+Caswell%2C+Biao+Zhang%2C+Xavier+Garcia%2C+Christopher+A%2E+Choquette%2DChoo%2C+Katherine+Lee%2C+Derrick+Xin%2C+Aditya+Kusupati%2C+Romi+Stella%2C+Ankur+Bapna%2C+and+Orhan+) [our card](https://md.research.elamur.ai/free/papers/kudugunta-2023-madlad-400-multilingual-document-level/)
41. Malik Marmonier, Benoît Sagot, and Rachel Bawden. 2025. A French Version of the OLDI Seed Corpus. In Proceedings of the Tenth Conference on Machine Translation (WMT 2025), pages 1048–1060, Suzhou, China. Association for Computational Linguistics. 11. Appendix: Readability and Linguistic Complexity M… [find](https://scholar.google.com/scholar?q=Malik+Marmonier%2C+Beno%C3%AEt+Sagot%2C+and+Rachel+Bawden%2E+2025%2E+A+French+Version+of+the+OLDI+Seed+Corpus%2E+In+Proceedings+of+the+Tenth+Conference+on+Machine+Translation+%28WMT+2025%29%2C+pages+1048%E2%80%931060%2C+Suzhou%2C+Chi)