---
title: "CometKiwi: IST-Unbabel 2022 Submission for the — Rei 2022"
description: "This paper describes the joint IST and Unbabel entry to the WMT 2022 Quality Estimation shared task."
authors: ["Ricardo Rei", "Marcos Treviso", "Nuno M. Guerreiro", "Chrysoula Zerva", "Ana C Farinha", "Christine Maroti", "José G. C. de Souza", "Taisiya Glushkova", "Duarte Alves", "Luisa Coheur", "Alon Lavie", "André F. T. Martins"]
year: 2022
venue: "Proceedings of the Seventh Conference on Machine Translation (WMT)"
source: "https://aclanthology.org/2022.wmt-1.60/"
doi: "10.18653/v1/2022.wmt-1.60"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rei-2022-cometkiwi-ist-unbabel-2022-submission/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rei-2022-cometkiwi-ist-unbabel-2022-submission/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/) / CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task

# CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task

**Authors:** [Ricardo Rei](https://md.research.elamur.ai/free/en/authors/ricardo-rei/), [Marcos Treviso](https://md.research.elamur.ai/free/en/authors/marcos-treviso/), [Nuno M. Guerreiro](https://md.research.elamur.ai/free/en/authors/nuno-m-guerreiro/), [Chrysoula Zerva](https://md.research.elamur.ai/free/en/authors/chrysoula-zerva/), [Ana C Farinha](https://md.research.elamur.ai/free/en/authors/ana-c-farinha/), Christine Maroti, [José G. C. de Souza](https://md.research.elamur.ai/free/en/authors/jose-g-c-de-souza/), [Taisiya Glushkova](https://md.research.elamur.ai/free/en/authors/taisiya-glushkova/), [Duarte Alves](https://md.research.elamur.ai/free/en/authors/duarte-alves/), [Luisa Coheur](https://md.research.elamur.ai/free/en/authors/luisa-coheur/), [Alon Lavie](https://md.research.elamur.ai/free/en/authors/alon-lavie/), [André F. T. Martins](https://md.research.elamur.ai/free/en/authors/andre-f-t-martins/) · **Year:** 2022 · **Venue:** Proceedings of the Seventh Conference on Machine Translation (WMT) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.wmt-1.60/) · [PDF](https://aclanthology.org/2022.wmt-1.60.pdf) · [DOI 10.18653/v1/2022.wmt-1.60](https://doi.org/10.18653/v1/2022.wmt-1.60) · [OpenAlex](https://openalex.org/W7126414507)

**Citations:** 16 citations (openalex-snapshot · 2026-10-14) · [308](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
This paper describes the joint IST and Unbabel entry to the WMT 2022 Quality Estimation shared task. Quality Estimation (QE) scores machine translation output without reference translations. The team entered all three subtasks: sentence- and word-level quality prediction, Explainable QE, and Critical Error Detection.

They built COMETKIWI, which ports the predictor-estimator design of OpenKiwi into the COMET framework and adds a word-level OK/BAD tagger. Models were first pretrained on WMT Metrics Direct Assessment data using the UniTE objective, which exploits reference translations. They were then fine-tuned on source–translation pairs, with sentence and word losses optionally optimized together. Language pairs absent from training were handled through few-shot adaptation.

For explanations, the authors combined attention weights with gradient norms and learned how much each attention head should contribute. Ensembles of these systems ranked first in the multilingual setting of every track, often by a wide margin. This shows that reference-aware pretraining, multi-task objectives and attention-gradient explanations transfer well across languages.

## Key points
- **Encoders and data:** three encoders were tested, XLM-R Large, InfoXLM Large and RemBERT.
  - Pretraining used WMT DA judgments from 2017–2019, covering 24 LPs and about 657k samples.
  - Fine-tuning used MLQE-PE (DA, HTER) and the WMT 2020/2021 MQM annotations.
- **Few-shot adaptation:** about 500 dev examples each were used for km-en, ps-en, en-ja and en-cs. This helped those pairs without hurting the others. en-yo remained zero-shot.
- **Ensembles:** the DA submission averaged six models with per-LP weights tuned via Optuna. The MQM submission combined eleven models.
- **Multilingual results:**
  - 0.572 Spearman on sentence-level DA, about 7% above the runner-up.
  - 0.341 MCC on word level, about 2.4% higher.
  - 0.486 R@K on Explainable QE, about 10% higher.
- **Per-LP results and explanation method:** the systems were best on 6/9 LPs at sentence level and 5/9 at word level. In Explainable QE they led on all pairs except km-en and ps-en. The new Attn × GradNorm method with Head Mix outperformed last year's Attn × Norm baseline.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

308 papers of our library cite it (of the 140673 whose bibliographies we read); 34 of them have a page here, 274 not yet.

- [Assessing the Role of Context in Chat Translation Evaluation: Is Context Helpful and Under What Conditions?](https://md.research.elamur.ai/free/en/papers/agrawal-2024-assessing-role-context-chat-translation/)
- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://md.research.elamur.ai/free/en/papers/alves-2024-tower-open-multilingual-large-language/)
- [Findings of the WMT 2023 Shared Task on Quality Estimation](https://md.research.elamur.ai/free/en/papers/blain-2023-findings-wmt-2023-shared-task/)
- [M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/feng-2025-m-mad-multidimensional-multi-agent/)
- [The Devil Is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/fernandes-2023-devil-is-errors-leveraging-large/)
- [Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent](https://md.research.elamur.ai/free/en/papers/freitag-2023-results-wmt23-metrics-shared-task/)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://md.research.elamur.ai/free/en/papers/guerreiro-2024-xcomet-transparent-machine-translation-evaluation/)
- [Exploring Human-Like Translation Strategy with Large Language Models](https://md.research.elamur.ai/free/en/papers/he-2024-exploring-human-like-translation-strategy/)
- [MetricX-23: The Google Submission to the WMT 2023 Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/juraska-2023-metricx-23-google-submission-wmt/)
- [MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/juraska-2024-metricx-24-google-submission-wmt/)
- [Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets](https://md.research.elamur.ai/free/en/papers/kocmi-2025-findings-wmt25-general-machine-translation/)
- [SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages?](https://md.research.elamur.ai/free/en/papers/li-2025-ssa-comet-do-llms-outperform/)
- [XQ-MEval: A Dataset with Cross-lingual Parallel Quality for Benchmarking Translation Metrics](https://md.research.elamur.ai/free/en/papers/liu-2026-xq-meval-dataset-cross-lingual/)
- [MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluators](https://md.research.elamur.ai/free/en/papers/lu-2025-mqm-ape-high-quality-error/)
- [Hindsight Quality Prediction Experiments in Multi-Candidate Human-Post-Edited Machine Translation](https://md.research.elamur.ai/free/en/papers/marmonier-2026-hindsight-quality-prediction-experiments-multi/)
- [EuroLLM: Multilingual Language Models for Europe](https://md.research.elamur.ai/free/en/papers/martins-2024-eurollm-multilingual-language-models-europe/)
- [When the Gold Standard Isn't Necessarily Standard: Challenges of Evaluating the Translation of User-Generated Content](https://md.research.elamur.ai/free/en/papers/nishimwe-2026-when-gold-standard-isn-t/)
- [BSC's Submission to the Instruction Following Track of IWSLT 2026](https://md.research.elamur.ai/free/en/papers/pareras-2026-bsc-s-submission-instruction-following/)
- [Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics](https://md.research.elamur.ai/free/en/papers/perrella-2024-beyond-correlation-interpretable-evaluation-machine/)
- [Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!](https://md.research.elamur.ai/free/en/papers/perrella-2024-guardians-machine-translation-meta-evaluation/)
- [Machine translation Evaluation Eng-Thai MQM Ranking dataset](https://md.research.elamur.ai/free/en/papers/phuangrot-2026-machine-translation-evaluation-eng-thai/)
- [PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translation](https://md.research.elamur.ai/free/en/papers/proietti-2026-pear-pairwise-evaluation-automatic-relative/)
- [COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/rei-2022-comet-22-unbabel-ist-2022/)
- [Scaling up CometKiwi: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task](https://md.research.elamur.ai/free/en/papers/rei-2023-scaling-up-cometkiwi-unbabel-ist/)
- [Biases in Translation: Assessing Opinion Distortion in Machine Translated Texts](https://md.research.elamur.ai/free/en/papers/shafiabadi-2026-biases-translation-assessing-opinion-distortion/)
- [IIIT-BGP IWSLT 2026 Systems for Low-resource ST](https://md.research.elamur.ai/free/en/papers/singh-2026-iiit-bgp-iwslt-2026-systems/)
- [Scoring the Translation: On Target Automatic Keyword-Based Evaluation of Machine Translation in the Sports Domain](https://md.research.elamur.ai/free/en/papers/steingrimsson-2026-scoring-translation-target-automatic-keyword/)
- [Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation](https://md.research.elamur.ai/free/en/papers/tian-2026-beyond-literal-mapping-benchmarking-improving/)
- [Multilingual Long-Form Speech Instruction Following: KIT's Submission to IWSLT 2026](https://md.research.elamur.ai/free/en/papers/ugan-2026-multilingual-long-form-speech-instruction/)
- [Diagnose, Then Repair: A Two-Stage MQM-Guided Post-Editing Framework for Domain-Specific Machine Translation](https://md.research.elamur.ai/free/en/papers/wang-2026-diagnose-then-repair-two-stage/)
- [On Temperature-Constrained Non-Deterministic Machine Translation: Potential and Evaluation](https://md.research.elamur.ai/free/en/papers/wang-2026-temperature-constrained-non-deterministic-machine/)
- [Embedding Similarity Is Not Quality Estimation: Lessons from Replacing a Dedicated QE Model](https://md.research.elamur.ai/free/en/papers/zaikis-2026-embedding-similarity-is-not-quality/)
- [Findings of the WMT 2022 Shared Task on Quality Estimation](https://md.research.elamur.ai/free/en/papers/zerva-2022-findings-wmt-2022-shared-task/)
- [Early-Exit and Instant Confidence Translation Quality Estimation](https://md.research.elamur.ai/free/en/papers/zouhar-2026-early-exit-instant-confidence-translation/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{rei2022cometkiwi,
  title = {{CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task}},
  author = {Rei, Ricardo and Treviso, Marcos and Guerreiro, Nuno M. and Zerva, Chrysoula and Farinha, Ana C and Maroti, Christine and Souza, José G. C. de and Glushkova, Taisiya and Alves, Duarte and Coheur, Luisa and Lavie, Alon and Martins, André F. T.},
  booktitle = {Proceedings of the Seventh Conference on Machine Translation (WMT)},
  year = {2022},
  doi = {10.18653/v1/2022.wmt-1.60},
  url = {https://doi.org/10.18653/v1/2022.wmt-1.60}
}
```

- [← previous COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/rei-2022-comet-22-unbabel-ist-2022/)
- [next → Comparing Native and Learner Englishes Using a Large Pre-trained Language Model](https://md.research.elamur.ai/free/en/papers/aoyama-2022-comparing-native-learner-englishes-large/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/)

## References

34 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. [find](https://scholar.google.com/scholar?q=Takuya+Akiba%2C+Shotaro+Sano%2C+Toshihiko+Yanase%2C+Takeru+Ohta%2C+and+Masanori+Koyama%2E+2019%2E+Optuna%3A+A+next%2Dgeneration+hyperparameter+optimization+framework%2E+In+Proceedings+of+the+25rd+ACM+SIGKDD+Internation) [our card](https://md.research.elamur.ai/free/en/papers/akiba-2019-optuna-next-generation-hyperparameter-optimization/)
2. Leila Arras, Franziska Horn, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek. 2016. Explaining predictions of non-linear classifiers in NLP. In Proceedings of the 1st Workshop on Representation Learning for NLP, pages 1–7, Berlin, Germany. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Leila+Arras%2C+Franziska+Horn%2C+Gr%C3%A9goire+Montavon%2C+Klaus%2DRobert+M%C3%BCller%2C+and+Wojciech+Samek%2E+2016%2E+Explaining+predictions+of+non%2Dlinear+classifiers+in+NLP%2E+In+Proceedings+of+the+1st+Workshop+on+Representa)
3. Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021. InfoXLM: An information-theoretic framework for cross-lingual language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Assoc… [find](https://scholar.google.com/scholar?q=Zewen+Chi%2C+Li+Dong%2C+Furu+Wei%2C+Nan+Yang%2C+Saksham+Singhal%2C+Wenhui+Wang%2C+Xia+Song%2C+Xian%2DLing+Mao%2C+Heyan+Huang%2C+and+Ming+Zhou%2E+2021%2E+InfoXLM%3A+An+information%2Dtheoretic+framework+for+cross%2Dlingual+language+) [our card](https://md.research.elamur.ai/free/en/papers/chi-2020-infoxlm-information-theoretic-framework-cross/)
4. George Chrysostomou and Nikolaos Aletras. 2022. An empirical study on explanations in out-of-domain settings. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6920–6938, Dublin, Ireland. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=George+Chrysostomou+and+Nikolaos+Aletras%2E+2022%2E+An+empirical+study+on+explanations+in+out%2Dof%2Ddomain+settings%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Linguistics)
5. Hyung Won Chung, Thibault Fevry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2021. Rethinking Embedding Coupling in Pre-trained Language Models. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Thibault+Fevry%2C+Henry+Tsai%2C+Melvin+Johnson%2C+and+Sebastian+Ruder%2E+2021%2E+Rethinking+Embedding+Coupling+in+Pre%2Dtrained+Language+Models%2E+In+International+Conference+on+Learning+Representa) [our card](https://md.research.elamur.ai/free/en/papers/chung-2020-rethinking-embedding-coupling-pre-trained/)
6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dlingu)
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
8. Marina Fomicheva, Piyawat Lertvittayakumjorn, Wei Zhao, Steffen Eger, and Yang Gao. 2021. The Eval4NLP shared task on explainable quality estimation: Overview and results. In Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, pages 165–178, Punta Cana, Dominican Republic. A… [find](https://scholar.google.com/scholar?q=Marina+Fomicheva%2C+Piyawat+Lertvittayakumjorn%2C+Wei+Zhao%2C+Steffen+Eger%2C+and+Yang+Gao%2E+2021%2E+The+Eval4NLP+shared+task+on+explainable+quality+estimation%3A+Overview+and+results%2E+In+Proceedings+of+the+2nd+Wo)
9. Marina Fomicheva, Shuo Sun, Erick Fonseca, Frédéric Blain, Vishrav Chaudhary, Francisco Guzmán, Nina Lopatina, Lucia Specia, and André F. T. Martins. 2022. MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset. In Proceedings of the Language Resources and Evaluation Conference, pages 4… [find](https://scholar.google.com/scholar?q=Marina+Fomicheva%2C+Shuo+Sun%2C+Erick+Fonseca%2C+Fr%C3%A9d%C3%A9ric+Blain%2C+Vishrav+Chaudhary%2C+Francisco+Guzm%C3%A1n%2C+Nina+Lopatina%2C+Lucia+Specia%2C+and+Andr%C3%A9+F%2E+T%2E+Martins%2E+2022%2E+MLQE%2DPE%3A+A+Multilingual+Quality+Estimation+a) [our card](https://md.research.elamur.ai/free/en/papers/fomicheva-2022-mlqe-pe-multilingual-quality-estimation/)
10. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021a. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460–1474. [find](https://scholar.google.com/scholar?q=Markus+Freitag%2C+George+Foster%2C+David+Grangier%2C+Viresh+Ratnakar%2C+Qijun+Tan%2C+and+Wolfgang+Macherey%2E+2021a%2E+Experts%2C+errors%2C+and+context%3A+A+large%2Dscale+study+of+human+evaluation+for+machine+translation%2E+) [our card](https://md.research.elamur.ai/free/en/papers/freitag-2021-experts-errors-context-large-scale/)
11. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021b. Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain. In Proceedings of the Sixth Conference on Machine Trans… [find](https://scholar.google.com/scholar?q=Markus+Freitag%2C+Ricardo+Rei%2C+Nitika+Mathur%2C+Chi%2Dkiu+Lo%2C+Craig+Stewart%2C+George+Foster%2C+Alon+Lavie%2C+and+Ondr%CC%8Cej+Bojar%2E+2021b%2E+Results+of+the+WMT21+metrics+shared+task%3A+Evaluating+metrics+with+expert%2Dbas) [our card](https://md.research.elamur.ai/free/en/papers/freitag-2021-results-wmt21-metrics-shared-task/)
12. Nuno M. Guerreiro and André F. T. Martins. 2021. SPECTRA: Sparse structured text rationalization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6534–6550, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nuno+M%2E+Guerreiro+and+Andr%C3%A9+F%2E+T%2E+Martins%2E+2021%2E+SPECTRA%3A+Sparse+structured+text+rationalization%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+6534%E2%80%93)
13. Julia Ive, Lucia Specia, Sara Szoc, Tom Vanallemeersch, Joachim Van den Bogaert, Eduardo Farah, Christine Maroti, Artur Ventura, and Maxim Khalilov. 2020. A post-editing dataset in the legal domain: Do we underestimate neural machine translation quality? In Proceedings of the 12th Language Resources… [find](https://scholar.google.com/scholar?q=Julia+Ive%2C+Lucia+Specia%2C+Sara+Szoc%2C+Tom+Vanallemeersch%2C+Joachim+Van+den+Bogaert%2C+Eduardo+Farah%2C+Christine+Maroti%2C+Artur+Ventura%2C+and+Maxim+Khalilov%2E+2020%2E+A+post%2Dediting+dataset+in+the+legal+domain%3A+D)
14. Fabio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, and André F. T. Martins. 2019. OpenKiwi: An open source framework for quality estimation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 117–122, Florence, Italy. Assoc… [find](https://scholar.google.com/scholar?q=Fabio+Kepler%2C+Jonay+Tr%C3%A9nous%2C+Marcos+Treviso%2C+Miguel+Vera%2C+and+Andr%C3%A9+F%2E+T%2E+Martins%2E+2019%2E+OpenKiwi%3A+An+open+source+framework+for+quality+estimation%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Ass) [our card](https://md.research.elamur.ai/free/en/papers/kepler-2019-openkiwi-open-source-framework-quality/)
15. Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020. Attention is not only a weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7057–7075, Online. Association for Computational… [find](https://scholar.google.com/scholar?q=Goro+Kobayashi%2C+Tatsuki+Kuribayashi%2C+Sho+Yokoi%2C+and+Kentaro+Inui%2E+2020%2E+Attention+is+not+only+a+weight%3A+Analyzing+transformers+with+vector+norms%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Met) [our card](https://md.research.elamur.ai/free/en/papers/kobayashi-2020-attention-is-not-only-weight/)
16. Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 107–117, Austin, Texas. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tao+Lei%2C+Regina+Barzilay%2C+and+Tommi+Jaakkola%2E+2016%2E+Rationalizing+neural+predictions%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+107%E2%80%93117%2C+Austin%2C+)
17. Andre Martins and Ramon Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International Conference on Machine Learning, pages 1614–1623. [find](https://scholar.google.com/scholar?q=Andre+Martins+and+Ramon+Astudillo%2E+2016%2E+From+softmax+to+sparsemax%3A+A+sparse+model+of+attention+and+multi%2Dlabel+classification%2E+In+International+Conference+on+Machine+Learning%2C+pages+1614%E2%80%931623%2E)
18. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno… [find](https://scholar.google.com/scholar?q=Matthew+E%2E+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+Proceedings+of+the+2018+Conference+o)
19. Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. 2020. TransQuest at WMT2020: Sentence-level direct assessment. In Proceedings of the Fifth Conference on Machine Translation, pages 1049– 1055, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tharindu+Ranasinghe%2C+Constantin+Orasan%2C+and+Ruslan+Mitkov%2E+2020%2E+TransQuest+at+WMT2020%3A+Sentence%2Dlevel+direct+assessment%2E+In+Proceedings+of+the+Fifth+Conference+on+Machine+Translation%2C+pages+1049%E2%80%93+105)
20. Ricardo Rei, Ana C Farinha, José G.C. de Souza, Pedro G. Ramos, André F.T. Martins, Luisa Coheur, and Alon Lavie. 2022. Searching for COMETINHO: The little metric that could. In Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, pages 61–70, Ghent, Belgium… [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Ana+C+Farinha%2C+Jos%C3%A9+G%2EC%2E+de+Souza%2C+Pedro+G%2E+Ramos%2C+Andr%C3%A9+F%2ET%2E+Martins%2C+Luisa+Coheur%2C+and+Alon+Lavie%2E+2022%2E+Searching+for+COMETINHO%3A+The+little+metric+that+could%2E+In+Proceedings+of+the+23r)
21. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ricardo+Rei%2C+Craig+Stewart%2C+Ana+C+Farinha%2C+and+Alon+Lavie%2E+2020%2E+COMET%3A+A+neural+framework+for+MT+evaluation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing+) [our card](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/)
22. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Why should I trust you?: Explaining the predictions of any classifier. In Proc. ACM SIGKDD, pages 1135–1144. ACM. [find](https://scholar.google.com/scholar?q=Marco+Tulio+Ribeiro%2C+Sameer+Singh%2C+and+Carlos+Guestrin%2E+2016%2E+Why+should+I+trust+you%3F%3A+Explaining+the+predictions+of+any+classifier%2E+In+Proc%2E+ACM+SIGKDD%2C+pages+1135%E2%80%931144%2E+ACM%2E) [our card](https://md.research.elamur.ai/free/en/papers/ribeiro-2016-why-should-i-trust-explaining/)
23. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In In Proceedings of Association for Machine Translation in the Americas, pages 223–231. [find](https://scholar.google.com/scholar?q=Matthew+Snover%2C+Bonnie+Dorr%2C+Richard+Schwartz%2C+Linnea+Micciulla%2C+and+John+Makhoul%2E+2006%2E+A+study+of+translation+edit+rate+with+targeted+human+annotation%2E+In+In+Proceedings+of+Association+for+Machine+T) [our card](https://md.research.elamur.ai/free/en/papers/snover-2006-study-translation-edit-rate-targeted/)
24. Orăsan, Marina Fomicheva, André F. T. Martins, and Lucia Specia. 2022. Findings of the WMT 2022 Shared Task on Quality Estimation. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ora%CC%86san%2C+Marina+Fomicheva%2C+Andr%C3%A9+F%2E+T%2E+Martins%2C+and+Lucia+Specia%2E+2022%2E+Findings+of+the+WMT+2022+Shared+Task+on+Quality+Estimation%2E+In+Proceedings+of+the+Seventh+Conference+on+Machine+Translation%2C+Abu) [our card](https://md.research.elamur.ai/free/en/papers/zerva-2022-findings-wmt-2022-shared-task/)
25. Chrysoula Zerva, Daan van Stigt, Ricardo Rei, Ana C Farinha, Pedro Ramos, José G. C. de Souza, Taisiya Glushkova, Miguel Vera, Fabio Kepler, and André F. T. Martins. 2021. [find](https://scholar.google.com/scholar?q=Chrysoula+Zerva%2C+Daan+van+Stigt%2C+Ricardo+Rei%2C+Ana+C+Farinha%2C+Pedro+Ramos%2C+Jos%C3%A9+G%2E+C%2E+de+Souza%2C+Taisiya+Glushkova%2C+Miguel+Vera%2C+Fabio+Kepler%2C+and+Andr%C3%A9+F%2E+T%2E+Martins%2E+2021%2E)
26. IST-unbabel 2021 submission for the quality estimation shared task. In Proceedings of the Sixth Conference on Machine Translation, pages 961–972, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=IST%2Dunbabel+2021+submission+for+the+quality+estimation+shared+task%2E+In+Proceedings+of+the+Sixth+Conference+on+Machine+Translation%2C+pages+961%E2%80%93972%2C+Online%2E+Association+for+Computational+Linguistics%2E)
27. A The data used for finetuning our QE systems is shown in Table 5. For DA data, we split the original development set to generate a new dev/test split, therefore the reported numbers in the table correspond to this “internal” dev split. [find](https://scholar.google.com/scholar?q=A+The+data+used+for+finetuning+our+QE+systems+is+shown+in+Table+5%2E+For+DA+data%2C+we+split+the+original+development+set+to+generate+a+new+dev%2Ftest+split%2C+therefore+the+reported+numbers+in+the+table+corr)
28. Lucia Specia, Kim Harris, Frédéric Blain, Aljoscha Burchardt, Viviven Macketanz, Inguna Skadin, Matteo Negri, and Marco Turchi. 2017. Translation quality and productivity: A study on rich morphology languages. In Proceedings of Machine Translation Summit XVI: Research Track, pages 55–71, Nagoya Japa… [find](https://scholar.google.com/scholar?q=Lucia+Specia%2C+Kim+Harris%2C+Fr%C3%A9d%C3%A9ric+Blain%2C+Aljoscha+Burchardt%2C+Viviven+Macketanz%2C+Inguna+Skadin%2C+Matteo+Negri%2C+and+Marco+Turchi%2E+2017%2E+Translation+quality+and+productivity%3A+A+study+on+rich+morphology+l)
29. LP TRAIN en-de en-mr en-zh et-en ne-en ro-en ru-en si-en en-de † en-ru † zh-en † Lucia Specia, Carolina Scarton, and Gustavo Henrique Paetzold. 2018. Quality estimation for machine translation. Synthesis Lectures on Human Language Technologies, 11(1):1–162. [find](https://scholar.google.com/scholar?q=LP+TRAIN+en%2Dde+en%2Dmr+en%2Dzh+et%2Den+ne%2Den+ro%2Den+ru%2Den+si%2Den+en%2Dde+%E2%80%A0+en%2Dru+%E2%80%A0+zh%2Den+%E2%80%A0+Lucia+Specia%2C+Carolina+Scarton%2C+and+Gustavo+Henrique+Paetzold%2E+2018%2E+Quality+estimation+for+machine+translation%2E+Synthe)
30. Marcos Treviso, Nuno M. Guerreiro, Ricardo Rei, and André F. T. Martins. 2021. [find](https://scholar.google.com/scholar?q=Marcos+Treviso%2C+Nuno+M%2E+Guerreiro%2C+Ricardo+Rei%2C+and+Andr%C3%A9+F%2E+T%2E+Martins%2E+2021%2E)
31. IST-unbabel 2021 submission for the explainable quality estimation shared task. In Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, pages 133– 145, Punta Cana, Dominican Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=IST%2Dunbabel+2021+submission+for+the+explainable+quality+estimation+shared+task%2E+In+Proceedings+of+the+2nd+Workshop+on+Evaluation+and+Comparison+of+NLP+Systems%2C+pages+133%E2%80%93+145%2C+Punta+Cana%2C+Dominican+Re)
32. DEV en-de en-mr en-zh et-en ne-en ro-en ru-en si-en en-cs en-ja km-en ps-en en-de † en-ru † zh-en † Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Process… [find](https://scholar.google.com/scholar?q=DEV+en%2Dde+en%2Dmr+en%2Dzh+et%2Den+ne%2Den+ro%2Den+ru%2Den+si%2Den+en%2Dcs+en%2Dja+km%2Den+ps%2Den+en%2Dde+%E2%80%A0+en%2Dru+%E2%80%A0+zh%2Den+%E2%80%A0+Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81+ukasz+Kais)
33. Yu Wan, Dayiheng Liu, Baosong Yang, Haibo Zhang, Boxing Chen, Derek Wong, and Lidia Chao. 2022. UniTE: Unified translation evaluation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8117–8127, Dublin, Ireland. Association for… [find](https://scholar.google.com/scholar?q=Yu+Wan%2C+Dayiheng+Liu%2C+Baosong+Yang%2C+Haibo+Zhang%2C+Boxing+Chen%2C+Derek+Wong%2C+and+Lidia+Chao%2E+2022%2E+UniTE%3A+Unified+translation+evaluation%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+) [our card](https://md.research.elamur.ai/free/en/papers/wan-2022-unite-unified-translation-evaluation/)
34. Chrysoula Zerva, Frédéric Blain, Ricardo Rei, Piyawat Lertvittayakumjorn, José G. C. de Souza, Steffen Eger, Diptesh Kanojia, Duarte Alves, Constantin Data Information Samples Source Tokens Target Tokens OK / BAD Target 9000 26000 9000 9000 9000 9000 9000 9000 54681 15628 75327 147870 690516 148657… [find](https://scholar.google.com/scholar?q=Chrysoula+Zerva%2C+Fr%C3%A9d%C3%A9ric+Blain%2C+Ricardo+Rei%2C+Piyawat+Lertvittayakumjorn%2C+Jos%C3%A9+G%2E+C%2E+de+Souza%2C+Steffen+Eger%2C+Diptesh+Kanojia%2C+Duarte+Alves%2C+Constantin+Data+Information+Samples+Source+Tokens+Target+Tok)