---
title: "TransQuest — paper by Ranasinghe et al. (2020)"
description: "The authors present TransQuest, an open-source PyTorch framework that puts a lightweight output… 29 references, cited by 88 papers here."
authors: ["Tharindu Ranasinghe", "Constantin Orasan", "Ruslan Mitkov"]
year: 2020
venue: "Proceedings of the 28th International Conference on Computational Linguistics"
source: "https://aclanthology.org/2020.coling-main.445/"
doi: "10.18653/v1/2020.coling-main.445"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ranasinghe-2020-transquest-translation-quality-estimation-cross/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ranasinghe-2020-transquest-translation-quality-estimation-cross/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/) / TransQuest: Translation Quality Estimation with Cross-lingual Transformers

# TransQuest: Translation Quality Estimation with Cross-lingual Transformers

**Authors:** [Tharindu Ranasinghe](https://md.research.elamur.ai/free/en/authors/tharindu-ranasinghe/), [Constantin Orasan](https://md.research.elamur.ai/free/en/authors/constantin-orasan/) and [Ruslan Mitkov](https://md.research.elamur.ai/free/en/authors/ruslan-mitkov/) · **Year:** 2020 · **Venue:** [Proceedings of the 28th International Conference on Computational Linguistics](https://md.research.elamur.ai/free/en/venues/coling/) ([COLING 2020](https://md.research.elamur.ai/free/en/venues/coling/2020/)) · **Publisher:** [International Committee on Computational Linguistics (ICCL)](https://md.research.elamur.ai/free/en/publishers/international-committee-on-computational-linguistics-iccl/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.coling-main.445/) · [PDF](https://aclanthology.org/2020.coling-main.445.pdf) · [DOI 10.18653/v1/2020.coling-main.445](https://doi.org/10.18653/v1/2020.coling-main.445) · [OpenAlex](https://openalex.org/W3095106730)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [88](https://md.research.elamur.ai/free/en/papers/ranasinghe-2020-transquest-translation-quality-estimation-cross/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Sentence-level quality estimation (QE) predicts how good a machine translation is without a reference translation. Earlier neural QE systems, such as Predictor-Estimator models in deepQuest and OpenKiwi, need large parallel corpora for pre-training. They are costly to train and usually have to be retrained for every language pair. The authors present TransQuest, an open-source PyTorch framework that puts a lightweight output layer on top of the pre-trained cross-lingual XLM-R-large model. It does not use any parallel data. The framework includes two architectures: MTransQuest and the siamese STransQuest. The authors evaluated both on WMT QE datasets covering 15 language pairs. The models outperform existing open-source QE tools and set new state-of-the-art results. A single model can also serve several language pairs at once. For low-resource pairs, initialising from models trained on mid-resource data gives strong results with very few annotated examples. This makes practical QE affordable for many languages, and the released pre-trained model zoo supports this.

## Key points
- Both architectures use XLM-R-large from HuggingFace Transformers, which covers 104 languages. MTransQuest is consistently more accurate, while STransQuest is faster. On a Tesla K80, training on 7,000 instances takes about 4,480s for MTransQuest and 3,900s for STransQuest. Inference on 1,000 instances takes 35s and 16s respectively.
- A tuned TransQuest won all 8 tasks of the WMT 2020 sentence-level direct assessment (DA) QE shared task.
- A model for one language pair takes about 2GB. A multi-language-pair model stays at or below 2.1GB, so one model can replace several.
- Transfer learning started from TransQuest-Mid, a model trained on Et-En, Ro-En and Ru-En. For Ne-En with only 100 training pairs, MTransQuest reached a Pearson correlation of 0.7417, compared with 0.1242 when trained from scratch. With 7,000 pairs it reached 0.7914. Si-En showed similar gains.
- The code and the TransQuest model zoo of pre-trained models are available at github.com/tharindudr/transquest.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Wolverhampton](https://md.research.elamur.ai/free/en/universities/university-of-wolverhampton/) — University, Wolverhampton, United Kingdom · 30 works · 6,272 citations
- [University of Surrey](https://md.research.elamur.ai/free/en/universities/university-of-surrey/) — University, Guildford, United Kingdom · 7 works · 566 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 958 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 467 works

## Appears in

Agglomerations: [🏙️ London](https://md.research.elamur.ai/free/en/agglomerations/london/)

## Cite

Cite as: Tharindu Ranasinghe, Constantin Orasan and Ruslan Mitkov. 2020. TransQuest: Translation Quality Estimation with Cross-lingual Transformers. Proceedings of the 28th International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ranasinghe2020transquest,
  title = {{TransQuest: Translation Quality Estimation with Cross-lingual Transformers}},
  author = {Tharindu Ranasinghe, Constantin Orasan and Ruslan Mitkov},
  booktitle = {Proceedings of the 28th International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.445},
  url = {https://doi.org/10.18653/v1/2020.coling-main.445}
}
```

- [← previous A Report on the 2020 Sarcasm Detection Shared Task](https://md.research.elamur.ai/free/en/papers/ghosh-2020-report-2020-sarcasm-detection-shared/)
- [next → Generationary or «How We Went beyond Word Sense Inventories and Learned to Gloss»](https://md.research.elamur.ai/free/en/papers/bevilacqua-2020-generationary/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/)

## References

29 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems 32, pages 7059–7069. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Alexis+Conneau+and+Guillaume+Lample%2E+2019%2E+Cross%2Dlingual+language+model+pretraining%2E+In+Advances+in+Neural+Information+Processing+Systems+32%2C+pages+7059%E2%80%937069%2E+Curran+Associates%2C+Inc%2E)
2. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzma%CC%81n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dling)
3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June. Association for Computational Linguistics.
4. Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020. [Unsupervised quality estimation for neural machine translation](https://md.research.elamur.ai/free/en/papers/fomicheva-2020-unsupervised-quality-estimation-neural-machine/). Transactions of the Association for Computational Linguistics, Vol 8 (2020). [↗](https://doi.org/10.1162/tacl_a_00330)
5. Erick Fonseca, Lisa Yankovskaya, André F. T. Martins, Mark Fishel, and Christian Federmann. 2019. [Findings of the WMT 2019 shared tasks on quality estimation](https://md.research.elamur.ai/free/en/papers/fonseca-2019-findings-wmt-2019-shared-tasks/). In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 1–10, Florence, Italy, August. Association for Computational Linguistics. 3 Several workshops like EM C 2 (Workshop on Energy Efficient Machine Learning and Cognitive Computing), SustaiNLP 2020 (Workshop on Simple and Efficient Natural Language Processing) has been organised on this aspect.
6. Yvette Graham, Timothy Baldwin, and Nitika Mathur. 2015. [Accurate evaluation of segment-level machine translation metrics](https://md.research.elamur.ai/free/en/papers/graham-2015-accurate-evaluation-segment-level-machine/). In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1183–1191, Denver, Colorado, May– June. Association for Computational Linguistics.
7. Yvette Graham, Timothy Baldwin, Meghan Dowling, Maria Eskevich, Teresa Lynn, and Lamia Tounsi. 2016. Is all that glitters in machine translation quality estimation really gold? In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3124–3134, Osaka, Japan, December. [find](https://scholar.google.com/scholar?q=Yvette+Graham%2C+Timothy+Baldwin%2C+Meghan+Dowling%2C+Maria+Eskevich%2C+Teresa+Lynn%2C+and+Lamia+Tounsi%2E+2016%2E+Is+all+that+glitters+in+machine+translation+quality+estimation+really+gold%3F+In+Proceedings+of+COLIN)
8. The COLING 2016 Organizing Committee. [find](https://scholar.google.com/scholar?q=The+COLING+2016+Organizing+Committee%2E)
9. Yvette Graham, Timothu Baldwin, Alistair Moffat, and Justin Zobel. 2017. Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering, 23(1):3–30. [find](https://scholar.google.com/scholar?q=Yvette+Graham%2C+Timothu+Baldwin%2C+Alistair+Moffat%2C+and+Justin+Zobel%2E+2017%2E+Can+machine+translation+systems+be+evaluated+by+the+crowd+alone%2E+Natural+Language+Engineering%2C+23%281%29%3A3%E2%80%9330%2E)
10. Julia Ive, Frédéric Blain, and Lucia Specia. 2018. deepQuest: A framework for neural-based quality estimation. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3146–3157, Santa Fe, New Mexico, USA, August. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Julia+Ive%2C+Fre%CC%81de%CC%81ric+Blain%2C+and+Lucia+Specia%2E+2018%2E+deepQuest%3A+A+framework+for+neural%2Dbased+quality+estimation%2E+In+Proceedings+of+the+27th+International+Conference+on+Computational+Linguistics%2C+pages)
11. Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual bert: An empirical study. In International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.1912.07840)
12. Fabio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, and André F. T. Martins. 2019. [OpenKiwi: An open source framework for quality estimation](https://md.research.elamur.ai/free/en/papers/kepler-2019-openkiwi-open-source-framework-quality/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 117–122, Florence, Italy, July. Association for Computational Linguistics.
13. Hyun Kim, Jong-Hyeok Lee, and Seung-Hoon Na. 2017. [Predictor-estimator using multilevel task learning with stack propagation for neural quality estimation](https://md.research.elamur.ai/free/en/papers/kim-2017-predictor-estimator-multilevel-task-learning/). In Proceedings of the Second Conference on Machine Translation, pages 562–568, Copenhagen, Denmark, September. Association for Computational Linguistics.
14. Julia Kreutzer, Shigehiko Schamoni, and Stefan Riezler. 2015. QUality estimation from ScraTCH (QUETCH): Deep learning for word-level translation quality estimation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 316–322, Lisbon, Portugal, September. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Julia+Kreutzer%2C+Shigehiko+Schamoni%2C+and+Stefan+Riezler%2E+2015%2E+QUality+estimation+from+ScraTCH+%28QUETCH%29%3A+Deep+learning+for+word%2Dlevel+translation+quality+estimation%2E+In+Proceedings+of+the+Tenth+Worksho)
15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+Robustly+Optimized+BERT+Pretraining+Approach%2E)
16. André F. T. Martins, Ramón Astudillo, Chris Hokamp, and Fabio Kepler. 2016. Unbabel’s participation in the WMT16 word-level translation quality estimation shared task. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 806–811, Berlin, Germany, August. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Andre%CC%81+F%2E+T%2E+Martins%2C+Ramo%CC%81n+Astudillo%2C+Chris+Hokamp%2C+and+Fabio+Kepler%2E+2016%2E+Unbabel%E2%80%99s+participation+in+the+WMT16+word%2Dlevel+translation+quality+estimation+shared+task%2E+In+Proceedings+of+the+First+Co)
17. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. [fairseq: A fast, extensible toolkit for sequence modeling](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 48–53, Minneapolis, Minnesota, June. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1904.01038)
18. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. [How multilingual is multilingual BERT?](https://md.research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/) In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996–5001, Florence, Italy, July. Association for Computational Linguistics.
19. Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. 2019. Semantic textual similarity with Siamese neural networks. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 1004–1011, Varna, Bulgaria, September. INCOMA Ltd. [find](https://scholar.google.com/scholar?q=Tharindu+Ranasinghe%2C+Constantin+Orasan%2C+and+Ruslan+Mitkov%2E+2019%2E+Semantic+textual+similarity+with+Siamese+neural+networks%2E+In+Proceedings+of+the+International+Conference+on+Recent+Advances+in+Natural+)
20. Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. 2020. TransQuest at WMT2020: Sentence-Level Direct Assessment. In Proceedings of the Fifth Conference on Machine Translation: Shared Task Papers, online, November. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tharindu+Ranasinghe%2C+Constantin+Orasan%2C+and+Ruslan+Mitkov%2E+2020%2E+TransQuest+at+WMT2020%3A+Sentence%2DLevel+Direct+Assessment%2E+In+Proceedings+of+the+Fifth+Conference+on+Machine+Translation%3A+Shared+Task+Pap)
21. Nils Reimers and Iryna Gurevych. 2019. [Sentence-BERT: Sentence embeddings using Siamese BERT-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China, November. Association for Computational Linguistics.
22. Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2019. A survey of cross-lingual word embedding models. J. Artif. Int. Res., 65(1):569–630, May. [↗](https://doi.org/10.1613/jair.1.11640)
23. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. In NeurIPS EM C 2 Workshop. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Lysandre+Debut%2C+Julien+Chaumond%2C+and+Thomas+Wolf%2E+2019%2E+Distilbert%2C+a+distilled+version+of+bert%3A+smaller%2C+faster%2C+cheaper+and+lighter%2E+In+NeurIPS+EM+C+2+Workshop%2E)
24. Lucia Specia, Kashif Shah, Jose G.C. de Souza, and Trevor Cohn. 2013. [QuEst - a translation quality estimation framework](https://md.research.elamur.ai/free/en/papers/specia-2013-quest-translation-quality-estimation-framework/). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 79–84, Sofia, Bulgaria, August. Association for Computational Linguistics.
25. Lucia Specia, Gustavo Paetzold, and Carolina Scarton. 2015. [Multi-level translation quality prediction with QuEst++](https://md.research.elamur.ai/free/en/papers/specia-2015-multi-level-translation-quality-prediction/). In Proceedings of ACL-IJCNLP 2015 System Demonstrations, pages 115–120, Beijing, China, July. Association for Computational Linguistics and The Asian Federation of Natural Language Processing.
26. Lucia Specia, Frédéric Blain, Varvara Logacheva, Ramón Astudillo, and André F. T. Martins. 2018. [Findings of the WMT 2018 shared task on quality estimation](https://md.research.elamur.ai/free/en/papers/specia-2018-findings-wmt-2018-shared-task/). In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 689–709, Belgium, Brussels, October. Association for Computational Linguistics.
27. Lucia Specia, Frédéric Blain, Marina Fomicheva, Erick Fonseca, Vishrav Chaudhary, Francisco Guzmán, and André FT Martins. 2020. [Findings of the wmt 2020 shared task on quality estimation](https://md.research.elamur.ai/free/en/papers/specia-2020-findings-wmt-2020-shared-task-2020-wmt-1-79/). In Proceedings of the Fifth Conference on Machine Translation: Shared Task Papers.
28. Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. [How to fine-tune bert for text classification?](https://md.research.elamur.ai/free/en/papers/sun-2019-fine-tune-bert-text-classification/) In Maosong Sun, Xuanjing Huang, Heng Ji, Zhiyuan Liu, and Yang Liu, editors, Chinese Computational Linguistics, pages 194–206, Cham. Springer International Publishing. [↗](https://doi.org/10.48550/arxiv.1905.05583)
29. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s [transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). ArXiv, abs/1910.03771. [↗](https://arxiv.org/abs/1910.03771)

*Updated 24 October 2026*