---
title: "Multi-level Translation Quality Prediction — Specia 2015"
description: "Quality estimation (QE) predicts how good a machine translation is without a reference translation."
authors: ["Lucia Specia", "Gustavo Paetzold", "Carolina Scarton"]
year: 2015
venue: "Proceedings of ACL-IJCNLP 2015 System Demonstrations"
source: "https://aclanthology.org/P15-4020/"
doi: "10.3115/v1/p15-4020"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/specia-2015-multi-level-translation-quality-prediction/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/specia-2015-multi-level-translation-quality-prediction/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2015](https://md.research.elamur.ai/free/papers/2015/) / Multi-level Translation Quality Prediction with QuEst++

# Multi-level Translation Quality Prediction with QuEst++

**Authors:** [Lucia Specia](https://md.research.elamur.ai/free/authors/lucia-specia/), [Gustavo Paetzold](https://md.research.elamur.ai/free/authors/gustavo-paetzold/), [Carolina Scarton](https://md.research.elamur.ai/free/authors/carolina-scarton/) · **Year:** 2015 · **Venue:** Proceedings of ACL-IJCNLP 2015 System Demonstrations · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/P15-4020/) · [PDF](https://aclanthology.org/P15-4020.pdf) · [DOI 10.3115/v1/p15-4020](https://doi.org/10.3115/v1/p15-4020) · [OpenAlex](https://openalex.org/W2251150371)

**Citations:** 119 citations (openalex-snapshot · 2026-10-14) · [65](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Quality estimation (QE) predicts how good a machine translation is without a reference translation. Most prior work and tooling covered only sentence-level QE, while word-level and document-level QE lacked a shared, extensible framework. The authors present QuEst++, a heavily refactored and expanded version of the QuEst toolkit. It adds feature extractors for words and documents and joins all three levels into one pipeline, so that predictions at a finer level can serve as features at a coarser one.

The system has two parts:
- **Feature extraction**, written in Java and built around extensible abstract classes.
- **Learning**, with Python scripts around scikit-learn plus a CRFSuite interface for sequence labelling at the word level.

On WMT14 word-level data, QuEst++ beat every official participant except the English-Spanish baseline in two of the subtasks. The pipeline experiments gave mixed results. Predicted word labels did not help sentence-level QE, but gold word labels helped a great deal. Document-level gains were not significant. The toolkit is released under the BSD license, which permits commercial use.

## Key points
- **Word level:** QuEst++ offers 40 features of seven types:
  - target context n-grams
  - alignment context
  - POS-based lexical features
  - a syntactic Null Link feature
  - WordNet polysemy (via Universal WordNet)
  - LM backoff and longest preceding n-gram features
  - a pseudo-reference feature, using Moses output as the pseudo-reference
- **Sentence level:** 80–123 features are available depending on the language pair. The experiments used the 80 features that could be extracted for every pair.
- **WMT14 word-level setup:**
  - Three label schemes were tested: Binary, Level 1 and Multi-Class (16 error types).
  - The metric was average F-1 over all classes except Good.
  - German-English used CRFs with Passive Aggressive learning; the other datasets used AROW.
  - Hyper-parameters were tuned with 10-fold cross validation.
- **Word-to-sentence pipeline:**
  - Four features were derived from word labels: count and proportion of Good words, and the same for Bad words.
  - Predicted labels left MAE essentially unchanged (about 0.159 to 0.158).
  - Oracle labels cut MAE to about 0.07.
- **Sentence-to-document pipeline:**
  - Data: 934 English-Spanish WMT13 documents, split into 560 for training and 374 for testing.
  - Labels: BLEU, TER and METEOR.
  - Model: SVR with an RBF kernel.
  - No significant improvement over the aggregated baseline was found.

## Interesting ideas
- **Stacking predictions across granularity levels.** Fine-grained outputs are summarised and fed upward as features. The word-to-sentence summaries are counts and proportions of Good/Bad labels. The sentence-to-document summaries are min, max, mean, median and quartiles of predicted scores. This is a cheap way to build hierarchical models for any task with token, sentence and document structure.
- **Using an oracle run to measure a pipeline's ceiling.** Gold lower-level labels were fed in to see what the pipeline could achieve at best. This separates "the idea doesn't work" from "the upstream predictor is too weak," and is a useful diagnostic for any cascaded system.
- **Pseudo-references as a reference-free signal.** A word is flagged according to whether it also appears in another MT system's translation. Agreement between independent systems serves as a proxy for correctness, and the trick carries over to low-resource settings without gold references.
- **Graceful resource handling.** Feature extraction is configured through files, and the toolkit can build missing resources such as language models automatically. This makes it easier to port to new language pairs where resources are scarce.
- **Being cautious about proxy labels.** The authors observe that BLEU, TER and METEOR were designed to compare translations of the same source. Used as targets across different documents, they may not separate quality well. This is a warning for anyone training on automatic-metric labels.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/)

## Cited by

Cited by

65 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 57 not yet.

- [Findings of the 2015 Workshop on Statistical Machine Translation](https://md.research.elamur.ai/free/papers/bojar-2015-findings-2015-workshop-statistical-machine/)
- [Findings of the 2016 Conference on Machine Translation](https://md.research.elamur.ai/free/papers/bojar-2016-findings-2016-conference-machine-translation/)
- [Findings of the 2017 Conference on Machine Translation (WMT17)](https://md.research.elamur.ai/free/papers/bojar-2017-findings-2017-conference-machine-translation/)
- [Findings of the WMT 2019 Shared Tasks on Quality Estimation](https://md.research.elamur.ai/free/papers/fonseca-2019-findings-wmt-2019-shared-tasks/)
- [OpenKiwi: An Open Source Framework for Quality Estimation](https://md.research.elamur.ai/free/papers/kepler-2019-openkiwi-open-source-framework-quality/)
- [TransQuest: Translation Quality Estimation with Cross-lingual Transformers](https://md.research.elamur.ai/free/papers/ranasinghe-2020-transquest-translation-quality-estimation-cross/)
- [Findings of the WMT 2018 Shared Task on Quality Estimation](https://md.research.elamur.ai/free/papers/specia-2018-findings-wmt-2018-shared-task/)
- [Findings of the WMT 2020 Shared Task on Quality Estimation](https://md.research.elamur.ai/free/papers/specia-2020-findings-wmt-2020-shared-task-2020-wmt-1-79/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{specia2015multi,
  title = {{Multi-level Translation Quality Prediction with QuEst++}},
  author = {Specia, Lucia and Paetzold, Gustavo and Scarton, Carolina},
  booktitle = {Proceedings of ACL-IJCNLP 2015 System Demonstrations},
  year = {2015},
  doi = {10.3115/v1/p15-4020},
  url = {https://doi.org/10.3115/v1/p15-4020}
}
```

- [← previous Minimum Risk Training for Neural Machine Translation](https://md.research.elamur.ai/free/papers/shen-2015-minimum-risk-training-neural-machine/)
- [next → Multi-Task Learning for Multiple Language Translation](https://md.research.elamur.ai/free/papers/dong-2015-multi-task-learning-multiple-language/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/papers/2015/)

## References

19 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. N. Bach, F. Huang, and Y. Al-Onaizan. 2011. Goodness: a method for measuring MT confidence. In ACL11. [find](https://scholar.google.com/scholar?q=N%2E+Bach%2C+F%2E+Huang%2C+and+Y%2E+Al%2DOnaizan%2E+2011%2E+Goodness%3A+a+method+for+measuring+MT+confidence%2E+In+ACL11%2E)
2. E. Bicici and A. Way. 2014. Referential Translation Machines for Predicting Translation Quality. In WMT14. [find](https://scholar.google.com/scholar?q=E%2E+Bicici+and+A%2E+Way%2E+2014%2E+Referential+Translation+Machines+for+Predicting+Translation+Quality%2E+In+WMT14%2E)
3. J. Blatz, E. Fitzgerald, G. Foster, S. Gandrabur, C. Goutte, A. Kulesza, A. Sanchis, and N. Ueffing. 2004. Confidence Estimation for Machine Translation. In COLING04. [find](https://scholar.google.com/scholar?q=J%2E+Blatz%2C+E%2E+Fitzgerald%2C+G%2E+Foster%2C+S%2E+Gandrabur%2C+C%2E+Goutte%2C+A%2E+Kulesza%2C+A%2E+Sanchis%2C+and+N%2E+Ueffing%2E+2004%2E+Confidence+Estimation+for+Machine+Translation%2E+In+COLING04%2E)
4. O. Bojar, C. Buck, C. Callison-Burch, C. Federmann, B. Haddow, P. Koehn, C. Monz, M. Post, R. Soricut, and L. Specia. 2013. Findings of the 2013 Workshop on SMT. In WMT13. [find](https://scholar.google.com/scholar?q=O%2E+Bojar%2C+C%2E+Buck%2C+C%2E+Callison%2DBurch%2C+C%2E+Federmann%2C+B%2E+Haddow%2C+P%2E+Koehn%2C+C%2E+Monz%2C+M%2E+Post%2C+R%2E+Soricut%2C+and+L%2E+Specia%2E+2013%2E+Findings+of+the+2013+Workshop+on+SMT%2E+In+WMT13%2E)
5. O. Bojar, C. Buck, C. Federmann, B. Haddow, P. Koehn, J. Leveling, C. Monz, P. Pecina, M. Post, H. Saint-Amand, R. Soricut, L. Specia, and A. Tamchyna. 2014. Findings of the 2014 Workshop on SMT. In WMT14. [find](https://scholar.google.com/scholar?q=O%2E+Bojar%2C+C%2E+Buck%2C+C%2E+Federmann%2C+B%2E+Haddow%2C+P%2E+Koehn%2C+J%2E+Leveling%2C+C%2E+Monz%2C+P%2E+Pecina%2C+M%2E+Post%2C+H%2E+Saint%2DAmand%2C+R%2E+Soricut%2C+L%2E+Specia%2C+and+A%2E+Tamchyna%2E+2014%2E+Findings+of+the+2014+Workshop+on+SMT%2E+In+W)
6. C. Callison-Burch, P. Koehn, C. Monz, M. Post, R. Soricut, and L. Specia. 2012. Findings of the 2012 Workshop on SMT. In WMT12. [find](https://scholar.google.com/scholar?q=C%2E+Callison%2DBurch%2C+P%2E+Koehn%2C+C%2E+Monz%2C+M%2E+Post%2C+R%2E+Soricut%2C+and+L%2E+Specia%2E+2012%2E+Findings+of+the+2012+Workshop+on+SMT%2E+In+WMT12%2E)
7. J. G. Camargo de Souza, J. González-Rubio, C. Buck, M. Turchi, and M. Negri. 2014. FBK-UPV-UEdin participation in the WMT14 Quality Estimation shared-task. In WMT14. [find](https://scholar.google.com/scholar?q=J%2E+G%2E+Camargo+de+Souza%2C+J%2E+Gonza%CC%81lez%2DRubio%2C+C%2E+Buck%2C+M%2E+Turchi%2C+and+M%2E+Negri%2E+2014%2E+FBK%2DUPV%2DUEdin+participation+in+the+WMT14+Quality+Estimation+shared%2Dtask%2E+In+WMT14%2E)
8. N. Q. Luong, L. Besacier, and B. Lecouteux. 2014. LIG System for Word Level QE task. In WMT14. [find](https://scholar.google.com/scholar?q=N%2E+Q%2E+Luong%2C+L%2E+Besacier%2C+and+B%2E+Lecouteux%2E+2014%2E+LIG+System+for+Word+Level+QE+task%2E+In+WMT14%2E)
9. N. Okazaki. 2007. CRFsuite: a fast implementation of Conditional Random Fields. [http://www.chokkan.org/software/crfsuite/](http://www.chokkan.org/software/crfsuite/). F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12. [↗](http://www.chokkan.org/software/crfsuite/)
10. S. Raybaud, D. Langlois, and K. Smali. 2011. This sentence is wrong. Detecting errors in machine-translated sentences. Machine Translation, 25(1). [find](https://scholar.google.com/scholar?q=S%2E+Raybaud%2C+D%2E+Langlois%2C+and+K%2E+Smali%2E+2011%2E+This+sentence+is+wrong%2E+Detecting+errors+in+machine%2Dtranslated+sentences%2E+Machine+Translation%2C+25%281%29%2E)
11. C. Scarton and L. Specia. 2014. Document-level translation quality estimation: exploring discourse and pseudo-references. In EAMT14. [find](https://scholar.google.com/scholar?q=C%2E+Scarton+and+L%2E+Specia%2E+2014%2E+Document%2Dlevel+translation+quality+estimation%3A+exploring+discourse+and+pseudo%2Dreferences%2E+In+EAMT14%2E)
12. C. Scarton, M. Zampieri, M. Vela, J. van Genabith, and L. Specia. 2015. Searching for Context: a Study on Document-Level Labels for Translation Quality Estimation. In EAMT15. [find](https://scholar.google.com/scholar?q=C%2E+Scarton%2C+M%2E+Zampieri%2C+M%2E+Vela%2C+J%2E+van+Genabith%2C+and+L%2E+Specia%2E+2015%2E+Searching+for+Context%3A+a+Study+on+Document%2DLevel+Labels+for+Translation+Quality+Estimation%2E+In+EAMT15%2E)
13. K. Shah and L. Specia. 2014. Quality estimation for translation selection. In EAMT14. [find](https://scholar.google.com/scholar?q=K%2E+Shah+and+L%2E+Specia%2E+2014%2E+Quality+estimation+for+translation+selection%2E+In+EAMT14%2E)
14. R. Soricut and A. Echihabi. 2010. Trustrank: Inducing trust in automatic translations via ranking. In ACL10. [find](https://scholar.google.com/scholar?q=R%2E+Soricut+and+A%2E+Echihabi%2E+2010%2E+Trustrank%3A+Inducing+trust+in+automatic+translations+via+ranking%2E+In+ACL10%2E)
15. L. Specia, K. Shah, J. G. C. de Souza, and T. Cohn. 2013. Quest - a translation quality estimation framework. In ACL13. [find](https://scholar.google.com/scholar?q=L%2E+Specia%2C+K%2E+Shah%2C+J%2E+G%2E+C%2E+de+Souza%2C+and+T%2E+Cohn%2E+2013%2E+Quest+%2D+a+translation+quality+estimation+framework%2E+In+ACL13%2E) [our card](https://md.research.elamur.ai/free/papers/specia-2013-quest-translation-quality-estimation-framework/)
16. L. Specia. 2011. Exploiting objective annotations for measuring translation post-editing effort. In EAMT11. [find](https://scholar.google.com/scholar?q=L%2E+Specia%2E+2011%2E+Exploiting+objective+annotations+for+measuring+translation+post%2Dediting+effort%2E+In+EAMT11%2E)
17. G. Wisniewski, N. Pcheux, A. Allauzen, and F. Yvon. 2014. LIMSI Submission for WMT’14 QE Task. In WMT14. [find](https://scholar.google.com/scholar?q=G%2E+Wisniewski%2C+N%2E+Pcheux%2C+A%2E+Allauzen%2C+and+F%2E+Yvon%2E+2014%2E+LIMSI+Submission+for+WMT%E2%80%9914+QE+Task%2E+In+WMT14%2E)
18. B. T. M. Wong and C. Kit. 2012. Extending machine translation evaluation metrics with lexical cohesion to document level. In EMNLP/CONLL. [find](https://scholar.google.com/scholar?q=B%2E+T%2E+M%2E+Wong+and+C%2E+Kit%2E+2012%2E+Extending+machine+translation+evaluation+metrics+with+lexical+cohesion+to+document+level%2E+In+EMNLP%2FCONLL%2E)
19. D. Xiong, M. Zhang, and H. Li. 2010. Error detection for SMT using linguistic features. In ACL10. [find](https://scholar.google.com/scholar?q=D%2E+Xiong%2C+M%2E+Zhang%2C+and+H%2E+Li%2E+2010%2E+Error+detection+for+SMT+using+linguistic+features%2E+In+ACL10%2E)