---
title: "QuEst - A translation quality estimation — Specia 2013"
description: "The paper presents QuEst, an open-source toolkit for machine translation quality estimation (QE), meaning predicting how good a translated segment is…"
authors: ["Lucia Specia", "Kashif Shah", "Jose G.C. de Souza", "Trevor Cohn"]
year: 2013
venue: "Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations"
source: "https://aclanthology.org/P13-4014/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/specia-2013-quest-translation-quality-estimation-framework/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/specia-2013-quest-translation-quality-estimation-framework/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2013](https://md.research.elamur.ai/free/papers/2013/) / QuEst - A translation quality estimation framework

# QuEst - A translation quality estimation framework

**Authors:** [Lucia Specia](https://md.research.elamur.ai/free/authors/lucia-specia/), Kashif Shah, [Jose G.C. de Souza](https://md.research.elamur.ai/free/authors/jose-g-c-de-souza/), [Trevor Cohn](https://md.research.elamur.ai/free/authors/trevor-cohn/) · **Year:** 2013 · **Venue:** Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/P13-4014/) · [PDF](https://aclanthology.org/P13-4014.pdf) · [OpenAlex](https://openalex.org/W2164984707)

**Citations:** 175 citations (openalex-snapshot · 2026-10-14) · [106](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
The paper presents QuEst, an open-source toolkit for machine translation quality estimation (QE), meaning predicting how good a translated segment is when no human reference is available. Building QE models usually requires a lot of manual feature engineering. QuEst addresses this by bundling many feature extractors with a learning layer. A Java module computes sentence-level indicators from four sources: the source text, the MT output, external resources such as corpora, language models and GIZA++ alignments, and optionally the internals of the MT system. A separate Python module connects these features to scikit-learn and GPy for regression, classification, feature selection and grid search.

The authors tested it on nine datasets drawn from WMT12, EAMT11, EAMT09 and GALE11, covering en-es, fr-en and ar-en. Adding features to the standard 17-feature baseline usually lowered error. Feature selection, especially Gaussian Process–based ranking, often gave the best results. Combining system-dependent and system-independent features produced significantly lower errors. QuEst matters because it lets researchers concentrate on the learning side of QE instead of rebuilding features.

## Key points
- QuEst has two independent parts:
  - a Java feature-extraction module with abstract classes for features, resources and preprocessing, controlled by configuration files;
  - a Python ML module built on scikit-learn (SVR, SVC, Lasso) and GPy (Gaussian Processes).
- Features are grouped into four families: complexity (source side), fluency (target side), adequacy (source–target comparison) and confidence (MT-system internals).
- Black-box (BB) features number 80–123 depending on the language pair. Glass-box (GB) features number 39–48 and rely on Moses-like outputs such as n-best lists and search-graph statistics. Some word-level features are also included.
- Four system types were compared: baseline (BL, 17 features), all features (AF), Randomised Lasso selection (FS(RL)), and GP selection (FS(GP), top 17). Errors were measured with MAE and RMSE.
- On WMT12, MAE dropped from 0.6802 (BL) to 0.6537 with FS(GP) using BB features, and to 0.6501 with FS(GP) using BB+GB features. GB features alone performed worse than BB features.

## Interesting ideas
- **Black-box vs. glass-box split.** Separating features that need access to the MT system from those that only look at the text lets the same tool run on outputs from unknown systems. It also allows richer signals to be added when the system is available. The same split could apply to any task that evaluates model outputs.
- **GP relevance for feature ranking.** Per-feature RBF widths (an ARD kernel) in a Gaussian Process serve as importance scores, and a cheaper SVR is then trained on the top-ranked features. This turns a strong but expensive model into a feature selector for a simpler one.
- **Stability-style Lasso selection.** The training data is resampled 1000 times, and a feature survives only when Lasso assigns it a weight other than zero in 25% or more of those fits. This makes selection more robust when features are numerous and correlated, a common situation with small annotated datasets.
- **Fixed feature budget for fair comparison.** Limiting GP selection to the same number of features as the baseline (17) shows the gain comes from choosing better features, not from using more of them. The authors also note that 25–35 features worked best on most tasks.
- **Language model built from n-best lists.** Training a small LM on the system's own n-best translations, then scoring the chosen output against it, measures how consistent the system is with its alternatives. This idea could carry over to confidence estimation in other generation tasks.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/)

## Cited by

Cited by

106 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 99 not yet.

- [SemEval-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Lingual Evaluation](https://md.research.elamur.ai/free/papers/agirre-2016-semeval-2016-task-1-semantic/)
- [Findings of the 2013 Workshop on Statistical Machine Translation](https://md.research.elamur.ai/free/papers/bojar-2013-findings-2013-workshop-statistical-machine/)
- [Findings of the 2014 Workshop on Statistical Machine Translation](https://md.research.elamur.ai/free/papers/bojar-2014-findings-2014-workshop-statistical-machine/)
- [Findings of the 2015 Workshop on Statistical Machine Translation](https://md.research.elamur.ai/free/papers/bojar-2015-findings-2015-workshop-statistical-machine/)
- [Findings of the 2017 Conference on Machine Translation (WMT17)](https://md.research.elamur.ai/free/papers/bojar-2017-findings-2017-conference-machine-translation/)
- [TransQuest: Translation Quality Estimation with Cross-lingual Transformers](https://md.research.elamur.ai/free/papers/ranasinghe-2020-transquest-translation-quality-estimation-cross/)
- [Multi-level Translation Quality Prediction with QuEst++](https://md.research.elamur.ai/free/papers/specia-2015-multi-level-translation-quality-prediction/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{specia2013quest,
  title = {{QuEst - A translation quality estimation framework}},
  author = {Specia, Lucia and Shah, Kashif and Souza, Jose G.C. de and Cohn, Trevor},
  booktitle = {Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations},
  year = {2013},
  url = {https://aclanthology.org/P13-4014/}
}
```

- [← previous Predicting the Presence of Discourse Connectives](https://md.research.elamur.ai/free/papers/patterson-2013-predicting-presence-discourse-connectives/)
- [next → Raising and resolving issues with scalar modifiers](https://md.research.elamur.ai/free/papers/coppock-2013-raising-resolving-issues-scalar-modifiers/)
- [↑ back to papers of 2013](https://md.research.elamur.ai/free/papers/2013/)

## References

11 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. N. Bach, F. Huang, and Y. Al-Onaizan. 2011. Goodness: a method for measuring machine translation confidence. In ACL11, pages 211–219, Portland. [find](https://scholar.google.com/scholar?q=N%2E+Bach%2C+F%2E+Huang%2C+and+Y%2E+Al%2DOnaizan%2E+2011%2E+Goodness%3A+a+method+for+measuring+machine+translation+confidence%2E+In+ACL11%2C+pages+211%E2%80%93219%2C+Portland%2E)
2. J. Blatz, E. Fitzgerald, G. Foster, S. Gandrabur, C. Goutte, A. Kulesza, A. Sanchis, and N. Ueffing. 2004. Confidence Estimation for Machine Translation. In Coling04, pages 315–321, Geneva. [find](https://scholar.google.com/scholar?q=J%2E+Blatz%2C+E%2E+Fitzgerald%2C+G%2E+Foster%2C+S%2E+Gandrabur%2C+C%2E+Goutte%2C+A%2E+Kulesza%2C+A%2E+Sanchis%2C+and+N%2E+Ueffing%2E+2004%2E+Confidence+Estimation+for+Machine+Translation%2E+In+Coling04%2C+pages+315%E2%80%93321%2C+Geneva%2E)
3. C. Callison-Burch, P. Koehn, C. Monz, M. Post, R. Soricut, and L. Specia. 2012. Findings of the 2012 workshop on statistical machine translation. In WMT12, pages 10–51, Montréal. [find](https://scholar.google.com/scholar?q=C%2E+Callison%2DBurch%2C+P%2E+Koehn%2C+C%2E+Monz%2C+M%2E+Post%2C+R%2E+Soricut%2C+and+L%2E+Specia%2E+2012%2E+Findings+of+the+2012+workshop+on+statistical+machine+translation%2E+In+WMT12%2C+pages+10%E2%80%9351%2C+Montre%CC%81al%2E) [our card](https://md.research.elamur.ai/free/papers/burch-2012-findings-2012-workshop-statistical-machine/)
4. Y. He, Y. Ma, J. van Genabith, and A. Way. 2010. Bridging SMT and TM with Translation Recommendation. In ACL10, pages 622–630, Uppsala. [find](https://scholar.google.com/scholar?q=Y%2E+He%2C+Y%2E+Ma%2C+J%2E+van+Genabith%2C+and+A%2E+Way%2E+2010%2E+Bridging+SMT+and+TM+with+Translation+Recommendation%2E+In+ACL10%2C+pages+622%E2%80%93630%2C+Uppsala%2E)
5. N. Meinshausen and P. Bühlmann. 2010. Stability selection. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 72:417–473. [find](https://scholar.google.com/scholar?q=N%2E+Meinshausen+and+P%2E+Bu%CC%88hlmann%2E+2010%2E+Stability+selection%2E+Journal+of+the+Royal+Statistical+Society%3A+Series+B+%28Statistical+Methodology%29%2C+72%3A417%E2%80%93473%2E)
6. K. Papineni, S. Roukos, T. Ward, and W. Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In ACL02, pages 311–318, Philadelphia. [find](https://scholar.google.com/scholar?q=K%2E+Papineni%2C+S%2E+Roukos%2C+T%2E+Ward%2C+and+W%2E+Zhu%2E+2002%2E+BLEU%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+ACL02%2C+pages+311%E2%80%93318%2C+Philadelphia%2E) [our card](https://md.research.elamur.ai/free/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
7. C.E. Rasmussen and C.K.I. Williams. 2006. Gaussian processes for machine learning, volume 1. MIT Press, Cambridge. [find](https://scholar.google.com/scholar?q=C%2EE%2E+Rasmussen+and+C%2EK%2EI%2E+Williams%2E+2006%2E+Gaussian+processes+for+machine+learning%2C+volume+1%2E+MIT+Press%2C+Cambridge%2E)
8. R. Soricut and A. Echihabi. 2010. Trustrank: Inducing trust in automatic translations via ranking. In ACL11, pages 612–621, Uppsala. [find](https://scholar.google.com/scholar?q=R%2E+Soricut+and+A%2E+Echihabi%2E+2010%2E+Trustrank%3A+Inducing+trust+in+automatic+translations+via+ranking%2E+In+ACL11%2C+pages+612%E2%80%93621%2C+Uppsala%2E)
9. L. Specia, M. Turchi, N. Cancedda, M. Dymetman, and N. Cristianini. 2009. Estimating the Sentence-Level Quality of Machine Translation Systems. In EAMT09, pages 28–37, Barcelona. [find](https://scholar.google.com/scholar?q=L%2E+Specia%2C+M%2E+Turchi%2C+N%2E+Cancedda%2C+M%2E+Dymetman%2C+and+N%2E+Cristianini%2E+2009%2E+Estimating+the+Sentence%2DLevel+Quality+of+Machine+Translation+Systems%2E+In+EAMT09%2C+pages+28%E2%80%9337%2C+Barcelona%2E) [our card](https://md.research.elamur.ai/free/papers/specia-2009-estimating-sentence-level-quality-machine/)
10. L. Specia, D. Raj, and M. Turchi. 2010. Machine translation evaluation versus quality estimation. Machine Translation, 24(1):39–50. [↗](https://doi.org/10.1007/s10590-010-9077-2)
11. L. Specia. 2011. Exploiting objective annotations for measuring translation post-editing effort. In EAMT11, pages 73–80, Leuven. [find](https://scholar.google.com/scholar?q=L%2E+Specia%2E+2011%2E+Exploiting+objective+annotations+for+measuring+translation+post%2Dediting+effort%2E+In+EAMT11%2C+pages+73%E2%80%9380%2C+Leuven%2E)