--- title: "OpenKiwi: An Open Source Framework for Quality — Kepler 2019" description: "Translation quality estimation (QE) predicts how good a machine translation is without a reference translation." authors: ["Fabio Kepler", "Jonay Trénous", "Marcos Treviso", "Miguel Vera", "André F. T. Martins"] year: 2019 venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations" source: "https://aclanthology.org/P19-3020/" doi: "10.18653/v1/p19-3020" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/kepler-2019-openkiwi-open-source-framework-quality/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/kepler-2019-openkiwi-open-source-framework-quality/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2019](https://md.research.elamur.ai/free/papers/2019/page/3/) / OpenKiwi: An Open Source Framework for Quality Estimation # OpenKiwi: An Open Source Framework for Quality Estimation **Authors:** Fabio Kepler, Jonay Trénous, [Marcos Treviso](https://md.research.elamur.ai/free/authors/marcos-treviso/), Miguel Vera, [André F. T. Martins](https://md.research.elamur.ai/free/authors/andre-f-t-martins/) · **Year:** 2019 · **Venue:** Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/P19-3020/) · [PDF](https://aclanthology.org/P19-3020.pdf) · [DOI 10.18653/v1/p19-3020](https://doi.org/10.18653/v1/p19-3020) · [OpenAlex](https://openalex.org/W2917862238) **Citations:** 112 citations (openalex-snapshot · 2026-10-14) · [103](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Translation quality estimation (QE) predicts how good a machine translation is without a reference translation. It works at the word level, tagging target words, gaps and source words as OK or BAD, and at the sentence level, for example predicting HTER. Strong QE systems had become hard to reproduce because they rely on ensembles, intricate architectures or poorly documented pre-training, and existing open toolkits trailed the WMT leaders. The authors release OpenKiwi, a PyTorch framework that reimplements the top WMT 2015–18 QE systems under one roof: QUETCH, NuQE, Predictor-Estimator and a stacked linear ensemble. It can be trained from YAML configs on the command line or imported as a Python package. On the WMT 2018 English-German SMT and NMT data, averaged and stacked combinations of its models give the best word-level scores reported. They also beat deepQuest everywhere and come close to the strongest sentence-level systems. The framework matters because it turns a scattered body of research into a shared, reproducible baseline. It went on to serve as the official WMT 2019 QE baseline and as the base for that year's winning systems. ## Key points - Implements four systems: - QUETCH, a window-based MLP. - NuQE, feed-forward layers plus a bi-directional GRU, trained only on shared-task data. - Predictor-Estimator, a predictor pre-trained on large parallel data that feeds an estimator, with multi-task HTER prediction. - A stacked ensemble with a feature-based linear model. - Benchmark data from WMT 2018 En-De, each split with 1,000 development triplets: - SMT: 26,273 training triplets. - NMT: 13,442 training triplets. - Extra resources: - 526,368 artificial triplets built through round-trip translation. - 3,396,364 in-domain parallel sentences, used to pre-train the predictor. - Test-set results: - SMT: word-level F1-mult of 62.70 (MT), 52.14 (gaps) and 48.88 (source), with Pearson r of 71.08. QE Brain reached 73.97 on r. - NMT: MT-tag F1-mult of 44.77. - Engineering for reproducibility: - Poetry for dependency management. - Tests covering about 80% of the code. - Optional MLflow tracking. - Affero GPL license. ## Interesting ideas - **Using APE output as a labeler:** an automatic post-editing model produces pseudo post-edits, which are then turned into word tags and HTER with the same tool (TERCOM) used for the gold labels. This approach could apply to any task where labels are derived from corrected outputs. - **Two ways to combine models:** simple averaging of model outputs is compared with stacking them as features in a linear model. Stacking was the stronger option for MT-word tags, which suggests a cheap linear layer over diverse neural systems is worth trying before building larger ensembles. - **Pre-training on related data:** the predictor first learns to reconstruct target tokens from large parallel corpora, and the estimator is then trained on scarce QE labels. This is a transferable recipe for low-resource labeled tasks that have plentiful raw parallel or monolingual text. - **Synthetic data recipe:** in-domain sentences are selected by cross-entropy filtering, round-trip translated, and stratified-sampled to create artificial triplets. The small genuine dataset is oversampled (×20) when mixed in, so it is not swamped by the synthetic data. - **Fixed pipeline, config-driven experiments:** data handling, training loop and evaluation are fixed, and experiments vary only through config files or a single model class. Combined with run tracking, this makes comparisons across architectures fair and easy to repeat. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/) Topics: [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/) ## Cited by Cited by 103 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 90 not yet. - [MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset](https://md.research.elamur.ai/free/papers/fomicheva-2022-mlqe-pe-multilingual-quality-estimation/) - [Findings of the WMT 2019 Shared Tasks on Quality Estimation](https://md.research.elamur.ai/free/papers/fonseca-2019-findings-wmt-2019-shared-tasks/) - [Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain](https://md.research.elamur.ai/free/papers/freitag-2021-results-wmt21-metrics-shared-task/) - [KoBE: Knowledge-Based Machine Translation Evaluation](https://md.research.elamur.ai/free/papers/gekhman-2020-kobe-knowledge-based-machine-translation/) - [Results of the WMT20 Metrics Shared Task](https://md.research.elamur.ai/free/papers/mathur-2020-results-wmt20-metrics-shared-task/) - [TransQuest: Translation Quality Estimation with Cross-lingual Transformers](https://md.research.elamur.ai/free/papers/ranasinghe-2020-transquest-translation-quality-estimation-cross/) - [Are References Really Needed? Unbabel-IST 2021 Submission for the Metrics Shared Task](https://md.research.elamur.ai/free/papers/rei-2021-are-references-really-needed-unbabel/) - [CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task](https://md.research.elamur.ai/free/papers/rei-2022-cometkiwi-ist-unbabel-2022-submission/) - [Findings of the WMT 2020 Shared Task on Quality Estimation](https://md.research.elamur.ai/free/papers/specia-2020-findings-wmt-2020-shared-task-2020-wmt-1-79/) - [Findings of the WMT 2021 Shared Task on Quality Estimation](https://md.research.elamur.ai/free/papers/specia-2021-findings-wmt-2021-shared-task/) - [Smarter edits? Post-editing with error highlights and translation suggestions](https://md.research.elamur.ai/free/papers/tellingen-2026-smarter-edits-post-editing-error/) - [UniTE: Unified Translation Evaluation](https://md.research.elamur.ai/free/papers/wan-2022-unite-unified-translation-evaluation/) - [Findings of the WMT 2022 Shared Task on Quality Estimation](https://md.research.elamur.ai/free/papers/zerva-2022-findings-wmt-2022-shared-task/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{kepler2019openkiwi, title = {{OpenKiwi: An Open Source Framework for Quality Estimation}}, author = {Kepler, Fabio and Trénous, Jonay and Treviso, Marcos and Vera, Miguel and Martins, André F. T.}, booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations}, year = {2019}, doi = {10.18653/v1/p19-3020}, url = {https://doi.org/10.18653/v1/p19-3020} } ``` - [← previous OpenDialKG: Explainable Conversational Reasoning with Attention-based Walks over Knowledge Graphs](https://md.research.elamur.ai/free/papers/moon-2019-opendialkg-explainable-conversational-reasoning-attention/) - [next → Optuna: A Next-generation Hyperparameter Optimization Framework](https://md.research.elamur.ai/free/papers/akiba-2019-optuna-next-generation-hyperparameter-optimization/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/papers/2019/page/3/) ## References 12 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648. [find](https://scholar.google.com/scholar?q=Chris+Dyer%2C+Victor+Chahuneau%2C+and+Noah+A+Smith%2E+2013%2E+A+simple%2C+fast%2C+and+effective+reparameterization+of+IBM+model+2%2E+In+Proceedings+of+the+2013+Conference+of+the+North+American+Chapter+of+the+Associ) 2. Lucia Specia, Frédéric Blain, Varvara Logacheva, Ramón Astudillo, and André F. T. Martins. 2018a. Findings of the wmt 2018 shared task on quality estimation. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, pages 702–722, Belgium, Brussels. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Lucia+Specia%2C+Fre%CC%81de%CC%81ric+Blain%2C+Varvara+Logacheva%2C+Ramo%CC%81n+Astudillo%2C+and+Andre%CC%81+F%2E+T%2E+Martins%2E+2018a%2E+Findings+of+the+wmt+2018+shared+task+on+quality+estimation%2E+In+Proceedings+of+the+Third+Conference) [our card](https://md.research.elamur.ai/free/papers/specia-2018-findings-wmt-2018-shared-task/) 3. Julia Ive, Frédéric Blain, and Lucia Specia. 2018. DeepQuest: a framework for neural-based Quality Estimation. In International Conference on Computational Linguistics (COLING). [find](https://scholar.google.com/scholar?q=Julia+Ive%2C+Fre%CC%81de%CC%81ric+Blain%2C+and+Lucia+Specia%2E+2018%2E+DeepQuest%3A+a+framework+for+neural%2Dbased+Quality+Estimation%2E+In+International+Conference+on+Computational+Linguistics+%28COLING%29%2E) 4. Lucia Specia, Gustavo Paetzold, and Carolina Scarton. 2015. Multi-level translation quality prediction with quest++. In ACL-IJCNLP 2015 System Demonstrations, pages 115–120, Beijing, China. [find](https://scholar.google.com/scholar?q=Lucia+Specia%2C+Gustavo+Paetzold%2C+and+Carolina+Scarton%2E+2015%2E+Multi%2Dlevel+translation+quality+prediction+with+quest%2B%2B%2E+In+ACL%2DIJCNLP+2015+System+Demonstrations%2C+pages+115%E2%80%93120%2C+Beijing%2C+China%2E) [our card](https://md.research.elamur.ai/free/papers/specia-2015-multi-level-translation-quality-prediction/) 5. Fábio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, António Góis, M. Amin Farajian, António V. Lopes, and André F. T. Martins. 2019. Unbabel’s Participation in the WMT19 Translation Quality Estimation Shared Task. In Conference on Machine Translation (WMT). [find](https://scholar.google.com/scholar?q=Fa%CC%81bio+Kepler%2C+Jonay+Tre%CC%81nous%2C+Marcos+Treviso%2C+Miguel+Vera%2C+Anto%CC%81nio+Go%CC%81is%2C+M%2E+Amin+Farajian%2C+Anto%CC%81nio+V%2E+Lopes%2C+and+Andre%CC%81+F%2E+T%2E+Martins%2E+2019%2E+Unbabel%E2%80%99s+Participation+in+the+WMT19+Translation+Qualit) 6. Lucia Specia, Carolina Scarton, and Gustavo Henrique Paetzold. 2018b. Quality estimation for machine translation. Synthesis Lectures on Human Language Technologies, 11(1):1–162. [↗](https://doi.org/10.1007/978-3-031-02168-8) 7. Hyun Kim, Jong-Hyeok Lee, and Seung-Hoon Na. 2017. Predictor-Estimator using Multilevel Task Learning with Stack Propagation for Neural Quality Estimation. In Conference on Machine Translation (WMT). [find](https://scholar.google.com/scholar?q=Hyun+Kim%2C+Jong%2DHyeok+Lee%2C+and+Seung%2DHoon+Na%2E+2017%2E+Predictor%2DEstimator+using+Multilevel+Task+Learning+with+Stack+Propagation+for+Neural+Quality+Estimation%2E+In+Conference+on+Machine+Translation+%28WMT%29%2E) [our card](https://md.research.elamur.ai/free/papers/kim-2017-predictor-estimator-multilevel-task-learning/) 8. Jiayi Wang, Kai Fan, Bo Li, Fengming Zhou, Boxing Chen, Yangbin Shi, and Luo Si. 2018. Alibaba Submission for WMT18 Quality Estimation Task. In Conference on Machine Translation (WMT). [find](https://scholar.google.com/scholar?q=Jiayi+Wang%2C+Kai+Fan%2C+Bo+Li%2C+Fengming+Zhou%2C+Boxing+Chen%2C+Yangbin+Shi%2C+and+Luo+Si%2E+2018%2E+Alibaba+Submission+for+WMT18+Quality+Estimation+Task%2E+In+Conference+on+Machine+Translation+%28WMT%29%2E) 9. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. arXiv preprint arXiv:1701.02810. [↗](https://arxiv.org/abs/1701.02810) [our card](https://md.research.elamur.ai/free/papers/klein-2017-opennmt-open-source-toolkit-neural/) 10. Julia Kreutzer, Shigehiko Schamoni, and Stefan Riezler. 2015. QUality Estimation from ScraTCH (QUETCH): Deep Learning for Word-level Translation Quality Estimation. In Workshop on Machine Translation (WMT). [find](https://scholar.google.com/scholar?q=Julia+Kreutzer%2C+Shigehiko+Schamoni%2C+and+Stefan+Riezler%2E+2015%2E+QUality+Estimation+from+ScraTCH+%28QUETCH%29%3A+Deep+Learning+for+Word%2Dlevel+Translation+Quality+Estimation%2E+In+Workshop+on+Machine+Translation+) 11. Varvara Logacheva, Chris Hokamp, and Lucia Specia. 2016. Marmot: A toolkit for translation quality estimation at the word level. In LREC. [find](https://scholar.google.com/scholar?q=Varvara+Logacheva%2C+Chris+Hokamp%2C+and+Lucia+Specia%2E+2016%2E+Marmot%3A+A+toolkit+for+translation+quality+estimation+at+the+word+level%2E+In+LREC%2E) 12. André F. T. Martins, Ramon Astudillo, Chris Hokamp, and Fábio Kepler. 2016. Unbabel’s Participation in the WMT16 Word-Level Translation Quality Estimation Shared Task. In Conference on Machine Translation (WMT). 7 [http://www.statmt.org/wmt19/](http://www.statmt.org/wmt19/) qe-results.html [↗](http://www.statmt.org/wmt19/)