--- title: "Predictor-Estimator using Multilevel Task — Kim 2017" description: "The paper describes a POSTECH/Chonbuk system for the WMT17 shared task on translation quality estimation (QE)." authors: ["Hyun Kim", "Jong-Hyeok Lee", "Seung-Hoon Na"] year: 2017 venue: "Proceedings of the Second Conference on Machine Translation" source: "https://aclanthology.org/W17-4763/" doi: "10.18653/v1/w17-4763" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/kim-2017-predictor-estimator-multilevel-task-learning/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/kim-2017-predictor-estimator-multilevel-task-learning/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2017](https://md.research.elamur.ai/free/papers/2017/page/2/) / Predictor-Estimator using Multilevel Task Learning with Stack Propagation for Neural Quality Estimation # Predictor-Estimator using Multilevel Task Learning with Stack Propagation for Neural Quality Estimation **Authors:** Hyun Kim, Jong-Hyeok Lee, Seung-Hoon Na · **Year:** 2017 · **Venue:** Proceedings of the Second Conference on Machine Translation · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/W17-4763/) · [PDF](https://aclanthology.org/W17-4763.pdf) · [DOI 10.18653/v1/w17-4763](https://doi.org/10.18653/v1/w17-4763) · [OpenAlex](https://openalex.org/W2757980860) **Citations:** 130 citations (openalex-snapshot · 2026-10-14) · [75](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary The paper describes a POSTECH/Chonbuk system for the WMT17 shared task on translation quality estimation (QE). QE predicts how good a machine translation is, without a reference, at the sentence, word, and phrase levels. The system builds on the authors' two-stage Predictor-Estimator. In this design, an attention-based bidirectional RNN word predictor is trained on large parallel corpora (Europarl, common crawl, news commentary, rapid, plus src-pe pairs). It produces QE feature vectors (QEFVs), which a Bi-RNN quality estimator uses. Previously the two stages were trained separately. The authors instead apply stack propagation, alternating updates between the word-prediction and QE objectives so that gradients reach the predictor. They also add multilevel task learning, which shares common network parts across all three QE granularities. The rationale is that every level's labels come from the same HTER-based comparison with post-edits. Single models improve step by step from these additions. Ensembles of 15 instances, varied by dimensionality and data shuffling, took first place in every WMT17 QE subtask for English-German and German-English. This shows the value of jointly training pre-task and task networks. ## Key points - Three variants are compared: the base Predictor-Estimator (Pre&Post-QEFV/Bi-RNN), a single-level Stackprop variant, and a MultiLevel Stackprop variant. The last one was consistently best. For example, sentence-level Pearson's r on the En-De dev set rose from 0.6436 to 0.6785. - For phrase-level QE in the multilevel setup, a word-level architecture is used. A phrase is labeled BAD if any word inside it is tagged BAD. - The main single-model configuration used a 70,000-word vocabulary, 500-dimensional embeddings, 700 hidden units in the predictor, and 100 hidden units in the estimator. - The best submission was PredictorEstimator-Combined-MultiLevel-Ensemble. On the En-De test set it reached Pearson's r 0.6954 (baseline 0.397), word F1-mult 0.5679 (baseline 0.361), and phrase F1-mult 0.5859 (baseline 0.327). - On the De-En test set, the same ensemble scored Pearson's r 0.7280, word F1-mult 0.5347, and phrase F1-mult 0.5611, ranking first in all subtasks. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) ## Cited by Cited by 75 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 66 not yet. - [Findings of the 2017 Conference on Machine Translation (WMT17)](https://md.research.elamur.ai/free/papers/bojar-2017-findings-2017-conference-machine-translation/) - [Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments](https://md.research.elamur.ai/free/papers/fomicheva-2019-taking-mt-evaluation-metrics-extremes/) - [MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset](https://md.research.elamur.ai/free/papers/fomicheva-2022-mlqe-pe-multilingual-quality-estimation/) - [OpenKiwi: An Open Source Framework for Quality Estimation](https://md.research.elamur.ai/free/papers/kepler-2019-openkiwi-open-source-framework-quality/) - [TransQuest: Translation Quality Estimation with Cross-lingual Transformers](https://md.research.elamur.ai/free/papers/ranasinghe-2020-transquest-translation-quality-estimation-cross/) - [Findings of the WMT 2018 Shared Task on Quality Estimation](https://md.research.elamur.ai/free/papers/specia-2018-findings-wmt-2018-shared-task/) - [Findings of the WMT 2020 Shared Task on Quality Estimation](https://md.research.elamur.ai/free/papers/specia-2020-findings-wmt-2020-shared-task-2020-wmt-1-79/) - [Findings of the WMT 2021 Shared Task on Quality Estimation](https://md.research.elamur.ai/free/papers/specia-2021-findings-wmt-2021-shared-task/) - [Findings of the WMT 2022 Shared Task on Quality Estimation](https://md.research.elamur.ai/free/papers/zerva-2022-findings-wmt-2022-shared-task/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{kim2017predictor, title = {{Predictor-Estimator using Multilevel Task Learning with Stack Propagation for Neural Quality Estimation}}, author = {Kim, Hyun and Lee, Jong-Hyeok and Na, Seung-Hoon}, booktitle = {Proceedings of the Second Conference on Machine Translation}, year = {2017}, doi = {10.18653/v1/w17-4763}, url = {https://doi.org/10.18653/v1/w17-4763} } ``` - [← previous PP Attachment: Where do We Stand?](https://md.research.elamur.ai/free/papers/onto-2017-kok-pp-attachment-where-do-we/) - [next → Prepositional Phrase Attachment over Word Embedding Products](https://md.research.elamur.ai/free/papers/lingbulk-2017-madhyastha-prepositional-phrase-attachment-over-word/) - [↑ back to papers of 2017](https://md.research.elamur.ai/free/papers/2017/page/2/) ## References 2 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR 2015. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+In+ICLR+2015%2E) [our card](https://md.research.elamur.ai/free/papers/bahdanau-2014-neural-machine-translation-jointly-learning/) 2. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Langua… [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2C+Bart+van+Merrienboer%2C+Caglar+Gulcehre%2C+Dzmitry+Bahdanau%2C+Fethi+Bougares%2C+Holger+Schwenk%2C+and+Yoshua+Bengio%2E+2014%2E+Learning+phrase+representations+using+rnn+encoder%E2%80%93decoder+for+statistic)