--- title: "Validation of an Automatic Metric for the — Werlen 2017" description: "Scoring pronoun translation in MT is difficult because the correct output depends on how the antecedent was translated and on non-referential uses." authors: ["Lesly Miculicich Werlen", "Andrei Popescu-Belis"] year: 2017 venue: "Proceedings of the Third Workshop on Discourse in Machine Translation" source: "https://aclanthology.org/W17-4802/" doi: "10.18653/v1/w17-4802" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/werlen-2017-validation-automatic-metric-accuracy-pronoun/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/werlen-2017-validation-automatic-metric-accuracy-pronoun/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2017](https://md.research.elamur.ai/free/papers/2017/page/3/) / Validation of an Automatic Metric for the Accuracy of Pronoun Translation (APT) # Validation of an Automatic Metric for the Accuracy of Pronoun Translation (APT) **Authors:** [Lesly Miculicich Werlen](https://md.research.elamur.ai/free/authors/lesly-miculicich-werlen/), [Andrei Popescu-Belis](https://md.research.elamur.ai/free/authors/andrei-popescu-belis/) · **Year:** 2017 · **Venue:** Proceedings of the Third Workshop on Discourse in Machine Translation · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/W17-4802/) · [PDF](https://aclanthology.org/W17-4802.pdf) · [DOI 10.18653/v1/w17-4802](https://doi.org/10.18653/v1/w17-4802) · [OpenAlex](https://openalex.org/W2582446770) **Citations:** 49 citations (openalex-snapshot · 2026-10-14) · [37](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Scoring pronoun translation in MT is difficult because the correct output depends on how the antecedent was translated and on non-referential uses. For this reason, the DiscoMT 2015 shared task used human gap-filling judgments, which are expensive and cannot be reused for new outputs. The authors propose APT, a reference-based automatic score for English *it*/*they* translated into French. APT works in three steps: - It word-aligns source, reference and candidate with GIZA++ (merged via Moses' grow-diag-final). - It repairs pronoun alignments with heuristics that search the target window around neighbouring words' alignments. - It places each source pronoun into one of six cases: identical, equivalent, different, absent in candidate, absent in reference, or absent in both. Weighted case counts give the share of correctly translated pronouns. On the seven DiscoMT 2015 systems, APT correlates with human scores at 0.993–0.999 Pearson and up to 1.000 Spearman. This beats BLEU, METEOR and the pronoun-specific DiscoMT metrics, showing that cheap, repeatable reference-based pronoun evaluation is viable. ## Key points - Validation data: DiscoMT 2015 English→French task with 2,093 source sentences. Humans judged a sample of 210 pronouns for each of six submitted systems plus a baseline. - Alignment check on 100 WIT3 (TED Talks) sentences: GIZA++ missed 19 pronoun alignments and got 4 wrong. The heuristics fixed 22 of these 23 errors. - Comparison metrics (BLEU, METEOR, AutoP/AutoR/AutoF) reached only 0.972–0.986 Pearson. - Weight tuning for "equivalent" pronouns (case 2) and both-missing pronouns (case 6) changed correlations only marginally. With heuristic alignment, w₂ = 0.5 performed slightly best. - Error analysis: 84% of identical-to-reference pronouns were judged correct by humans. However, 26% of "different" ones and 53% of reference-missing ones were also correct, revealing legitimate variation the metric cannot capture. - The code is released at github.com/idiap/APT. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) ## Cited by Cited by 37 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 31 not yet. - [BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation](https://md.research.elamur.ai/free/papers/jiang-2022-blonde-automatic-evaluation-metric-document/) - [文档级神经机器翻译综述](https://md.research.elamur.ai/free/papers/lv-2024-document-level-neural/) - [Document-Level Neural Machine Translation with Hierarchical Attention Networks](https://md.research.elamur.ai/free/papers/miculicich-2018-document-level-neural-machine-translation/) - [Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric into a Document-Level Metric](https://md.research.elamur.ai/free/papers/vernikos-2022-embarrassingly-easy-document-level-mt/) - [Context-Aware Neural Machine Translation Learns Anaphora Resolution](https://md.research.elamur.ai/free/papers/voita-2018-context-aware-neural-machine-translation/) - [GuoFeng: A Benchmark for Zero Pronoun Recovery and Translation](https://md.research.elamur.ai/free/papers/xu-2022-guofeng-benchmark-zero-pronoun-recovery/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{werlen2017validation, title = {{Validation of an Automatic Metric for the Accuracy of Pronoun Translation (APT)}}, author = {Werlen, Lesly Miculicich and Popescu-Belis, Andrei}, booktitle = {Proceedings of the Third Workshop on Discourse in Machine Translation}, year = {2017}, doi = {10.18653/v1/w17-4802}, url = {https://doi.org/10.18653/v1/w17-4802} } ``` - [← previous Using the Output Embedding to Improve Language Models](https://md.research.elamur.ai/free/papers/press-2017-output-embedding-improve-language-models/) - [next → Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://md.research.elamur.ai/free/papers/anderson-2017-vision-language-navigation-interpreting-visually/) - [↑ back to papers of 2017](https://md.research.elamur.ai/free/papers/2017/page/3/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. WIT 3 : Web inventory of transcribed and translated talks. In Proceedings of the 16 th Conference of the European Association for Machine Translation (EAMT). Trento, Italy, pages 261–268. [find](https://scholar.google.com/scholar?q=Mauro+Cettolo%2C+Christian+Girardi%2C+and+Marcello+Federico%2E+2012%2E+WIT+3+%3A+Web+inventory+of+transcribed+and+translated+talks%2E+In+Proceedings+of+the+16+th+Conference+of+the+European+Association+for+Machine)