--- title: "Part of Speech and Universal Dependency — Rafaeli 2021" description: "The paper asks how English→Arabic machine translation can be probed for weaknesses tied to syntax, instead of relying only on corpus-level scores such as…" authors: ["Ofek Rafaeli", "Omri Abend", "Leshem Choshen", "Dmitry Nikolaev"] year: 2021 venue: "arXiv:2106.00745" source: "https://arxiv.org/abs/2106.00745" doi: "10.48550/arxiv.2106.00745" arxiv: "2106.00745" licence: "CC0 1.0 (arXiv)" licence_url: "http://creativecommons.org/publicdomain/zero/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/rafaeli-2021-part-speech-universal-dependency-effects/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rafaeli-2021-part-speech-universal-dependency-effects/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / Part of Speech and Universal Dependency effects on English Arabic Machine Translation # Part of Speech and Universal Dependency effects on English Arabic Machine Translation **Authors:** Ofek Rafaeli, [Omri Abend](https://md.research.elamur.ai/free/en/authors/omri-abend/), [Leshem Choshen](https://md.research.elamur.ai/free/en/authors/leshem-choshen/), [Dmitry Nikolaev](https://md.research.elamur.ai/free/en/authors/dmitry-nikolaev/) · **Year:** 2021 · **Venue:** arXiv:2106.00745 · **Licence:** [CC0 1.0 (arXiv)](http://creativecommons.org/publicdomain/zero/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2106.00745) · [PDF](https://arxiv.org/pdf/2106.00745) · [DOI 10.48550/arxiv.2106.00745](https://doi.org/10.48550/arxiv.2106.00745) · [arXiv 2106.00745](https://arxiv.org/abs/2106.00745) · [OpenAlex](https://openalex.org/W3170057700) **Citations:** 2 citations (openalex-snapshot · 2026-10-14) · [1](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary The paper asks how English→Arabic machine translation can be probed for weaknesses tied to syntax, instead of relying only on corpus-level scores such as BLEU. It does not release a fixed test suite. Instead, it looks for a reusable rule that pulls "challenge sets" automatically out of annotated parallel data. Two sources guide which phenomena to target: - **Corpus divergences.** The author hand-aligned the English–Arabic PUD corpus from Universal Dependencies and compared POS tags and dependency paths across the pair, summarizing the mismatches as confusion matrices. - **Known Arabic constructions.** These are well-documented structures with no stable English counterpart, such as the cognate (absolute) object and verb–preposition idioms. A handful of hand-picked sentences show Google Translate mishandling such cases. The main finding is negative: sentences chosen by the tag patterns from manual alignment were not harder for Google Translate than an ordinary, larger sample. The work matters as a template for cheap, syntax-driven evaluation. It is also a candid record of where low-effort extraction falls short. ## Key points - The data is the English–Arabic PUD treebank: 1000 one-to-one aligned sentences from news and Wikipedia. Of these, 750 were originally English and 250 came from German, French, Italian or Spanish via English. Professional translators produced the Arabic, Google annotated it, and the UD community converted it to UD v2. - Manual alignment produced POS and dependency-path confusion matrices, shipped as appendix files: en-ar_pos_cm.xlsx, en-ar_pos_cm_percent.xlsx, en-ar_path_cm_no_subcats.xlsx, compute_confusion_matrices.py and challenge_set.xlsx. - The cognate object could not be extracted automatically for two reasons: - The Arabic UD data usually fills the root field only on the verb, not on the advmod. - Available root extractors were about 75% accurate, struggled with the relevant suffix, or were too slow or unpublished. - One illustrative failure: an Arabic idiom meaning "attacked the commander" came out of Google Translate as "came out on the commander". Verb–preposition distance was left to Choshen & Abend (2019). - Proposed next steps include: - the Arabic→English direction; - semantic metrics instead of BLEU; - checking whether source UD/POS tags reappear in the target. ## Interesting ideas - Aligning a parallel treebank word by word and tabulating POS and dependency-path confusions maps where two languages encode the same content differently. This turns a hunch about "hard constructions" into counts. — For Mova (back): The same matrices over aligned English–target UD trees would show which constructions need explicit restructuring when translating through the meta-world. - The paper ships a rule that extracts challenge sentences, not a frozen list, so anyone can regenerate fresh sets from new data at low human cost. — For Mova (learn): Express each hard construction as a UD tree query and regenerate regression sets from silver data after every retraining, so tests never go stale. - The challenge set was judged against the system's own performance on a larger ordinary sample, and that comparison exposed that the set was not actually hard. — For Mova (learn): Every construction-specific test should report its score beside a random-sample control, so a "hard" set that is no harder than baseline gets flagged. - In the Hebrew sentiment-verb example, the syntactic subject is not the one who feels the emotion. A translation system that maps subject to agent swaps who likes whom. — For Mova (frames): Seed tests where VerbNet role assignment must override the naive subject=agent mapping, such as experiencer/stimulus alternations. - Extraction of the cognate object failed because a needed morphological feature (root) was sparsely annotated. Auditing feature coverage should come before designing a query. — For Mova (tokens): Measure how often lemma and morphological fields are filled per POS in each treebank before building rules or features that depend on them. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) Topics: [Part-of-speech tagging](https://md.research.elamur.ai/free/en/topics/part-of-speech-tagging/) · [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) ## Cited by Cited by 1 paper of our library cites it (of the 140884 whose bibliographies we read); each has a page here. - [On the Relation between Syntactic Divergence and Zero-Shot Performance](https://md.research.elamur.ai/free/en/papers/arviv-2021-relation-between-syntactic-divergence-zero/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{rafaeli2021part, title = {{Part of Speech and Universal Dependency effects on English Arabic Machine Translation}}, author = {Rafaeli, Ofek and Abend, Omri and Choshen, Leshem and Nikolaev, Dmitry}, year = {2021}, doi = {10.48550/arxiv.2106.00745}, eprint = {2106.00745}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2106.00745} } ``` - [← previous ParaShoot: A Hebrew Question Answering Dataset](https://md.research.elamur.ai/free/en/papers/ud-2021-keren-parashoot-hebrew-question-answering-dataset/) - [next → Perceiver: General Perception with Iterative Attention](https://md.research.elamur.ai/free/en/papers/jaegle-2021-perceiver-general-perception-iterative-attention/) - [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Przepiórkowski, Adam, and Agnieszka Patejuk. "Arguments and adjuncts in Universal Dependencies." Proceedings of the 27th International Conference on Computational Linguistics. 2018. - Choshen, Leshem, and Omri Abend. "Automatically Extracting Challenge Sets for Non -local Phenomena Neural Machine Translation." arXiv preprint arXiv:1909.06814 (2019). - Nikolaev, D., Arviv, O., Karidi, T., Kenneth, N., Mitnik, V., Saeboe, L. M., & Abend, O. (2020). Fine -Grained Analysis of Cross -Linguistic Syntactic Divergences. arXiv preprint arXiv:2005.03436. - Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. 2010. Automatic evaluation of translation quality for distant language pairs. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 944–952. Association for Computational Linguistics. - Pierre Isabelle, Colin Cherry, and George F. Foster. 2017. A challenge set approach to evaluating machine translation. In EMNLP - Rico Sennrich. 2016. How grammatical is characterlevel neural machine translation. Assessing MT quality with contrastive translation pairs. CoRR, abs/1612.04629. - Koehn, Philipp, and Rebecca Knowles. "Six challenges for neural machine translation." arXiv preprint arXiv:1706.03872 (2017). - Sulem, Elior, Omri Abend, and Ari Rappoport. "Bleu is not suitable for the evaluation of text simplification." arXiv preprint arXiv:1810.05995 (2018). - https://github.com/UniversalDependencies/UD_Arabic -PUD - https://universaldependencies.org/ [↗](https://arxiv.org/abs/1909.06814) [our card](https://md.research.elamur.ai/free/en/papers/isabelle-2017-challenge-set-approach-evaluating-machine/)