--- title: "The Effect of Machine Translation on the Performance of" description: "The paper measures how much machine translation of questions degrades a question answering pipeline in an… 0.12 precision. 9 references." authors: ["Azzah Al-Maskari", "Mark Sanderson"] year: 2006 venue: "Proceedings of the Workshop on Multilingual Question Answering - MLQA `06" source: "https://aclanthology.org/W06-1902/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/maskari-2006-effect-machine-translation-performance-arabic/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/maskari-2006-effect-machine-translation-performance-arabic/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2006](https://md.research.elamur.ai/free/en/papers/2006/page/2/) / The Effect of Machine Translation on the Performance of Arabic-EnglishQA System # The Effect of Machine Translation on the Performance of Arabic-EnglishQA System **Authors:** Azzah Al-Maskari and Mark Sanderson · **Year:** 2006 · **Venue:** Proceedings of the Workshop on Multilingual Question Answering - MLQA \`06 · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/W06-1902/) · [PDF](https://aclanthology.org/W06-1902.pdf) ## Summary The paper measures how much machine translation of questions degrades a question answering pipeline in an Arabic-to-English cross-language setting. The authors randomly drew 199 English questions from several TREC QA tracks (92 factoid, 51 definition, 56 list) and ran them through the AnswerFinder system to establish a monolingual baseline. One author, a native Arabic speaker, then translated each question into Arabic by hand. Systran converted the Arabic questions back into English, and the results were fed to AnswerFinder again. Answers were judged manually with iCLEF-style categories: correct, non-exact, wrong and no answer. Accuracy fell from 42.6% to about 10%, a loss of 32.4 points. The authors also built a taxonomy of the translation errors, with name transliteration as the leading cause. This agrees with an earlier finding by Rosso et al. (2005), which used a different QA system, MT system and test set. The work shows that translation quality, not only retrieval, can be the main bottleneck in cross-language QA. It also argues that MT evaluation should check whether translations are correct, not just how retrieval performs downstream. ## Key points - Data: 199 questions from TREC-8, TREC-9, TREC-11, TREC-2003 and TREC-2004. QA system: AnswerFinder. MT system: Systran (Arabic→English). - Monolingual baseline results: - 42.6% correct, 8.1% non-exact, 25.8% wrong, 23.9% no answer. - Factoid questions did best (63 of 92 correct), definition questions worst (6 of 51). - Effect of translation: - Recall dropped from 0.51 to 0.12, precision from 0.76 to 0.41, and F-measure from 0.6 to 0.2. - Only 23 translated questions were answered correctly. Of these, 12 had clean translations and 11 had errors, and most were factoid questions. - Error types: - Wrong transliteration: 45.7% (91 questions), e.g. a person's name rendered literally as "spiritual goodness". - Wrong word sense: about 31%, e.g. psychology became "flag of breath". A quarter of these cases could be fixed with diacritics. - Wrong word order: 25%. - Wrong pronoun: 13.5%, e.g. the Arabic word for "who" became "from". - Proposed remedies: - Combine several MT systems, while noting that they may all agree on a wrong output. - Use regex-based pre- or post-processing to fix Systran's systematic errors. - Offer users an interactive choice among alternative translations. - Prioritise the correct translation of the most important keywords. ## Interesting ideas - Hold the question set fixed and pass it through a manual-translation plus MT round trip, so that any accuracy loss can be blamed on translation alone. - Break the damage down by both error type and question type (factoid vs definition vs list), instead of reporting only one overall number. - Most damage came from proper names treated as ordinary words with dictionary meanings, so names must be identified before any sense lookup happens. - A rule-based system makes regular, repeatable mistakes, and these can be fixed with simple pattern-based correction rules applied after its output. - Several translators can serve as alternative witnesses, but their agreement does not guarantee correctness, because they may share the same wrong output. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (back): Round-trip questions through text → graph → text and rerun the same readers, so the accuracy gap directly measures how lossy the compressor is. - **I2** (answer): Report reader scores per question class and per upstream failure kind, so we can tell which failures hurt which kinds of questions. - **I3** (tokens): Make the tagger flag likely proper names early, so the lemmatizer and WordNet-based sense lookup never turn a name into common nouns. - **I4** (learn): Mine recurring parser or frame errors and compile them into self-induced repair rules, checking each rule against negative controls. - **I5** (learn): When building silver data from multiple witness parsers, treat consensus as evidence only, and audit cases where all witnesses agree but could be jointly wrong. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works 🧭 Topics (2): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 1,105 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works ## Cite Cite as: Azzah Al-Maskari and Mark Sanderson. 2006. The Effect of Machine Translation on the Performance of Arabic-EnglishQA System. Proceedings of the Workshop on Multilingual Question Answering - MLQA \`06. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{almaskari2006effect, title = {{The Effect of Machine Translation on the Performance of Arabic-EnglishQA System}}, author = {Azzah Al-Maskari and Mark Sanderson}, booktitle = {Proceedings of the Workshop on Multilingual Question Answering - MLQA `06}, year = {2006}, url = {https://aclanthology.org/W06-1902/} } ``` - [← previous Standard Arabic Formalization and Linguistic Platform for its Analysis](https://md.research.elamur.ai/free/en/papers/mesfar-2006-standard-arabic-formalization-linguistic-platform/) - [next → Tips and Tricks of the Prague Arabic Dependency Treebank](https://md.research.elamur.ai/free/en/papers/smrz-2006-tips-tricks-prague-arabic-dependency/) - [↑ back to papers of 2006](https://md.research.elamur.ai/free/en/papers/2006/page/2/) ## References 9 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Burger, J., Cardie, C., Chaudhri, V., Gaizauskas, R., Harabagiu, S., Israel, D., Jacquemin, C., Lin,C., Maiorano, S., Miller, G., Moldovan, D., Ogden, B., Prager, J., Riloff, E., Singhal, A., Shrihari, R., Strzalkowski, T., Voorhees, E., Weishedel, R., (2001) Issues, Tasks and Program Structures to Roadmap Research in Question & Answering (Q&A). Technical report, National Institute of Standards and Technology (NIST) [find](https://scholar.google.com/scholar?q=Burger%2C+J%2E%2C+Cardie%2C+C%2E%2C+Chaudhri%2C+V%2E%2C+Gaizauskas%2C+R%2E%2C+Harabagiu%2C+S%2E%2C+Israel%2C+D%2E%2C+Jacquemin%2C+C%2E%2C+Lin%2CC%2E%2C+Maiorano%2C+S%2E%2C+Miller%2C+G%2E%2C+Moldovan%2C+D%2E%2C+Ogden%2C+B%2E%2C+Prager%2C+J%2E%2C+Riloff%2C+E%2E%2C+Singhal%2C+A%2E%2C+Shrihari) 2. Del Castillo, M., Montes y Gómez, M., and Villaseñor, L. (2004) QA on the web: A preliminary study for Spanish language. Proceedings of the 5 th Mexican International Conference on Computer Science (ENC04), Colima, Mexico. [find](https://scholar.google.com/scholar?q=Del+Castillo%2C+M%2E%2C+Montes+y+G%C3%B3mez%2C+M%2E%2C+and+Villase%C3%B1or%2C+L%2E+%282004%29+QA+on+the+web%3A+A+preliminary+study+for+Spanish+language%2E+Proceedings+of+the+5+th+Mexican+International+Conference+on+Computer+Science+%28E) 3. Hull, D. A., and Grefenstette, G. (1996) Querying across languages: a dictionary-based approach to multilingual information retrieval. Research and Development in Information Retrieval, pp46-57. [find](https://scholar.google.com/scholar?q=Hull%2C+D%2E+A%2E%2C+and+Grefenstette%2C+G%2E+%281996%29+Querying+across+languages%3A+a+dictionary%2Dbased+approach+to+multilingual+information+retrieval%2E+Research+and+Development+in+Information+Retrieval%2C+pp46%2D57%2E) 4. Lin, F., & Mitamura, T. (2004) Keyword Translation from English to Chinese for Multilingual QA. In: The 6 th Conference of the Association for Machine Translation in the Americas (AMTA-2004) [find](https://scholar.google.com/scholar?q=Lin%2C+F%2E%2C+%26+Mitamura%2C+T%2E+%282004%29+Keyword+Translation+from+English+to+Chinese+for+Multilingual+QA%2E+In%3A+The+6+th+Conference+of+the+Association+for+Machine+Translation+in+the+Americas+%28AMTA%2D2004%29) 5. Magnini, B., Vallin, A., Ayache C., Erbach, G., Peñas, A., Rijke, M. Rocha, P., Simov, K., Sutcliffe, R. (2004) Overview of the CLEF 2004 Multi-lingual Question Answering Track. In: Proceedings of CLEF 2004 [find](https://scholar.google.com/scholar?q=Magnini%2C+B%2E%2C+Vallin%2C+A%2E%2C+Ayache+C%2E%2C+Erbach%2C+G%2E%2C+Pe%C3%B1as%2C+A%2E%2C+Rijke%2C+M%2E+Rocha%2C+P%2E%2C+Simov%2C+K%2E%2C+Sutcliffe%2C+R%2E+%282004%29+Overview+of+the+CLEF+2004+Multi%2Dlingual+Question+Answering+Track%2E+In%3A+Proceedings+of+CLE) 6. Perrot, L. (2004). Question Answering system for the French Language. In: Proceedings of CLEF 2004 [find](https://scholar.google.com/scholar?q=Perrot%2C+L%2E+%282004%29%2E+Question+Answering+system+for+the+French+Language%2E+In%3A+Proceedings+of+CLEF+2004) 7. Plamondon, L. and Foster, G. (2003) Quantum, a French/English Cross-Language Question Answering System. Proceedings of CLEF 2003 [find](https://scholar.google.com/scholar?q=Plamondon%2C+L%2E+and+Foster%2C+G%2E+%282003%29+Quantum%2C+a+French%2FEnglish+Cross%2DLanguage+Question+Answering+System%2E+Proceedings+of+CLEF+2003) 8. Tanev, H., Negri, M., Magnini, B., Kouylekov, M. (2004) The Diogenes Question Answering System at CLEF-2004. Proceedings of CLEF 2004 [find](https://scholar.google.com/scholar?q=Tanev%2C+H%2E%2C+Negri%2C+M%2E%2C+Magnini%2C+B%2E%2C+Kouylekov%2C+M%2E+%282004%29+The+Diogenes+Question+Answering+System+at+CLEF%2D2004%2E+Proceedings+of+CLEF+2004) 9. Yaser Al-Onaizan and Kevin Knight. (2002) [Translating Named Entities Using Monolingual and Bilingual Resources](https://md.research.elamur.ai/free/en/papers/onaizan-2002-translating-named-entities-monolingual-bilingual/). In: Proceedings of the ACL Workshop on Computational Approaches to Semitic Languages *Updated 2 November 2026*