---
title: "Examining the Effect of Improved Context Sensitive Morphology"
description: "Arabic words carry many attached prefixes and suffixes, so retrieval systems must decide how to… 19 references, cited by 3 papers here."
authors: ["Kareem Darwish", "Hany Hassan", "Ossama Emam"]
year: 2005
venue: "Proceedings of the ACL Workshop on Computational Approaches to Semitic Languages"
source: "https://aclanthology.org/W05-0704/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/darwish-2005-examining-effect-improved-context-sensitive/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/darwish-2005-examining-effect-improved-context-sensitive/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2005](https://md.research.elamur.ai/free/en/papers/2005/) / Examining the Effect of Improved Context Sensitive Morphology on Arabic Information Retrieval

# Examining the Effect of Improved Context Sensitive Morphology on Arabic Information Retrieval

**Authors:** [Kareem Darwish](https://md.research.elamur.ai/free/en/authors/kareem-darwish/), [Hany Hassan](https://md.research.elamur.ai/free/en/authors/hany-hassan/) and [Ossama Emam](https://md.research.elamur.ai/free/en/authors/ossama-emam/) · **Year:** 2005 · **Venue:** [Proceedings of the ACL Workshop on Computational Approaches to Semitic Languages](https://md.research.elamur.ai/free/en/venues/acl-workshop-on-computational-approaches-to-semitic-languages/) ([ACL Workshop on Computational Approaches to Semitic Languages 2005](https://md.research.elamur.ai/free/en/venues/acl-workshop-on-computational-approaches-to-semitic-languages/2005/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W05-0704/) · [PDF](https://aclanthology.org/W05-0704.pdf) · [OpenAlex](https://openalex.org/W2033828558)

**Citations:** 22 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/darwish-2005-examining-effect-improved-context-sensitive/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Arabic words carry many attached prefixes and suffixes, so retrieval systems must decide how to merge related word forms. Earlier work found that full morphological analysis rarely beat simple light stemming. The authors suspected that weak coverage and frequent analysis errors were to blame. They tested the IBM-LM analyzer, which segments words using a trigram language model trained on the manually segmented LDC Arabic Treebank. It adds a filter for invalid prefix–suffix pairs and acquires new stems without supervision. They ran it both with sentence context and on isolated words, and obtained roots by passing its stems through Sebawai. These index terms were compared with plain words, Al-Stem light stemming, and Sebawai stems and roots on the TREC 2002 Arabic collection, using Lemur with Okapi BM-25. The more accurate analyzer clearly outperformed Sebawai. Context added roughly 3% mean average precision for stems, which was close to but short of significance. The gain over light stemming was small and not significant, and it came at a much higher processing cost.

## Key points
- The test collection was LDC2001T55, the TREC 2002 cross-language track data: 383,872 AFP Arabic news articles and 50 topics, with between 10 and 523 relevant documents per topic (118 on average). Queries combined each topic's title and description fields.
- Retrieval used Lemur with BM-25 weighting, run with and without blind relevance feedback (20 terms taken from the top 5 documents). Effectiveness was measured by mean uninterpolated average precision, and runs were compared with a Wilcoxon signed-rank test at p \< 0.05.
- Lee et al. report a 2.9% analysis error rate for IBM-LM, compared with 7.3% for Sebawai. In retrieval, IBM-LM terms beat Sebawai terms by a wide margin.
- Using context raised stem-based MAP by about 3%, with p = 0.063 with feedback and p = 0.054 without. By comparison, blind feedback alone gave about 6%. Context changed the analysis of only 0.12% of word occurrences.
- IBM-LM stems were significantly better than plain words, while Al-Stem light stemming was not. However, in-context analysis of the collection took 16 hours, compared with 10 minutes for light stemming.

## Interesting ideas
- The same analyzer was run twice, once on full sentences and once on each unique word in isolation, so the measured difference comes from context alone and not from a different system.
- The authors counted how many tokens actually changed analysis when context was added (0.12%) and how dominant the top analysis was for ambiguous types (over 98%). This puts the small downstream gain in proportion.
- The paper argues that consistent but sometimes wrong conflation (light stemming) holds up well, because downstream matching cares more about uniformity than linguistic correctness.
- Candidate segmentations are pruned by a table of legal prefix–suffix combinations before a language model scores them, so a few rules prevent impossible outputs.
- The authors observe that over-merging forms hurts retrieval less than under-merging, so the right granularity of conflation depends on the task.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Mova could evaluate any context-dependent tagger or lemmatizer against its own context-free run, which works as a clean negative control for the value of context.
- **I2** (tokens): Before claiming improvements, Mova could report what fraction of lemmas or tags a contextual model changes relative to a lexicon-default baseline.
- **I3** (tokens): Mova's en lemmatizer should guarantee deterministic output for identical inputs, so that frame and entity matching later in the pipeline never splits one word into several forms.
- **I4** (tokens): A small compatibility table for English affixes and clitics could filter morphological candidates before statistical ranking, which keeps every rejection explainable.
- **I5** (answer): When readers match question words to entities in the text world, Mova could lean towards broader lemma or synonym grouping and then use the structured frames to recover precision.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [IBM (Egypt)](https://md.research.elamur.ai/free/en/companies/ibm-egypt/) — Company, Giza, Egypt · 8 works · 301 citations
- [German University in Cairo](https://md.research.elamur.ai/free/en/universities/german-university-in-cairo/) — University, Cairo, Egypt · 6 works · 74 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (3): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 473 works · [🧩 Morphology](https://md.research.elamur.ai/free/en/topics/morphology/) · 405 works

## Cite

Cite as: Kareem Darwish, Hany Hassan and Ossama Emam. 2005. Examining the Effect of Improved Context Sensitive Morphology on Arabic Information Retrieval. Proceedings of the ACL Workshop on Computational Approaches to Semitic Languages.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{darwish2005examining,
  title = {{Examining the Effect of Improved Context Sensitive Morphology on Arabic Information Retrieval}},
  author = {Kareem Darwish, Hany Hassan and Ossama Emam},
  booktitle = {Proceedings of the ACL Workshop on Computational Approaches to Semitic Languages},
  year = {2005},
  doi = {10.3115/1621787.1621793},
  url = {https://doi.org/10.3115/1621787.1621793}
}
```

- [← previous Memory-Based Morphological Analysis Generation and Part-of-Speech Tagging of Arabic](https://md.research.elamur.ai/free/en/papers/marsi-2005-memory-based-morphological-analysis-generation/)
- [next → A Probabilistic Setting and Lexical Coocurrence Model for Textual Entailment](https://md.research.elamur.ai/free/en/papers/glickman-2005-probabilistic-setting-lexical-coocurrence-model/)
- [↑ back to papers of 2005](https://md.research.elamur.ai/free/en/papers/2005/)

## References

19 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abu-Salem, H., M. Al-Omari, and M. Evens. Stemming Methodologies Over Individual Query Words for Arabic Information Retrieval. JASIS, 1999. 50(6): p. 524-529. [find](https://scholar.google.com/scholar?q=Abu%2DSalem%2C+H%2E%2C+M%2E+Al%2DOmari%2C+and+M%2E+Evens%2E+Stemming+Methodologies+Over+Individual+Query+Words+for+Arabic+Information+Retrieval%2E+JASIS%2C+1999%2E+50%286%29%3A+p%2E+524%2D529%2E)
2. Aljlayl, M., S. Beitzel, E. Jensen, A. Chowdhury, D. Holmes, M. Lee, D. Grossman, and O. Frieder. IIT at TREC-10. In TREC. 2001. Gaithersburg, MD. [find](https://scholar.google.com/scholar?q=Aljlayl%2C+M%2E%2C+S%2E+Beitzel%2C+E%2E+Jensen%2C+A%2E+Chowdhury%2C+D%2E+Holmes%2C+M%2E+Lee%2C+D%2E+Grossman%2C+and+O%2E+Frieder%2E+IIT+at+TREC%2D10%2E+In+TREC%2E+2001%2E+Gaithersburg%2C+MD%2E)
3. Al-Kharashi, I. and M Evens. Comparing Words, Stems, and Roots as Index Terms in an Arabic Information Retrieval System. JASIS, 1994. 45(8): p. 548 - 560. [↗](https://doi.org/10.1002/(sici)1097-4571(199409)45:8<548::aid-asi3>3.0.co;2-x)
4. Antworth, E. PC-KIMMO: a two-level processor for morphological analysis. In Occasional Publications in Academic Computing. 1990. Dallas, TX: Summer Institute of Linguistics. [find](https://scholar.google.com/scholar?q=Antworth%2C+E%2E+PC%2DKIMMO%3A+a+two%2Dlevel+processor+for+morphological+analysis%2E+In+Occasional+Publications+in+Academic+Computing%2E+1990%2E+Dallas%2C+TX%3A+Summer+Institute+of+Linguistics%2E)
5. Ahmed, Mohamed Attia. A Large-Scale Computational Processor of the Arabic Morphology, and Applications. A Master’s Thesis, Faculty of Engineering, Cairo University, Cairo, Egypt, 2000. [find](https://scholar.google.com/scholar?q=Ahmed%2C+Mohamed+Attia%2E+A+Large%2DScale+Computational+Processor+of+the+Arabic+Morphology%2C+and+Applications%2E+A+Master%E2%80%99s+Thesis%2C+Faculty+of+Engineering%2C+Cairo+University%2C+Cairo%2C+Egypt%2C+2000%2E)
6. Chen, A. and F. Gey. Translation Term Weighting and Combining Translation Resources in Cross-Language Retrieval. In TREC, 2001. Gaithersburg, MD. [find](https://scholar.google.com/scholar?q=Chen%2C+A%2E+and+F%2E+Gey%2E+Translation+Term+Weighting+and+Combining+Translation+Resources+in+Cross%2DLanguage+Retrieval%2E+In+TREC%2C+2001%2E+Gaithersburg%2C+MD%2E)
7. Darwish, K. Building a Shallow Morphological Analyzer in One Day. ACL Workshop on Computational Approaches to Semitic Languages. 2002. [find](https://scholar.google.com/scholar?q=Darwish%2C+K%2E+Building+a+Shallow+Morphological+Analyzer+in+One+Day%2E+ACL+Workshop+on+Computational+Approaches+to+Semitic+Languages%2E+2002%2E)
8. Darwish, K. and D. Oard. CLIR Experiments at Maryland for TREC 2002: Evidence Combination for Arabic-English Retrieval. In TREC. 2002. Gaithersburg, MD. [↗](https://doi.org/10.21236/ada452814)
9. Darwish, K. and D. Oard. Term Selection for Searching Printed Arabic. SIGIR, 2002. Tampere, Finland. p. 261 - 268. [find](https://scholar.google.com/scholar?q=Darwish%2C+K%2E+and+D%2E+Oard%2E+Term+Selection+for+Searching+Printed+Arabic%2E+SIGIR%2C+2002%2E+Tampere%2C+Finland%2E+p%2E+261+%2D+268%2E)
10. Gey, F. and D. Oard. The TREC-2001 Cross-Language Information Retrieval Track: Searching Arabic Using English, French or Arabic Queries. TREC, 2001. Gaithersburg, MD. p. 16-23. [find](https://scholar.google.com/scholar?q=Gey%2C+F%2E+and+D%2E+Oard%2E+The+TREC%2D2001+Cross%2DLanguage+Information+Retrieval+Track%3A+Searching+Arabic+Using+English%2C+French+or+Arabic+Queries%2E+TREC%2C+2001%2E+Gaithersburg%2C+MD%2E+p%2E+16%2D23%2E)
11. Hmeidi, I., G. Kanaan, and M. Evens. Design and Implementation of Automatic Indexing for Information Retrieval with Arabic Documents. JASIS, 1997. 48(10): p. 867 - 881. [↗](https://doi.org/10.1002/(sici)1097-4571(199710)48:10<867::aid-asi3>3.3.co;2-r)
12. Kiraz, G. Arabic Computation Morphology in the West. In The 6th International Conference and Exhibition on Cambridge. Multi-lingual Computing. 1998. [find](https://scholar.google.com/scholar?q=Kiraz%2C+G%2E+Arabic+Computation+Morphology+in+the+West%2E+In+The+6th+International+Conference+and+Exhibition+on+Cambridge%2E+Multi%2Dlingual+Computing%2E+1998%2E)
13. Koskenniemi, K., Two Level Morphology: A General Computational Model for Word-form Recognition and Production. 1983, Department of General Linguistics, University of Helsinki. [find](https://scholar.google.com/scholar?q=Koskenniemi%2C+K%2E%2C+Two+Level+Morphology%3A+A+General+Computational+Model+for+Word%2Dform+Recognition+and+Production%2E+1983%2C+Department+of+General+Linguistics%2C+University+of+Helsinki%2E)
14. Larkey, L., L. Ballesteros, and M. Connell. Improving Stemming for Arabic Information Retrieval: Light Stemming and Co-occurrence Analysis. SIGIR 2002. p. 275-282, 2002. [find](https://scholar.google.com/scholar?q=Larkey%2C+L%2E%2C+L%2E+Ballesteros%2C+and+M%2E+Connell%2E+Improving+Stemming+for+Arabic+Information+Retrieval%3A+Light+Stemming+and+Co%2Doccurrence+Analysis%2E+SIGIR+2002%2E+p%2E+275%2D282%2C+2002%2E)
15. Lee, Y., K. Papineni, S. Roukos, O. Emam, and H. Hassan. [Language Model Based Arabic Word Segmentation](https://md.research.elamur.ai/free/en/papers/lee-2003-language-model-based-arabic-word/). In the Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, July 2003, Sapporo, Japan. p. 399 - 406.
16. Mayfield, J., P. McNamee, C. Costello, C. Piatko, and A. Banerjee. JHU/APL at TREC 2001: Experiments in Filtering and in Arabic, Video, and Web Retrieval. In TREC 2001. Gaithersburg, MD. p. 322-329. [find](https://scholar.google.com/scholar?q=Mayfield%2C+J%2E%2C+P%2E+McNamee%2C+C%2E+Costello%2C+C%2E+Piatko%2C+and+A%2E+Banerjee%2E+JHU%2FAPL+at+TREC+2001%3A+Experiments+in+Filtering+and+in+Arabic%2C+Video%2C+and+Web+Retrieval%2E+In+TREC+2001%2E+Gaithersburg%2C+MD%2E+p%2E+322%2D329%2E)
17. Oard, D. and F. Gey. The TREC 2002 Arabic/English CLIR Track. In TREC 2002. Gaithersburg, MD. [find](https://scholar.google.com/scholar?q=Oard%2C+D%2E+and+F%2E+Gey%2E+The+TREC+2002+Arabic%2FEnglish+CLIR+Track%2E+In+TREC+2002%2E+Gaithersburg%2C+MD%2E)
18. Sanderson, M. Word sense disambiguation and information retrieval. In Proceedings of the 17th ACM SIGIR Conference, p. 142-151, 1994 [find](https://scholar.google.com/scholar?q=Sanderson%2C+M%2E+Word+sense+disambiguation+and+information+retrieval%2E+In+Proceedings+of+the+17th+ACM+SIGIR+Conference%2C+p%2E+142%2D151%2C+1994)
19. Xu, J., A. Fraser, and R. Weischedel. 2001 Cross-Lingual Retrieval at BBN. In TREC, 2001. Gaithersburg, MD. p. 68 - 75. [find](https://scholar.google.com/scholar?q=Xu%2C+J%2E%2C+A%2E+Fraser%2C+and+R%2E+Weischedel%2E+2001+Cross%2DLingual+Retrieval+at+BBN%2E+In+TREC%2C+2001%2E+Gaithersburg%2C+MD%2E+p%2E+68+%2D+75%2E)

*Updated 1 November 2026*