---
title: "A New Approach to Lexical Disambiguation of Arabic Text (2010)"
description: "Arabic words are hard to analyse because clitics are attached without spaces, short… 96.2% accuracy. 23 references, cited by 1 paper here."
authors: ["Rushin Shah", "Paramveer S. Dhillon", "Mark Liberman", "Dean Foster", "Mohamed Maamouri", "Lyle Ungar"]
year: 2010
venue: "Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D10-1071/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shah-2010-new-approach-lexical-disambiguation-arabic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shah-2010-new-approach-lexical-disambiguation-arabic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2010](https://md.research.elamur.ai/free/en/papers/2010/page/2/) / A New Approach to Lexical Disambiguation of Arabic Text

# A New Approach to Lexical Disambiguation of Arabic Text

**Authors:** [Rushin Shah](https://md.research.elamur.ai/free/en/authors/rushin-shah/), Paramveer S. Dhillon, [Mark Liberman](https://md.research.elamur.ai/free/en/authors/mark-liberman/), Dean Foster, [Mohamed Maamouri](https://md.research.elamur.ai/free/en/authors/mohamed-maamouri/) and [Lyle Ungar](https://md.research.elamur.ai/free/en/authors/lyle-ungar/) · **Year:** 2010 · **Venue:** [Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2010](https://md.research.elamur.ai/free/en/venues/emnlp/2010/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/D10-1071/) · [PDF](https://aclanthology.org/D10-1071.pdf) · [OpenAlex](https://openalex.org/W1911120693)

**Citations:** 9 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/shah-2010-new-approach-lexical-disambiguation-arabic/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Arabic words are hard to analyse because clitics are attached without spaces, short vowels are usually not written, and inflection is rich. A broad-coverage analyzer such as SAMA v3.1 typically returns about a dozen candidate analyses per word, sometimes hundreds, and gives no plausibility ranking. The authors frame lexical analysis as choosing among these candidates. Each candidate couples a stable lemma ID with a full morphological analysis, so lemma and morphology are disambiguated together.

Each label is broken into label elements (LEs), such as the lemma, POS and individual morphosyntactic pieces. Features are drawn from nearby words, which mainly help morphology, and from the wider document, which mainly helps lemmas. Learning has two stages. First, a cheap initial weight is computed for each feature–LE pair by counting co-occurrences. Second, a supervised step shrinks those weights, using only SAMA's candidate sets.

The system picks the correct analysis first 90.6% of the time and within its top two 96.2% of the time. It is far cheaper to train than a CRF. It already supports a reading aid for Arabic learners and a ranking tool for Arabic Treebank annotators.

## Key points
- Candidate source is SAMA v3.1, a successor to the Buckwalter analyzer aligned with LDC Arabic Treebank 3-v3.2. It finds the correct analysis somewhere in its list roughly 95% of the time.
- The full model, M_full, has 9.4% top-1 error and 3.8% top-2 error under 10-fold cross-validation, i.e. 90.6% and 96.2% accuracy.
- Ablations show the value of the design:
  - The whole-label baseline reaches 13.6 / 9.1 error.
  - Separately predicting lemma and morphology (M_ind) reaches 18.7 / 6.0.
  - Context-free unigram features (M_uni) reach 11.6 / 6.4.
  - These results show that factored LEs, joint prediction and n-gram context all help.
- According to the authors, training the two-stage estimate-then-shrink scheme takes roughly four orders of magnitude less effort than training a CRF, yet it retains much of what joint modelling offers.
- The authors argue that the method should carry over unchanged to other highly inflected languages, such as Turkish, Hungarian and Finnish.

## Interesting ideas
- Instead of predicting from a huge, sparse label inventory, split every composite label into parts and let features vote for those parts, which keeps rare full labels learnable.
- Treat disambiguation as reranking a closed candidate list produced by a high-coverage analyzer, so the model can never output an impossible analysis.
- Learning can be split into a counting stage that gives raw feature–label associations and a light supervised stage that only learns how much to shrink them, which trains very cheaply.
- Lemma choice and morphology choice need different context: wide topical context helps decide which word it is, while nearby words decide its inflected form.
- Reporting top-2 accuracy alongside top-1 shows when a system is already good enough for assisted human annotation, even if not for full automation.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): The tagger could score lemma, UPOS and each UD feature as separate elements and then pick the best whole analysis from the lexicon's candidate list, helping with rare tag/feature combinations.
- **I2** (tokens): The en morphology could enumerate every lemma+feature reading of a word form and have the tagger only rank them, keeping each choice explainable as "which candidate won and why".
- **I3** (learn): Training could start from co-occurrence counts over silver data and learn a small set of shrinkage weights, giving fast retraining and transparent per-feature weights.
- **I4** (world): Lemma/sense selection could draw on entities and topics already established in the world of the text, while morphology stays driven by local syntactic context.
- **I5** (learn): Evaluation of the tagger and parser could track top-k accuracy to decide where an LLM teacher or human only needs to choose between two shortlisted options.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 539 works · 179,533 citations
- [University of Pennsylvania](https://md.research.elamur.ai/free/en/universities/university-of-pennsylvania/) — University, Philadelphia, United States · 221 works · 63,234 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [🧩 Morphology](https://md.research.elamur.ai/free/en/topics/morphology/) · 405 works

## Cite

Cite as: Rushin Shah, Paramveer S. Dhillon, Mark Liberman, Dean Foster, Mohamed Maamouri and Lyle Ungar. 2010. A New Approach to Lexical Disambiguation of Arabic Text. Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shah2010new,
  title = {{A New Approach to Lexical Disambiguation of Arabic Text}},
  author = {Rushin Shah, Paramveer S. Dhillon, Mark Liberman, Dean Foster, Mohamed Maamouri and Lyle Ungar},
  booktitle = {Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing},
  year = {2010},
  url = {https://aclanthology.org/D10-1071/}
}
```

- [← previous The Formation of Old English Adverbs: Structural Description and Functional Explanation](https://md.research.elamur.ai/free/en/papers/villalta-2010-formation-old-english-adverbs-structural/)
- [next → CCASH: A Web Application Framework for Efficient, Distributed Language Resource Development](https://md.research.elamur.ai/free/en/papers/felt-2010-ccash-web-application-framework-efficient/)
- [↑ back to papers of 2010](https://md.research.elamur.ai/free/en/papers/2010/page/2/)

## References

23 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Meni Adler and Michael Elhadad. 2006. [An Unsupervised Morpheme-Based HMM for Hebrew Morphological Disambiguation](https://md.research.elamur.ai/free/en/papers/adler-2006-unsupervised-morpheme-based-hmm-hebrew/). In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics.
2. Farag Ahmed and Andreas Nürnberger. 2008. Arabic/English Word Translation Disambiguation using Parallel Corpora and Matching Schemes. In Proceedings of EAMT’08, Hamburg, Germany. [find](https://scholar.google.com/scholar?q=Farag+Ahmed+and+Andreas+Nu%CC%88rnberger%2E+2008%2E+Arabic%2FEnglish+Word+Translation+Disambiguation+using+Parallel+Corpora+and+Matching+Schemes%2E+In+Proceedings+of+EAMT%E2%80%9908%2C+Hamburg%2C+Germany%2E)
3. Tim Buckwalter. 2004. Buckwalter Arabic Morphological Analyzer version 2.0. [↗](https://doi.org/10.35111/050q-5r95)
4. Michael Collins. 2002. [Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms](https://md.research.elamur.ai/free/en/papers/collins-2002-discriminative-training-methods-hidden-markov/). In Proceedings of EMNLP’02.
5. Koby Crammer, Ofer Dekel, Joseph Keshet, Shai Shalev-Shwartz, and Yoram Singer. 2006. Online Passive-Aggressive Algorithms. Journal of Machine Learning Research, 7:551–585. [find](https://scholar.google.com/scholar?q=Koby+Crammer%2C+Ofer+Dekel%2C+Joseph+Keshet%2C+Shai+Shalev%2DShwartz%2C+and+Yoram+Singer%2E+2006%2E+Online+Passive%2DAggressive+Algorithms%2E+Journal+of+Machine+Learning+Research%2C+7%3A551%E2%80%93585%2E)
6. Mona Diab, Kadri Hacioglu, and Daniel Jurafsky. 2004. [Automatic Tagging of Arabic text: From Raw Text to Base Phrase Chunks](https://md.research.elamur.ai/free/en/papers/diab-2004-automatic-tagging-arabic-text-raw/). In Proceedings of the 5th Meeting of the North American Chapter of the Association for Computational Linguistics/Human Language Technologies Conference (HLT-NAACL’04).
7. Elad Dinur, Dmitry Davidov, and Ari Rappoport. 2009. [Unsupervised Concept Discovery in Hebrew Using Simple Unsupervised Word Prefix Segmentation for Hebrew and Arabic](https://md.research.elamur.ai/free/en/papers/dinur-2009-unsupervised-concept-discovery-hebrew-simple/). In Proceedings of the EACL 2009 Workshop on Computational Approaches to Semitic Languages.
8. Kuzman Ganchev and Georgi Georgiev. 2009. Edlin: An Easy to Read Linear Learning Framework. In Proceedings of RANLP’09. [find](https://scholar.google.com/scholar?q=Kuzman+Ganchev+and+Georgi+Georgiev%2E+2009%2E+Edlin%3A+An+Easy+to+Read+Linear+Learning+Framework%2E+In+Proceedings+of+RANLP%E2%80%9909%2E)
9. Yoav Goldberg, Meni Adler, and Michael Elhadad. 2008. EM Can Find Pretty Good HMM POS-Taggers (When Given a Good Start)*. In Proceedings of ACL’08. [find](https://scholar.google.com/scholar?q=Yoav+Goldberg%2C+Meni+Adler%2C+and+Michael+Elhadad%2E+2008%2E+EM+Can+Find+Pretty+Good+HMM+POS%2DTaggers+%28When+Given+a+Good+Start%29%2A%2E+In+Proceedings+of+ACL%E2%80%9908%2E)
10. Nizar Habash and Owen Rambow. 2005. [Arabic Tokenization, Part-of-Speech Tagging and Morphological Disambiguation in One Fell Swoop](https://md.research.elamur.ai/free/en/papers/habash-2005-arabic-tokenization-part-speech-tagging/). In Proceedings of ACL’05, Ann Arbor, MI, USA.
11. Jan Hajic. 2000. [Morphological Tagging: Data vs. Dictionaries](https://md.research.elamur.ai/free/en/papers/hajic-2000-morphological-tagging-data-vs-dictionaries/). In Proceedings of the 1st Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL’00).
12. Jun Hatori, Yusuke Miyao, and Jun’ichi Tsujii. 2008. Word Sense Disambiguation for All Words using Tree-Structured Conditional Random Fields. In Proceedings of COLing’08. [find](https://scholar.google.com/scholar?q=Jun+Hatori%2C+Yusuke+Miyao%2C+and+Jun%E2%80%99ichi+Tsujii%2E+2008%2E+Word+Sense+Disambiguation+for+All+Words+using+Tree%2DStructured+Conditional+Random+Fields%2E+In+Proceedings+of+COLing%E2%80%9908%2E)
13. W. James and Charles Stein. 1961. Estimation with Quadratic Loss. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1. [find](https://scholar.google.com/scholar?q=W%2E+James+and+Charles+Stein%2E+1961%2E+Estimation+with+Quadratic+Loss%2E+In+Proceedings+of+the+Fourth+Berkeley+Symposium+on+Mathematical+Statistics+and+Probability%2C+Volume+1%2E)
14. András Kornai. 1994. On Hungarian morphology (Linguistica, Series A: Studia et Dissertationes 14). Linguistics Institute of Hungarian Academy of Sciences, Budapest. [find](https://scholar.google.com/scholar?q=Andra%CC%81s+Kornai%2E+1994%2E+On+Hungarian+morphology+%28Linguistica%2C+Series+A%3A+Studia+et+Dissertationes+14%29%2E+Linguistics+Institute+of+Hungarian+Academy+of+Sciences%2C+Budapest%2E)
15. Seth Kulick. 2010. [Simultaneous Tokenization and Part-of-Speech Tagging for Arabic without a Morphological Analyzer](https://md.research.elamur.ai/free/en/papers/kulick-2010-simultaneous-tokenization-part-speech-tagging/). In Proceedings of ACL’10.
16. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In Proceedings of ICML’01, pages 282–289. [find](https://scholar.google.com/scholar?q=John+D%2E+Lafferty%2C+Andrew+McCallum%2C+and+Fernando+C%2E+N%2E+Pereira%2E+2001%2E+Conditional+Random+Fields%3A+Probabilistic+Models+for+Segmenting+and+Labeling+Sequence+Data%2E+In+Proceedings+of+ICML%E2%80%9901%2C+pages+282%E2%80%93289)
17. Mohamed Maamouri, Ann Bies, and Tim Buckwalter. 2004. The Penn Arabic Treebank: Building a Large Scale Annotated Arabic Corpus. In Proceedings of NEMLAR Conference on Arabic Language Resources and Tools. [find](https://scholar.google.com/scholar?q=Mohamed+Maamouri%2C+Ann+Bies%2C+and+Tim+Buckwalter%2E+2004%2E+The+Penn+Arabic+Treebank%3A+Building+a+Large+Scale+Annotated+Arabic+Corpus%2E+In+Proceedings+of+NEMLAR+Conference+on+Arabic+Language+Resources+and+Too)
18. Mohamed Maamouri, David Graff, Basma Bouziri, Sondos Krouna, and Seth Kulick. 2009. LDC Standard Arabic Morphological Analyzer (SAMA) v. 3.0. [find](https://scholar.google.com/scholar?q=Mohamed+Maamouri%2C+David+Graff%2C+Basma+Bouziri%2C+Sondos+Krouna%2C+and+Seth+Kulick%2E+2009%2E+LDC+Standard+Arabic+Morphological+Analyzer+%28SAMA%29+v%2E+3%2E0%2E)
19. Andrew McCallum, 2001. MALLET: A Machine Learning for Language Toolkit. Software available at [http://mallet.cs.umass.edu](http://mallet.cs.umass.edu). [↗](http://mallet.cs.umass.edu)
20. Emad Mohamed and Sandra Kübler. 2010. [Arabic Part of Speech Tagging](https://md.research.elamur.ai/free/en/papers/mohamed-2010-arabic-part-speech-tagging/). In Proceedings of LREC’10.
21. Ryan Roth, Owen Rambow, Nizar Habash, Mona Diab, and Cynthia Rudin. 2008. [Arabic Morphological Tagging, Diacritization, and Lemmatization Using Lexeme Models and Feature Ranking](https://md.research.elamur.ai/free/en/papers/roth-2008-arabic-morphological-tagging-diacritization-lemmatization/). In Proceedings of ACL’08, Columbus, Ohio, USA.
22. Noah A. Smith, David A. Smith, and Roy W. Tromble. 2005. Context-Based Morphological Disambiguation with Random Fields*. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP). [find](https://scholar.google.com/scholar?q=Noah+A%2E+Smith%2C+David+A%2E+Smith%2C+and+Roy+W%2E+Tromble%2E+2005%2E+Context%2DBased+Morphological+Disambiguation+with+Random+Fields%2A%2E+In+Proceedings+of+Human+Language+Technology+Conference+and+Conference+on+Empiri)
23. Kristina Toutanova and Colin Cherry. 2009. [A Global Model for Joint Lemmatization and Part-of-Speech Prediction](https://md.research.elamur.ai/free/en/papers/toutanova-2009-global-model-joint-lemmatization-part/). In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing, pages 486–494.

*Updated 1 November 2026*