---
title: "Handling Unknown Words in Arabic FST Morphology (2012)"
description: "Shaalan and Attia present what they describe as the first attempt to lemmatize Arabic unknown words. 15 references, cited by 2 papers here."
authors: ["Khaled Shaalan", "Mohammed Attia"]
year: 2012
venue: "Proceedings of the 10th International Workshop on Finite State Methods and Natural Language Processing"
source: "https://aclanthology.org/W12-6203/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shaalan-2012-handling-unknown-words-arabic-fst/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shaalan-2012-handling-unknown-words-arabic-fst/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2012](https://md.research.elamur.ai/free/en/papers/2012/page/3/) / Handling Unknown Words in Arabic FST Morphology

# Handling Unknown Words in Arabic FST Morphology

**Authors:** [Khaled Shaalan](https://md.research.elamur.ai/free/en/authors/khaled-shaalan/) and [Mohammed Attia](https://md.research.elamur.ai/free/en/authors/mohammed-attia/) · **Year:** 2012 · **Venue:** [Proceedings of the 10th International Workshop on Finite State Methods and Natural Language Processing](https://md.research.elamur.ai/free/en/venues/finite-state-methods-and-natural-language-processing/) ([International Workshop on Finite State Methods and Natural Language Processing 2012](https://md.research.elamur.ai/free/en/venues/finite-state-methods-and-natural-language-processing/2012/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W12-6203/) · [PDF](https://aclanthology.org/W12-6203.pdf) · [OpenAlex](https://openalex.org/W393951510)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · 2 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Arabic morphological analysers depend on a lexicon, so newly coined or borrowed words are not analysed and cannot be lemmatized by lookup. Shaalan and Attia present what they describe as the first attempt to lemmatize Arabic unknown words. They ran MADA, which relies internally on SAMA, over a corpus of roughly 1.09 billion words from Arabic Gigaword (4th ed.) and Al-Jazeera news. Out-of-lexicon types were filtered with a spell checker and a frequency threshold. Each surviving word then passed through a pipeline. An XFST finite-state guesser lists every plausible segmentation into clitics, affixes and stem. The candidate whose morphological features best agree with MADA's top-scoring analysis is kept. A weighting formula then ranks the resulting lemmas for lexicographers to review. On a hand-annotated gold set of 1,310 types, lemmatization accuracy rises from a 45% baseline to 54%, or 63% when definite-article differences are ignored. This shows that a heavily overgenerating FST guesser can be useful in practice for growing a lexicon. The resulting unknown-word lexicon is released as open source.

## Key points
- Filtering funnel:
  - 2,116,180 types had no SAMA analysis.
  - 208,188 of these were accepted by the MS Office 2010 Arabic spell checker.
  - 40,277 remained after keeping only types with frequency ≥10.
  - 1,310 were sampled for gold annotation (lemma, POS, include-in-dictionary judgement).
- The guesser treats a noun stem as any 2–24 non-numeric Arabic characters and a verb stem as 2–6 characters. It averages 12.6 analyses per word, compared with 2.1 for AraComLex. The gold lemma appears among its candidates 97% of the time.
- POS accuracy:
  - 45% when every word is tagged as a proper noun (baseline);
  - 60% with MADA alone;
  - 69% with "Voted POS Tagging" across forms that share a lemma.
- Lemmatization reduced 40,277 surface types to 18,399 lemmas, a 54% reduction.
- The ranking score combines three factors: sister-form count × 800, cumulative frequency of the sister forms (the two halved), and a POS factor that penalizes proper nouns.
  - Combined criteria: 78 good words in the top 100 and 15 in the bottom 100.
  - Frequency alone: 63 in the top 100 and 50 in the bottom 100.

## Interesting ideas
- Let a cheap rule-based guesser overgenerate every possible analysis, then pick the one whose features best match a statistical tagger's contextual prediction; candidate generation stays transparent while context decides.
- Pool evidence across all surface forms that map to one lemma: if most of a lemma's forms carry a certain POS, override the minority tag.
- Report an oracle ceiling (is the gold answer anywhere in the candidate set?) next to first-choice accuracy, so generation failures are separated from selection failures.
- Put a spell-checker gate and a minimum-frequency threshold in front of lexicon growth, so typos are not mistaken for new vocabulary.
- Rank new entries for human review by how many distinct inflected forms a lemma has, not only by raw frequency, and down-weight proper nouns whose frequency is inflated.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): An affix-stripping guesser for OOV English words in crate en could propose lemma/feature candidates, and the tagger's chosen features would select among them, with the rejected options kept as an explanation trail.
- **I2** (tokens): Corpus-level voting over lemma families could correct sporadic tagger errors on rare or new words before they propagate into UD trees.
- **I3** (learn): Evaluations of parsers or frame selection could log both candidate recall and top-1 accuracy, showing whether the fix belongs in the generator or the ranker.
- **I4** (learn): Self-induced lexicon or repair-rule additions could require that a candidate survives a spelling filter and recurs often enough before it is accepted.
- **I5** (world): When extending compiled knowledge, such as verb classes or ontology entries, from corpus discoveries, morphological productivity could be used as a signal of a genuine lexical item to prioritise curation.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Dublin City University](https://md.research.elamur.ai/free/en/universities/dublin-city-university/) — University, Dublin, Ireland · 72 works · 9,746 citations
- [British University in Dubai](https://md.research.elamur.ai/free/en/universities/british-university-in-dubai/) — University, Dubai, United Arab Emirates · 11 works · 589 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works · [🧩 Morphology](https://md.research.elamur.ai/free/en/topics/morphology/) · 420 works

## Cite

Cite as: Khaled Shaalan and Mohammed Attia. 2012. Handling Unknown Words in Arabic FST Morphology. Proceedings of the 10th International Workshop on Finite State Methods and Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shaalan2012handling,
  title = {{Handling Unknown Words in Arabic FST Morphology}},
  author = {Khaled Shaalan and Mohammed Attia},
  booktitle = {Proceedings of the 10th International Workshop on Finite State Methods and Natural Language Processing},
  year = {2012},
  doi = {10.13140/2.1.3572.2242},
  url = {https://doi.org/10.13140/2.1.3572.2242}
}
```

- [← previous Exploiting Discourse Relations for Sentiment Analysis](https://md.research.elamur.ai/free/en/papers/wang-2012-exploiting-discourse-relations-sentiment-analysis/)
- [next → Keyphrase Based Arabic Summarizer (KPAS)](https://md.research.elamur.ai/free/en/papers/el-shishtawy-2012-keyphrase-based-arabic-summarizer-kpas/)
- [↑ back to papers of 2012](https://md.research.elamur.ai/free/en/papers/2012/page/3/)

## References

15 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Adler, M., Goldberg, Y., Gabay, D. and Elhadad, M. 2008. Unsupervised Lexicon-Based Resolution of Unknown Words for Full Morpholological Analysis. In: Proceedings of Association for Computational Linguistics (ACL), Columbus, Ohio. [find](https://scholar.google.com/scholar?q=Adler%2C+M%2E%2C+Goldberg%2C+Y%2E%2C+Gabay%2C+D%2E+and+Elhadad%2C+M%2E+2008%2E+Unsupervised+Lexicon%2DBased+Resolution+of+Unknown+Words+for+Full+Morpholological+Analysis%2E+In%3A+Proceedings+of+Association+for+Computational+Ling)
2. Attia, M. 2006. An Ambiguity-Controlled Morphological Analyzer for Modern Standard Arabic Modelling Finite State Networks. In: Challenges of Arabic for NLP/MT Conference, The British Computer Society, London, UK. [find](https://scholar.google.com/scholar?q=Attia%2C+M%2E+2006%2E+An+Ambiguity%2DControlled+Morphological+Analyzer+for+Modern+Standard+Arabic+Modelling+Finite+State+Networks%2E+In%3A+Challenges+of+Arabic+for+NLP%2FMT+Conference%2C+The+British+Computer+Society%2C)
3. Attia, Mohammed, Pavel Pecina, Lamia Tounsi, Antonio Toral, Josef van Genabith. 2011. [An Open-Source Finite State Morphological Transducer for Modern Standard Arabic](https://md.research.elamur.ai/free/en/papers/attia-2011-open-source-finite-state-morphological/). International Workshop on Finite State Methods and Natural Language Processing (FSMNLP). Blois, France.
4. Beesley, K. R. 2001. Finite-State Morphological Analysis and Generation of Arabic at Xerox Research: Status and Plans in 2001. In: The ACL 2001 Workshop on Arabic Language Processing: Status and Prospects, Toulouse, France. [find](https://scholar.google.com/scholar?q=Beesley%2C+K%2E+R%2E+2001%2E+Finite%2DState+Morphological+Analysis+and+Generation+of+Arabic+at+Xerox+Research%3A+Status+and+Plans+in+2001%2E+In%3A+The+ACL+2001+Workshop+on+Arabic+Language+Processing%3A+Status+and+Prosp)
5. Beesley, K. R., and Karttunen, L.. 2003. Finite State Morphology: CSLI studies in computational linguistics. Stanford, Calif.: Csli. [find](https://scholar.google.com/scholar?q=Beesley%2C+K%2E+R%2E%2C+and+Karttunen%2C+L%2E%2E+2003%2E+Finite+State+Morphology%3A+CSLI+studies+in+computational+linguistics%2E+Stanford%2C+Calif%2E%3A+Csli%2E)
6. Buckwalter, T. 2004. Buckwalter Arabic Morphological Analyzer (BAMA) Version 2.0. Linguistic Data Consortium (LDC) catalogue number LDC2004L02, ISBN1-58563-324-0 [find](https://scholar.google.com/scholar?q=Buckwalter%2C+T%2E+2004%2E+Buckwalter+Arabic+Morphological+Analyzer+%28BAMA%29+Version+2%2E0%2E+Linguistic+Data+Consortium+%28LDC%29+catalogue+number+LDC2004L02%2C+ISBN1%2D58563%2D324%2D0)
7. Dichy, J. 2001. On lemmatization in Arabic, A formal definition of the Arabic entries of multilingual lexical databases. ACL/EACL 2001 Workshop on Arabic Language Processing: Status and Prospects. Toulouse, France. [find](https://scholar.google.com/scholar?q=Dichy%2C+J%2E+2001%2E+On+lemmatization+in+Arabic%2C+A+formal+definition+of+the+Arabic+entries+of+multilingual+lexical+databases%2E+ACL%2FEACL+2001+Workshop+on+Arabic+Language+Processing%3A+Status+and+Prospects%2E+Tou)
8. Dichy, J., and Farghaly, A. 2003. Roots & Patterns vs. Stems plus Grammar-Lexis Specifications: on what basis should a multilingual lexical database centred on Arabic be built? In: The MT-Summit IX workshop on Machine Translation for Semitic Languages, New Orleans. [find](https://scholar.google.com/scholar?q=Dichy%2C+J%2E%2C+and+Farghaly%2C+A%2E+2003%2E+Roots+%26+Patterns+vs%2E+Stems+plus+Grammar%2DLexis+Specifications%3A+on+what+basis+should+a+multilingual+lexical+database+centred+on+Arabic+be+built%3F+In%3A+The+MT%2DSummit+IX+wo)
9. Erjavec, T., and Džerosk, S. 2004. Machine Learning of Morphosyntactic Structure: Lemmatizing Unknown Slovene Words. Applied Artificial Intelligence, 18:17–41. [find](https://scholar.google.com/scholar?q=Erjavec%2C+T%2E%2C+and+D%C5%BEerosk%2C+S%2E+2004%2E+Machine+Learning+of+Morphosyntactic+Structure%3A+Lemmatizing+Unknown+Slovene+Words%2E+Applied+Artificial+Intelligence%2C+18%3A17%E2%80%9341%2E)
10. Kiraz, G. A. 2001. Computational Nonlinear Morphology: With Emphasis on Semitic Languages. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Kiraz%2C+G%2E+A%2E+2001%2E+Computational+Nonlinear+Morphology%3A+With+Emphasis+on+Semitic+Languages%2E+Cambridge+University+Press%2E)
11. Lindén, K. 2008. A Probabilistic Model for Guessing Base Forms of New Words by Analogy. In CICling-2008, 9th International Conference on Intelligent Text Processing and Computational Linguistics, Haifa, Israel, pp. 106-116. [find](https://scholar.google.com/scholar?q=Lind%C3%A9n%2C+K%2E+2008%2E+A+Probabilistic+Model+for+Guessing+Base+Forms+of+New+Words+by+Analogy%2E+In+CICling%2D2008%2C+9th+International+Conference+on+Intelligent+Text+Processing+and+Computational+Linguistics%2C+Haif)
12. Maamouri, M., Graff, D., Bouziri, B., Krouna, S., and Kulick, S. 2010. LDC Standard Arabic Morphological Analyzer (SAMA) v. 3.1. LDC Catalog No. LDC2010L01. ISBN: 1-58563-555-3. [find](https://scholar.google.com/scholar?q=Maamouri%2C+M%2E%2C+Graff%2C+D%2E%2C+Bouziri%2C+B%2E%2C+Krouna%2C+S%2E%2C+and+Kulick%2C+S%2E+2010%2E+LDC+Standard+Arabic+Morphological+Analyzer+%28SAMA%29+v%2E+3%2E1%2E+LDC+Catalog+No%2E+LDC2010L01%2E+ISBN%3A+1%2D58563%2D555%2D3%2E)
13. Parker, R., Graff, D., Chen, K., Kong, J., and Maeda, K. 2009. Arabic Gigaword Fourth Edition. LDC Catalog No. LDC2009T30. ISBN: 1-58563-532-4. [find](https://scholar.google.com/scholar?q=Parker%2C+R%2E%2C+Graff%2C+D%2E%2C+Chen%2C+K%2E%2C+Kong%2C+J%2E%2C+and+Maeda%2C+K%2E+2009%2E+Arabic+Gigaword+Fourth+Edition%2E+LDC+Catalog+No%2E+LDC2009T30%2E+ISBN%3A+1%2D58563%2D532%2D4%2E)
14. Roth, R., Rambow, O., Habash, N., Diab, M., and Rudin, C. 2008. [Arabic Morphological Tagging, Diacritization, and Lemmatization Using Lexeme Models and Feature Ranking](https://md.research.elamur.ai/free/en/papers/roth-2008-arabic-morphological-tagging-diacritization-lemmatization/). In: Proceedings of Association for Computational Linguistics (ACL), Columbus, Ohio.
15. Shaalan, K., Magdy, M., Fahmy, A., Morphological Analysis of Il-formed Arabic Verbs for Second Language Learners, In Eds. McCarthy P., Boonthum, C., Applied Natural Language Processing: Identification, Investigation and Resolution, PP. 383-397, IGI Global, PA, USA, 2012. [find](https://scholar.google.com/scholar?q=Shaalan%2C+K%2E%2C+Magdy%2C+M%2E%2C+Fahmy%2C+A%2E%2C+Morphological+Analysis+of+Il%2Dformed+Arabic+Verbs+for+Second+Language+Learners%2C+In+Eds%2E+McCarthy+P%2E%2C+Boonthum%2C+C%2E%2C+Applied+Natural+Language+Processing%3A+Identification)

*Updated 2 November 2026*