--- title: "Language Model Based Arabic Word Segmentation — Lee 2003" description: "The paper tackles splitting Arabic words into their meaningful pieces. 1 reference, cited by 43 papers here." authors: ["Young-Suk Lee", "Kishore Papineni", "Salim Roukos", "Ossama Emam", "Hany Hassan"] year: 2003 venue: "Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics" source: "https://aclanthology.org/P03-1051/" doi: "10.3115/1075096.1075147" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/lee-2003-language-model-based-arabic-word/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lee-2003-language-model-based-arabic-word/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2003](https://md.research.elamur.ai/free/en/papers/2003/) / Language Model Based Arabic Word Segmentation # Language Model Based Arabic Word Segmentation **Authors:** [Young-Suk Lee](https://md.research.elamur.ai/free/en/authors/young-suk-lee/), [Kishore Papineni](https://md.research.elamur.ai/free/en/authors/kishore-papineni/), [Salim Roukos](https://md.research.elamur.ai/free/en/authors/salim-roukos/), [Ossama Emam](https://md.research.elamur.ai/free/en/authors/ossama-emam/) and [Hany Hassan](https://md.research.elamur.ai/free/en/authors/hany-hassan/) · **Year:** 2003 · **Venue:** [Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2003](https://md.research.elamur.ai/free/en/venues/acl/2003/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/P03-1051/) · [PDF](https://aclanthology.org/P03-1051.pdf) · [DOI 10.3115/1075096.1075147](https://doi.org/10.3115/1075096.1075147) · [OpenAlex](https://openalex.org/W2097341304) **Citations:** 138 citations (OpenAlex, 2026-10-14) · [43](https://md.research.elamur.ai/free/en/papers/lee-2003-language-model-based-arabic-word/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper tackles splitting Arabic words into their meaningful pieces. Arabic packs prepositions, articles, pronouns and agreement markers into a single orthographic word. The authors model every word as zero or more prefixes, one stem, and zero or more suffixes. A trigram language model over morphemes then picks the most likely split of a whole sentence. The model starts from a small hand-segmented LDC corpus of about 110K words and a table of known affixes. It is then grown without supervision: new stems are harvested from a 155-million-word raw corpus, judged by how closely they resemble stems already seen in training, and the model is retrained on the larger vocabulary. On a held-out set of 28,449 tokens, the segmenter reaches roughly 97% exact-match accuracy. Adding part-of-speech information in a joint model lowers the error rate a little further. The method matters because it needs only a modest seed corpus, an affix list and raw text, so it could transfer to other heavily inflected or agglutinative languages. ## Key points - Resources: an LDC hand-segmented corpus of about 140K tokens, split into training and development test data, plus a 155M-word unsegmented Arabic corpus. The affix table has 51 prefixes and 72 suffixes taken from LDC data, with 14 more prefixes and 122 more suffixes from IBM-Egypt. - Error rate falls as components are added. With 110K training words, the plain trigram LM has 11.0% error, adding the prefix-suffix filter brings it to 5.5%, and adding new stems brings it to 2.9%. With only 10K words, the same steps go from 26.0% to 8.5%. - Stem acquisition sharply cuts unknown words. In the 110K setup, LM vocabulary grows from 8,196 to 25,306 entries, and the out-of-vocabulary ratio drops from 5.8% to 1.9%. - Unseen stems are handled through an UNKNOWN class. Its probability is multiplied by a tiny constant, UNK_Fraction = 1e-9, chosen empirically. - A joint segmentation and POS model reduces the best error rate from 2.9% to 2.6%, roughly a 10% relative gain. - It targets cases that a morpheme LM alone cannot resolve, such as Alywm ("today" or the name Al-Youm vs. "the day"). - It also targets over-split foreign names like bwtyn (Putin). ## Interesting ideas - Build candidate splits only from a fixed affix table, but also break each matched compound affix into its parts (wA# → w# A#), then let a sequence model choose among the candidates; this keeps the search small while still reaching correct analyses that a flat lookup would miss. - Give out-of-vocabulary stems a shared UNKNOWN class with a heavily discounted probability, so new words can still be segmented without overwhelming known analyses. - Start from a small gold seed, use it to grow the lexicon from a huge raw corpus, then retrain on the expanded data; most of the gain shows up when the seed is tiny. - Sort errors into a few interpretable buckets (unknown stem, one notorious ambiguous token, everything else) and track how the shares shift as training size grows; this reveals which component should be improved next. - When a split depends on word class, score segmentation and POS tags together rather than one after the other. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (tokens): English clitics and contractions ("won't", "y'all'd", "'s") could be split this way, with candidates scored in context instead of resolved by fixed rules. - **I2** (tokens): a calibrated unknown-lemma class could let the morphology step fall back gracefully on rare words while keeping its decision traceable. - **I3** (learn): this matches the silver-data loop, and measuring the gain at several seed sizes (10K to 110K) would show where silver data pays off most. - **I4** (learn): evaluation reports could include such bucketed error tables, so repair rules and teacher effort go where the remaining errors actually sit. - **I5** (tokens): English "'s" (is / has / possessive) could be decided jointly with the tagger, avoiding errors that propagate into the UD tree. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [IBM](https://md.research.elamur.ai/free/en/companies/ibm/) — Company, Armonk, United States · 139 works · 70,691 citations - [IBM Research - Thomas J. Watson Research Center](https://md.research.elamur.ai/free/en/labs/ibm-research-thomas-j-watson-research-center/) — Research institute, Yorktown Heights, United States · 53 works · 49,193 citations - [IBM (Egypt)](https://md.research.elamur.ai/free/en/companies/ibm-egypt/) — Company, Giza, Egypt · 8 works · 301 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/) · 119 works ## Cite Cite as: Young-Suk Lee, Kishore Papineni, Salim Roukos, Ossama Emam and Hany Hassan. 2003. Language Model Based Arabic Word Segmentation. Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{lee2003language, title = {{Language Model Based Arabic Word Segmentation}}, author = {Young-Suk Lee, Kishore Papineni, Salim Roukos, Ossama Emam and Hany Hassan}, booktitle = {Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics}, year = {2003}, doi = {10.3115/1075096.1075147}, url = {https://doi.org/10.3115/1075096.1075147} } ``` - [← previous Coreference Resolution Using Competition Learning Approach](https://md.research.elamur.ai/free/en/papers/yang-2003-coreference-resolution-competition-learning-approach/) - [next → Supersense Tagging of Unknown Nouns in WordNet](https://md.research.elamur.ai/free/en/papers/ciaramita-2003-supersense-tagging-unknown-nouns-wordnet/) - [↑ back to papers of 2003](https://md.research.elamur.ai/free/en/papers/2003/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work. 1. Beesley, K. 1996. [Arabic Finite-State Morphological Analysis and Generation](https://md.research.elamur.ai/free/en/papers/beesley-1996-arabic-finite-state-morphological-analysis/). Proceedings of COLING-96, pages 89− 94. *Updated 29 October 2026*