--- title: "Build Fast and Accurate Lemmatization for Arabic — Mubarak 2018" description: "The author builds a fast open-source Java lemmatizer. 12 references, cited by 7 papers here." authors: ["Hamdy Mubarak"] year: 2018 venue: "Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)" source: "https://aclanthology.org/L18-1181/" doi: "10.63317/2a5pftey6kdr" licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)" licence_url: "https://creativecommons.org/licenses/by-nc/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/mubarak-2018-build-fast-accurate-lemmatization-arabic/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mubarak-2018-build-fast-accurate-lemmatization-arabic/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/) / Build Fast and Accurate Lemmatization for Arabic # Build Fast and Accurate Lemmatization for Arabic **Authors:** [Hamdy Mubarak](https://md.research.elamur.ai/free/en/authors/hamdy-mubarak/) · **Year:** 2018 · **Venue:** Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) **Link:** [aclanthology.org/L18-1181](https://aclanthology.org/L18-1181/) · library `lingbulk-2018-mubarak-build-fast-accurate-lemmatization-arabic` · **Licence:** [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) (ELRA, LREC proceedings) — summary only [**18** citations](https://openalex.org/works?filter=cites:W2962692626) OpenAlex, 2026-10-14 · [**7** citations in our library](https://md.research.elamur.ai/free/en/papers/mubarak-2018-build-fast-accurate-lemmatization-arabic/cited-by/) of the 140,884 papers whose bibliographies we read · [Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)](https://md.research.elamur.ai/free/en/venues/lrec/) · published by [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · [licence](https://creativecommons.org/licenses/by-nc/4.0/) - [Publisher's page](https://aclanthology.org/L18-1181/) - [PDF](https://aclanthology.org/L18-1181.pdf) - [DOI 10.63317/2a5pftey6kdr](https://doi.org/10.63317/2a5pftey6kdr) - [OpenAlex](https://openalex.org/W2962692626) **Authors (1)** 1. [Mubarak, Hamdy](https://openalex.org/A5102924560) [ORCID](https://orcid.org/0000-0002-9051-6240) · [Semantic Scholar](https://www.semanticscholar.org/author/143779235) · h 30 · 3,692 citations · 113 works Citations and author metrics: OpenAlex, 2026-10-14; authors with the same name are not always told apart. ## What it is about Arabic lemmatization differs from surface stemming: it must restore the citation form, which can require changing stem patterns, not only cutting affixes. The author builds a fast open-source Java lemmatizer. For each undiacritized word it takes the most frequent diacritized form from a 9.7-million-word corpus, about 73% MSA, and picks the Buckwalter analysis whose diacritized form matches best. Lemma order is fixed by corpus usage where analyses tie. A new test set of 70 WikiNews articles (18,300 words) is lemmatized by an expert and scored on undiacritized lemmas: 73.68% for the Farasa surface stemmer, 96.61% for MADAMIRA and 97.32% for the new system. The paper's text claims +7% relative over MADAMIRA, but the table gap is under one point. Speed: 7.4 million words in 2 minutes, against 2.5 hours for MADAMIRA. The test set is NC, so it is for measurement only. ## Ideas - I2. Order lemmas by usage in the lexicon, not in the code: where two analyses share a form (the paper's car/walker case), the lemma with more corpus occurrences comes first, and the order is stored in the dictionary builder. ## For Mova - I1. For unpointed Hebrew, in `tools/hbo` unpointed mode: when a consonantal form has several analyses, prefer the one whose pointed form is most frequent in a pointed corpus (MACULA gives the pointed forms). It is table-based and needs no context, as the paper does. Where: [languages/hbo/10-lexicon.md](https://mova.elamur.ai/free/en/docs/languages/hbo/10-lexicon/). - I3. Report lemma accuracy on undiacritized text, with an error table in the paper's style (POS error, named-entity segmentation error), as a test for `tools/hbo`. Build the set from our own open data; do not carry the NC set into a release. I1 goes into `tools/hbo` (unpointed mode) and [languages/hbo/10-lexicon.md](https://mova.elamur.ai/free/en/docs/languages/hbo/10-lexicon/). I2 goes into the dictionary builder. I3 is the test for `tools/hbo`. 📚 Shelves 5[Computer Science · 7,349 works](https://md.research.elamur.ai/free/en/shelves/computer-science/) · [Artificial Intelligence · 7,183 works](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · [Arts and Humanities · 7,418 works](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · [Language and Linguistics · 7,357 works](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · [Information Systems · 402 works](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) 🧭 Topics 2[Semitic languages · 1,336 works](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · [Morphology · 245 works](https://md.research.elamur.ai/free/en/topics/morphology/) ## Cite Cite as Hamdy Mubarak. 2018. Build Fast and Accurate Lemmatization for Arabic. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.63317/2a5pftey6kdr](https://doi.org/10.63317/2a5pftey6kdr) ```bibtex @inproceedings{mubarak2018build, title = {{Build Fast and Accurate Lemmatization for Arabic}}, author = {Hamdy Mubarak}, booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)}, year = {2018}, doi = {10.63317/2a5pftey6kdr}, url = {https://doi.org/10.63317/2a5pftey6kdr} } ``` - [← previous A Probabilistic Annotation Model for Crowdsourcing Coreference](https://md.research.elamur.ai/free/en/papers/paun-2018-probabilistic-annotation-model-crowdsourcing-coreference/) - [next → Creating a Translation Matrix of the Bible's Names Across 591 Languages](https://md.research.elamur.ai/free/en/papers/wu-2018-creating-translation-matrix-bible-s/) - [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/) ## References 12 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Abdelali, A., Darwish, K., Durrani, N., and Mubarak, H. (2016). Farasa: A fast and furious segmenter for arabic. In HLT-NAACL Demos, pages 11–16. [our card](https://md.research.elamur.ai/free/en/papers/abdelali-2016-farasa-fast-furious-segmenter-arabic/) 2. Beesley, K. (1996). Arabic finite-state morphological analysis and generation. In In COLING-96: Proceedings of the 16th international, pages 89–94. [find](https://scholar.google.com/scholar?q=Beesley%2C+K%2E+%281996%29%2E+Arabic+finite%2Dstate+morphological+analysis+and+generation%2E+In+In+COLING%2D96%3A+Proceedings+of+the+16th+international%2C+pages+89%E2%80%9394%2E) 3. Buckwalter, T. (2002). Arabic finite-state morphological analysis and generation. In [http://members.aol.com/ArabicLexicons/](http://members.aol.com/ArabicLexicons/). Figure 3: Summary of lemmatization algorithm Figure 4: Lemmatization online demo (part of Farasa Arabic NLP tools) [↗](http://members.aol.com/ArabicLexicons/) 4. Darwish, K. and Mubarak, H. (2016). Farasa: A new fast and accurate arabic word segmenter. In LREC. [find](https://scholar.google.com/scholar?q=Darwish%2C+K%2E+and+Mubarak%2C+H%2E+%282016%29%2E+Farasa%3A+A+new+fast+and+accurate+arabic+word+segmenter%2E+In+LREC%2E) 5. Darwish, K., Mubarak, H., and Abdelali, A. (2017). Arabic diacritization: Stats, rules, and hacks. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 9–17. [our card](https://md.research.elamur.ai/free/en/papers/darwish-2017-arabic-diacritization-stats-rules-hacks/) 6. Dichy, J. and Fargaly, A. (2003). Roots and patterns vs. stems plus grammar-lexis specifications: on what basis should a multilingual lexical database centred on arabic be built? In Proceedings of the MTSummit, New-Orleans. [find](https://scholar.google.com/scholar?q=Dichy%2C+J%2E+and+Fargaly%2C+A%2E+%282003%29%2E+Roots+and+patterns+vs%2E+stems+plus+grammar%2Dlexis+specifications%3A+on+what+basis+should+a+multilingual+lexical+database+centred+on+arabic+be+built%3F+In+Proceedings+of+the) 7. El-Shishtawy, T. and Al-Sammak, A. (2009). Arabic keyphrase extraction using linguistic knowledge and machine learning techniques. In Proceedings of the Second International Conference on Arabic Language Resources and Tools, The MEDAR Consortium. [find](https://scholar.google.com/scholar?q=El%2DShishtawy%2C+T%2E+and+Al%2DSammak%2C+A%2E+%282009%29%2E+Arabic+keyphrase+extraction+using+linguistic+knowledge+and+machine+learning+techniques%2E+In+Proceedings+of+the+Second+International+Conference+on+Arabic+Langu) 8. El-Shishtawy, T. and El-Ghannam, F. (2012). An accurate arabic root-based lemmatizer for information retrieval purposes. arXiv preprint arXiv:1203.3584. [↗](https://arxiv.org/abs/1203.3584) [our card](https://md.research.elamur.ai/free/en/papers/el-shishtawy-2012-accurate-arabic-root-based-lemmatizer/) 9. Graff, D., Maamouri, M., Bouziri, B., Krouna, S., Kulick, S., and Buckwalter, T. (2009). Standard arabic morphological analyzer (sama) version 3.1. In Linguistic Data Consortium LDC2009E73. [find](https://scholar.google.com/scholar?q=Graff%2C+D%2E%2C+Maamouri%2C+M%2E%2C+Bouziri%2C+B%2E%2C+Krouna%2C+S%2E%2C+Kulick%2C+S%2E%2C+and+Buckwalter%2C+T%2E+%282009%29%2E+Standard+arabic+morphological+analyzer+%28sama%29+version+3%2E1%2E+In+Linguistic+Data+Consortium+LDC2009E73%2E) 10. Khoja, S. (1999). Stemming arabic text. In Computing Department, Lancaster University. [find](https://scholar.google.com/scholar?q=Khoja%2C+S%2E+%281999%29%2E+Stemming+arabic+text%2E+In+Computing+Department%2C+Lancaster+University%2E) 11. Pasha, A., Al-Badrashiny, M., Diab, M., El Kholy, A., Eskander, R., Habash, N., Pooleery, M., Rambow, O., and Roth, R. M. (2014). Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic. Proc. LREC. [our card](https://md.research.elamur.ai/free/en/papers/pasha-2014-madamira-fast-comprehensive-tool-morphological/) 12. Plisson, J., Lavrac, N., and Mladenic, M. (2004). A rule based approach to word lemmatization. In researchgate.net. [find](https://scholar.google.com/scholar?q=Plisson%2C+J%2E%2C+Lavrac%2C+N%2E%2C+and+Mladenic%2C+M%2E+%282004%29%2E+A+rule+based+approach+to+word+lemmatization%2E+In+researchgate%2Enet%2E) Updated 23 October 2026