--- title: "DIWAN: A Dialectal Word Annotation Tool for Arabic (2015)" description: "The authors present DIWAN, a Windows desktop tool for word-level morphological annotation of… 15 references, cited by 12 papers here." authors: ["Faisal Al-Shargi", "Owen Rambow"] year: 2015 venue: "Proceedings of the Second Workshop on Arabic Natural Language Processing" source: "https://aclanthology.org/W15-3206/" doi: "10.18653/v1/w15-3206" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/shargi-2015-diwan-dialectal-word-annotation-tool/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shargi-2015-diwan-dialectal-word-annotation-tool/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/) / DIWAN: A Dialectal Word Annotation Tool for Arabic # DIWAN: A Dialectal Word Annotation Tool for Arabic **Authors:** [Faisal Al-Shargi](https://md.research.elamur.ai/free/en/authors/faisal-al-shargi/) and [Owen Rambow](https://md.research.elamur.ai/free/en/authors/owen-rambow/) · **Year:** 2015 · **Venue:** [Proceedings of the Second Workshop on Arabic Natural Language Processing](https://md.research.elamur.ai/free/en/venues/workshop-on-arabic-natural-language-processing/) ([Workshop on Arabic Natural Language Processing 2015](https://md.research.elamur.ai/free/en/venues/workshop-on-arabic-natural-language-processing/2015/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/W15-3206/) · [PDF](https://aclanthology.org/W15-3206.pdf) · [DOI 10.18653/v1/w15-3206](https://doi.org/10.18653/v1/w15-3206) · [OpenAlex](https://openalex.org/W2250358209) **Citations:** 53 citations (OpenAlex, 2026-10-14) · [12](https://md.research.elamur.ai/free/en/papers/shargi-2015-diwan-dialectal-word-annotation-tool/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Arabic dialects have few annotated resources. Tools built for Modern Standard Arabic (MSA) perform poorly on dialectal text, which often lacks spelling norms and mixes freely with MSA. The authors present DIWAN, a Windows desktop tool for word-level morphological annotation of dialectal Arabic. Its main design idea is that dialects share much of their morphology and lexicon with MSA and with each other, so the tool should not treat each one as an unrelated language. DIWAN preloads each token with a contextual analysis from MADAMIRA and lets the annotator browse candidate readings from SAMA (MSA) and CALIMA-EGY (Egyptian). The annotator can accept, edit, borrow or build analyses from scratch. Annotations record a conventionalized CODA spelling alongside the original orthography and keep the text order. The resulting data can therefore train both context-free morphological analyzers and in-context taggers. The tool has been used for Palestinian, Yemeni and Moroccan corpora. The authors do not report any controlled evaluation of annotation speed or quality. ## Key points - The tool builds on three existing resources: SAMA 3.1 (MSA analyzer), CALIMA-EGY (Egyptian analyzer) and MADAMIRA (contextual tagger for MSA and Egyptian). - Words can be queued for annotation in three ways: - by frequency; - by text order; - by "coverage", which puts first the words MADAMIRA cannot analyze, since these are likely dialect-specific. - Each annotated token in the output records: - CODA spelling, lexeme, Buckwalter morpheme split and POS; - MSA equivalent and English gloss; - functional gender and number; - dialect subregion; - provenance: which resource was used, whether it was modified, and the search key. - Corpus sizes: - Palestinian: 45,000 annotated tokens; annotators became fluent with the tool after about 100 words. - Yemeni (Sana'ani): 32,325 tokens. - Moroccan: 64,171 words collected. - Roles are split between administrators and annotators. Data is stored in three tables (D_sentences, D_madamira, D_result), and annotators can work offline with batch upload later. ## Interesting ideas - A "coverage" queue puts tokens that existing analyzers cannot handle at the front, which steers human effort toward truly new material first. - Analysis by substitution: when a word gets no analysis, the annotator looks up a known word with the same inflectional pattern and swaps only the stem and gloss. This turns a hard from-scratch task into a quick edit. - Each annotation carries provenance fields: which source supplied it, whether a human changed it, and what query was used to retrieve it. This lets you later measure how often each source was right. - A normalized spelling is stored alongside the raw orthography instead of replacing it, so one dataset serves both noisy-input and canonical-form models. - The workflow starts with frequency-ordered annotation to cover many tokens quickly, then switches to text order to finish the long tail in context. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (tokens): Rank corpus words that the en morphology cannot analyze (slang, neologisms, domain terms) as a worklist for extending the lexicon. - **I2** (tokens): For unknown English words, copy the morphological analysis of a known analogue with the same affix pattern and swap in the new lemma, keeping the borrowing explainable. - **I3** (learn): Tag every silver token or arc with the witness parser or LLM tier that produced it and a "human-modified" flag, so the reliability of each source can be audited. - **I4** (tokens): Keep the raw surface token and a normalized form side by side for social-media or misspelled input, letting the tagger train on one and tests check the other. - **I5** (learn): Schedule teacher or human correction rounds by fixing high-frequency error types first, then sweeping through full documents for the remaining contextual cases. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Columbia University](https://md.research.elamur.ai/free/en/universities/columbia-university/) — University, New York, United States · 241 works · 54,053 citations - [Leipzig University](https://md.research.elamur.ai/free/en/universities/leipzig-university/) — University, Leipzig, Germany · 26 works · 8,622 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (2): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [🧩 Morphology](https://md.research.elamur.ai/free/en/topics/morphology/) · 405 works ## Cite Cite as: Faisal Al-Shargi and Owen Rambow. 2015. DIWAN: A Dialectal Word Annotation Tool for Arabic. Proceedings of the Second Workshop on Arabic Natural Language Processing. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{alshargi2015diwan, title = {{DIWAN: A Dialectal Word Annotation Tool for Arabic}}, author = {Faisal Al-Shargi and Owen Rambow}, booktitle = {Proceedings of the Second Workshop on Arabic Natural Language Processing}, year = {2015}, doi = {10.18653/v1/w15-3206}, url = {https://doi.org/10.18653/v1/w15-3206} } ``` - [← previous A Graph-based Lattice Dependency Parser for Joint Morphological Segmentation and Syntactic Analysis](https://md.research.elamur.ai/free/en/papers/seeker-2015-graph-based-lattice-dependency-parser/) - [next → A Hierarchy with, of, and for Preposition Supersenses](https://md.research.elamur.ai/free/en/papers/schneider-2015-hierarchy-preposition-supersenses/) - [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/) ## References 15 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Y. Benajiba and M. Diab. 2010. A web application for dialectal Arabic text annotation. In Proceedings of the LREC Workshop for Language Resources (LRs) and Human Language Technologies (HLT) for Semitic Languages: Status, Updates, and Prospects. [find](https://scholar.google.com/scholar?q=Y%2E+Benajiba+and+M%2E+Diab%2E+2010%2E+A+web+application+for+dialectal+Arabic+text+annotation%2E+In+Proceedings+of+the+LREC+Workshop+for+Language+Resources+%28LRs%29+and+Human+Language+Technologies+%28HLT%29+for+Semiti) 2. T. Buckwalter. 2004. Buckwalter Arabic Morphological Analyzer Version 2.0. Linguistic Data Consortium, University of Pennsylvania. LDC Catalog No.: LDC2004L02, ISBN 1-58563-324-0 [find](https://scholar.google.com/scholar?q=T%2E+Buckwalter%2E+2004%2E+Buckwalter+Arabic+Morphological+Analyzer+Version+2%2E0%2E+Linguistic+Data+Consortium%2C+University+of+Pennsylvania%2E+LDC+Catalog+No%2E%3A+LDC2004L02%2C+ISBN+1%2D58563%2D324%2D0) 3. D. Chiang, M. Diab, N. Habash, O. Rambow, and S. Shareef. 2006. Parsing Arabic Dialect. In Proceedings of the European chapter of the Association of Computational Linguistics (EACL). [find](https://scholar.google.com/scholar?q=D%2E+Chiang%2C+M%2E+Diab%2C+N%2E+Habash%2C+O%2E+Rambow%2C+and+S%2E+Shareef%2E+2006%2E+Parsing+Arabic+Dialect%2E+In+Proceedings+of+the+European+chapter+of+the+Association+of+Computational+Linguistics+%28EACL%29%2E) 4. M. Diab, N. Habash, O. Rambow, M. AlTantawy, and Y. Benajiba. 2010a. COLABA: Arabic Dialect Annotation and Processing. In Proceedings of the Language Resources (LRs) and Human Language Technologies (HLT) for Semitic Languages at LREC. [find](https://scholar.google.com/scholar?q=M%2E+Diab%2C+N%2E+Habash%2C+O%2E+Rambow%2C+M%2E+AlTantawy%2C+and+Y%2E+Benajiba%2E+2010a%2E+COLABA%3A+Arabic+Dialect+Annotation+and+Processing%2E+In+Proceedings+of+the+Language+Resources+%28LRs%29+and+Human+Language+Technologies+%28H) 5. M. Diab, K. Hacioglu, and D. Jurafsky. 2007. Automated Methods for Processing Arabic Text: From Tokenization to Base Phrase Chunking. In van den Bosch, A. and Soudi, A., editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods. Kluwer/Springer. [find](https://scholar.google.com/scholar?q=M%2E+Diab%2C+K%2E+Hacioglu%2C+and+D%2E+Jurafsky%2E+2007%2E+Automated+Methods+for+Processing+Arabic+Text%3A+From+Tokenization+to+Base+Phrase+Chunking%2E+In+van+den+Bosch%2C+A%2E+and+Soudi%2C+A%2E%2C+editors%2C+Arabic+Computational+) 6. D. Graff, M. Maamouri, B. Bouziri, S. Krouna, S. Kulick, and T. Buckwalter. 2009. Standard Arabic Morphological Analyzer (SAMA) Version 3.1. Linguistic Data Consortium LDC2009E73. [find](https://scholar.google.com/scholar?q=D%2E+Graff%2C+M%2E+Maamouri%2C+B%2E+Bouziri%2C+S%2E+Krouna%2C+S%2E+Kulick%2C+and+T%2E+Buckwalter%2E+2009%2E+Standard+Arabic+Morphological+Analyzer+%28SAMA%29+Version+3%2E1%2E+Linguistic+Data+Consortium+LDC2009E73%2E) 7. N. Habash and O. Rambow. 2005. [Arabic Tokenization, Part-of-Speech Tagging and Morphological Disambiguation in One Fell Swoop](https://md.research.elamur.ai/free/en/papers/habash-2005-arabic-tokenization-part-speech-tagging/). In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 573–580, Ann Arbor, Michigan. 8. N. Habash and O. Rambow. 2006. [MAGEAD: A Morphological Analyzer and Generator for the Arabic Dialects](https://md.research.elamur.ai/free/en/papers/habash-2006-magead-morphological-analyzer-generator-arabic/). In Proceedings of ACL, Sydney, Australia. [↗](https://doi.org/10.7916/d87s7x73) 9. N. Habash, M. Diab, and O. Rabmow. 2012a. [Conventional Orthography for Dialectal Arabic](https://md.research.elamur.ai/free/en/papers/habash-2012-conventional-orthography-dialectal-arabic/). In Proceedings of the Language Resources and Evaluation Conference (LREC), Istanbul. 10. N. Habash, R. Eskander, and A. Hawwari. 2012b. [A Morphological Analyzer for Egyptian Arabic](https://md.research.elamur.ai/free/en/papers/habash-2012-morphological-analyzer-egyptian-arabic/). In Proc. of the Special Interest Group on Computational Morphology and Phonology, Montréal, Canada. 11. N. Habash, O. Rambow, and R. Roth. 2009. MADA+TOKAN: A toolkit for Arabic tokenization, diacritization, morphological disambiguation, POS tagging, stemming and lemmatization. In Choukri, K. and Maegaard, B., editors, Proceedings of the Second International Conference on Arabic Language Resources and Tools. The MEDAR Consortium, April. [find](https://scholar.google.com/scholar?q=N%2E+Habash%2C+O%2E+Rambow%2C+and+R%2E+Roth%2E+2009%2E+MADA%2BTOKAN%3A+A+toolkit+for+Arabic+tokenization%2C+diacritization%2C+morphological+disambiguation%2C+POS+tagging%2C+stemming+and+lemmatization%2E+In+Choukri%2C+K%2E+and+Maegaa) 12. N. Habash, R. Roth, O. Rambow, R. Eskander, and N. Tomeh. 2013. [Morphological Analysis and Disambiguation for Dialectal Arabic](https://md.research.elamur.ai/free/en/papers/habash-2013-morphological-analysis-disambiguation-dialectal-arabic/). In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Atlanta, GA. 13. M. Jarrar, N. Habash, D. Akra, and N. Zalmout 2014. "[Building a Corpus for Palestinian Arabic: a Preliminary Study](https://md.research.elamur.ai/free/en/papers/jarrar-2014-building-corpus-palestinian-arabic-preliminary/)." In Proceedings of the Workshop on Arabic Natural Language Processing (ANLP 2014). 14. M. Maamouri, A. Bies, S. Kulick, M. Ciul, N. Habash and R. Eskander. 2014. Developing a dialectal Egyptian Arabic Treebank: Impact of Morphology and Syntax on Annotation and Tool Development. In Proceedings of the Lan- guage Resources and Evaluation Conference (LREC), Reykjavik, Iceland. [find](https://scholar.google.com/scholar?q=M%2E+Maamouri%2C+A%2E+Bies%2C+S%2E+Kulick%2C+M%2E+Ciul%2C+N%2E+Habash+and+R%2E+Eskander%2E+2014%2E+Developing+a+dialectal+Egyptian+Arabic+Treebank%3A+Impact+of+Morphology+and+Syntax+on+Annotation+and+Tool+Development%2E+In+Proce) 15. A. Pasha, M. Al-Badrashiny, M. Diab, A. El Kholy, R. Eskander, N. Habash, M. Pooleery, O. Rambow and R. M. Roth. [MADAMIRA: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Arabic](https://md.research.elamur.ai/free/en/papers/pasha-2014-madamira-fast-comprehensive-tool-morphological/). In Proc. of LREC, Reykjavik, Iceland, 2014. *Updated 1 November 2026*