--- title: "Arabic Dialects Identification for All Arabic countries — Aliwy 2020" description: "The authors therefore build a five-stage pipeline: Normalization. 12 references, cited by 3 papers here." authors: ["Ahmed Aliwy", "Hawraa Taher", "Zena AboAltaheen"] year: 2020 venue: "Proceedings of the Fifth Arabic Natural Language Processing Workshop" source: "https://aclanthology.org/2020.wanlp-1.32/" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/aliwy-2020-arabic-dialects-identification-all-arabic/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/aliwy-2020-arabic-dialects-identification-all-arabic/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/8/) / Arabic Dialects Identification for All Arabic countries # Arabic Dialects Identification for All Arabic countries **Authors:** Ahmed Aliwy, Hawraa Taher and Zena AboAltaheen · **Year:** 2020 · **Venue:** [Proceedings of the Fifth Arabic Natural Language Processing Workshop](https://md.research.elamur.ai/free/en/venues/wanlp/) ([WANLP 2020](https://md.research.elamur.ai/free/en/venues/wanlp/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2020.wanlp-1.32/) · [PDF](https://aclanthology.org/2020.wanlp-1.32.pdf) · [OpenAlex](https://openalex.org/W3119933877) **Citations:** 8 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/aliwy-2020-arabic-dialects-identification-all-arabic/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper addresses country-level Arabic dialect identification for Twitter text, covering all 21 Arab countries. This is subtask 1 of the NADI 2020 shared task. The NADI tweets are noisy: they include Persian and Kurdish posts, as well as MSA and Classical Arabic mixed in with dialect. The authors therefore build a five-stage pipeline: 1. Normalization. 2. Noise filtering, using a Naïve Bayes foreign-language detector and a semi-supervised k-means step that separates dialectal from non-dialectal tweets. 3. Removal of standard-Arabic clitics and stop words, keeping dialect-specific ones. 4. Features: TF-IDF words plus selected prefixes and suffixes. 5. Classification by a vote of Naïve Bayes, Logistic Regression and Decision Tree. Voting is applied either as one 21-way classifier or as 21 one-vs-rest detectors. Ambiguous cases go to a specialist model trained only on the candidate countries. On the development set, the best setup (binary detectors plus MSA-cleaned training data) reaches a macro F-measure of 52.38. The test scores are much lower, which highlights how hard fine-grained dialect identification is when labels and data are noisy. ## Key points - **Data:** the NADI Twitter set has 21,000 training, 4,957 development and 5,000 test tweets. Each tweet is labeled by country (21 classes) and by province (100 classes). Roughly 504 training tweets were found to be non-Arabic. - **Foreign-tweet detector:** Naïve Bayes using word unigrams and character bigrams. On 132 development foreign tweets it scored P 0.985, R 1.0, F 0.992. - **Dev F-measure:** 27.17 for voting without clustering (the official submission), 41.34 with clustering, and 52.38 for 21 binary detectors with clustering. The matching test scores are 12.45, 17.61 and 20.05. - **Forced labels for evaluation:** foreign tweets were labeled Iraq, MSA tweets Egypt, and tweets rejected by all detectors Egypt. The authors note this lowers macro scores. - **Error sources identified by the authors:** - Labels reflect user location rather than actual dialect. - Neighbouring dialects are close to each other. - Some cities speak like another country (e.g., Basrah sounds closer to Kuwait). - Some tweets look like MSA but could be read in many dialects. ## Interesting ideas - About 2,000 hand-labeled tweets serve as fixed anchors in k-means, so cluster membership spreads to unlabeled data while the seeds never move; this cleans a noisy training set cheaply. - A cascade first runs one-vs-rest detectors; only when several fire does a specialist trained solely on those candidates decide, and when none fire the input is explicitly "unknown". - Normalization strips standard-language clitics and stop words but deliberately keeps dialect-specific clitics and function words, because those carry the signal. - The authors make the cost of forcing out-of-scope inputs (foreign, MSA) into real classes visible, noting it drags down macro scores. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): Clamped-seed clustering could filter silver parses or frames before training, with a small trusted gold core anchoring which disputed examples to drop. - **I2** (tokens): The tagger could resolve hard ambiguities by calling a small model trained only on the competing tags, keeping each decision local and explainable. - **I3** (tokens): Tokenizer and lemmatizer normalization should keep register and dialect markers (contractions, nonstandard forms) as features instead of erasing them before parsing. - **I4** (learn): Evaluations should keep an explicit "out of scope / no answer" bucket and report it separately, similar to a negative control, so forced guesses do not hide real accuracy. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works 🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works ## Cite Cite as: Ahmed Aliwy, Hawraa Taher and Zena AboAltaheen. 2020. Arabic Dialects Identification for All Arabic countries. Proceedings of the Fifth Arabic Natural Language Processing Workshop. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{aliwy2020arabic, title = {{Arabic Dialects Identification for All Arabic countries}}, author = {Ahmed Aliwy, Hawraa Taher and Zena AboAltaheen}, booktitle = {Proceedings of the Fifth Arabic Natural Language Processing Workshop}, year = {2020}, url = {https://aclanthology.org/2020.wanlp-1.32/} } ``` - [← previous AccentDB: A Database of Non-Native English Accents to Assist Neural Speech Recognition](https://md.research.elamur.ai/free/en/papers/ahamad-2020-accentdb-database-non-native-english/) - [next → DRS at MRP 2020: Dressing up Discourse Representation Structures as Graphs](https://md.research.elamur.ai/free/en/papers/abzianidze-2020-drs-mrp-2020-dressing-up/) - [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/8/) ## References 12 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Itani. M. 2018 "Sentiment analysis and resources for informal Arabic text on social media." PhD diss., Sheffield Hallam University. [find](https://scholar.google.com/scholar?q=Itani%2E+M%2E+2018+%22Sentiment+analysis+and+resources+for+informal+Arabic+text+on+social+media%2E%22+PhD+diss%2E%2C+Sheffield+Hallam+University%2E) 2. El-Haj M., Ryson P., and Aboelezz M., 2018 “[Arabic Dialect Identification in the Context of Bivalency and Code-Switching](https://md.research.elamur.ai/free/en/papers/haj-2018-arabic-dialect-identification-context-bivalency/),” Eleventh international conference on language Resources and Evaluation LREC 2018. 3. Belgacem, M., Antoniadis, G., & Besacier, L. 2010. [Automatic Identification of Arabic Dialects](https://md.research.elamur.ai/free/en/papers/belgacem-2010-automatic-identification-arabic-dialects/). In LREC2010. 4. Darwish, K., Sajjad, H., & Mubarak, H., 2014. [Verifiably effective Arabic dialect identification](https://md.research.elamur.ai/free/en/papers/darwish-2014-verifiably-effective-arabic-dialect-identification/). In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 1465-1468). 5. Elfardy, H., & Diab, M. (2013, August). [Sentence level dialect identification in Arabic](https://md.research.elamur.ai/free/en/papers/elfardy-2013-sentence-level-dialect-identification-arabic/). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (pp. 456-461). 6. Cotterell, R., & Callison-Burch, C. 2014. [A Multi-Dialect, Multi-Genre Corpus of Informal Written Arabic](https://md.research.elamur.ai/free/en/papers/cotterell-2014-multi-dialect-multi-genre-corpus/). In LREC (pp. 241-245). 7. Sadat, F., Kazemi, F., & Farzindar, A. 2014. [Automatic identification of Arabic language varieties and dialects in social media](https://md.research.elamur.ai/free/en/papers/sadat-2014-automatic-identification-arabic-language-varieties/). In Proceedings of the Second Workshop on Natural Language Processing for Social Media (SocialNLP) (pp. 22-27). 8. Malmasi, S., & Zampieri, M. 2016. [Arabic dialect identification in speech transcripts](https://md.research.elamur.ai/free/en/papers/malmasi-2016-arabic-dialect-identification-speech-transcripts/). In Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3) (pp. 106-113). 9. Mishra, P., & Mujadia, V. 2019, [Arabic Dialect Identification for Travel and Twitter Text](https://md.research.elamur.ai/free/en/papers/mishra-2019-arabic-dialect-identification-travel-twitter/). In Proceedings of the Fourth Arabic Natural Language Processing Workshop (pp. 234-238). 10. Bouamor, H., Hassan, S., & Habash, N. 2019. “[The MADAR shared task on Arabic fine-grained dialect identification](https://md.research.elamur.ai/free/en/papers/bouamor-2019-madar-shared-task-arabic-fine/). In Proceedings of the Fourth Arabic Natural Language Processing Workshop (pp. 199-207). 11. Salameh, M., Bouamor, H., & Habash, N. 2018. [Fine-grained Arabic dialect identification](https://md.research.elamur.ai/free/en/papers/salameh-2018-fine-grained-arabic-dialect-identification/). In Proceedings of the 27th International Conference on Computational Linguistics (pp. 1332-1344). 12. Abdul-Mageed, Muhammad, Zhang, Chiyu, Bouamor, Houda and Habash, Nizar 2020 “The Shared Task on Nuanced Arabic Dialect Identification (NADI)”, Proceedings of the Fifth Arabic Natural Language Processing Workshop (WANLP2020), Barcelona, Spain [find](https://scholar.google.com/scholar?q=Abdul%2DMageed%2C+Muhammad%2C+Zhang%2C+Chiyu%2C+Bouamor%2C+Houda+and+Habash%2C+Nizar+2020+%E2%80%9CThe+Shared+Task+on+Nuanced+Arabic+Dialect+Identification+%28NADI%29%E2%80%9D%2C+Proceedings+of+the+Fifth+Arabic+Natural+Language+Processi) *Updated 2 November 2026*