--- title: "Arabic Dialect Identification Using BERT Fine-Tuning (2020)" description: "The paper describes a submission to sub-task 1 of the NADI 2020 shared task, which asks a system to… 5 references, cited by 5 papers here." authors: ["Moataz Mansour", "Moustafa Tohamy", "Zeyad Ezzat", "Marwan Torki"] year: 2020 venue: "Proceedings of the Fifth Arabic Natural Language Processing Workshop" source: "https://aclanthology.org/2020.wanlp-1.33/" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/mansour-2020-arabic-dialect-identification-bert-fine/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mansour-2020-arabic-dialect-identification-bert-fine/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/8/) / Arabic Dialect Identification Using BERT Fine-Tuning # Arabic Dialect Identification Using BERT Fine-Tuning **Authors:** Moataz Mansour, Moustafa Tohamy, Zeyad Ezzat and [Marwan Torki](https://md.research.elamur.ai/free/en/authors/marwan-torki/) · **Year:** 2020 · **Venue:** [Proceedings of the Fifth Arabic Natural Language Processing Workshop](https://md.research.elamur.ai/free/en/venues/wanlp/) ([WANLP 2020](https://md.research.elamur.ai/free/en/venues/wanlp/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2020.wanlp-1.33/) · [PDF](https://aclanthology.org/2020.wanlp-1.33.pdf) · [OpenAlex](https://openalex.org/W3121063397) **Citations:** 9 citations (OpenAlex, 2026-10-14) · [5](https://md.research.elamur.ai/free/en/papers/mansour-2020-arabic-dialect-identification-bert-fine/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper describes a submission to sub-task 1 of the NADI 2020 shared task, which asks a system to guess which of 21 Arab countries a tweet's dialect comes from. The labelled data is small and very skewed, and many neighbouring dialects, especially in the Gulf, are hard to tell apart. The authors tried two approaches. The first was a classical baseline: TF-IDF features over word unigrams and character n-grams, fed to an equal-weight voting ensemble of linear SVM, RBF SVM and Bernoulli Naive Bayes. The second started from an Arabic BERT checkpoint from Safaya et al. (2020). They first continued its language-model training on the 10 million unlabelled tweets supplied by the organisers, then fine-tuned a classification head on the labelled set. BERT clearly beat the classical ensemble. Adapting the language model to tweets gave a further small gain. The final system scored an F1 of 24.05 in development and 22.85 in testing, which placed it 5th among 18 teams. The work shows that adapting a pretrained model to the target domain helps on a small, noisy classification task. ## Key points - Data (NADI 2020): 21,000 labelled training tweets across 21 countries, 4,957 dev, 5,000 test, plus 10M unlabelled tweets. The authors used 19,950 tweets for training and held out 1,050 for validation. - Class imbalance is severe. Egypt makes up about 21.3% of the training tweets, while Bahrain is nearly absent. The similar Gulf dialects together account for roughly 37%. - Classical ensemble: word-unigram TF-IDF plus word-boundary character (2,9)-grams, with linear SVM, RBF SVM and Bernoulli NB in a voting classifier. It scored 18.1 dev F1. - Language-model adaptation ran for one epoch over the 10M tweets, split into ten sessions of 1M tweets each. Inputs were cut or padded to 10 words. Training took about 120 h on a Tesla M60 with AdamW (lr 2e-5). - The classification head was batch norm with tanh, dropout 0.1 and a linear layer, trained for 4 epochs. - Without language-model adaptation, BERT reached 23.5 dev F1. - With adaptation, it reached 24.05 dev F1 and 22.85 on test. - Most Gulf tweets were labelled as Saudi Arabia, and no Bahrain tweet was classified correctly. ## Interesting ideas - Before task fine-tuning, continue language-model pretraining on unlabelled text from the target genre, because the original pretraining corpus was stylistically far from tweets. - Report a cheap, carefully tuned character n-gram baseline next to the neural model, so the real gain from the expensive system can be measured. - Read the confusion matrix, not just the F1 score. Here it showed the model folding a whole confusable cluster into its majority member and never predicting the rarest class. - The authors propose (but did not test) a coarse-to-fine scheme: first pick a dialect group (Sham, Gulf, Nile, other), then the exact country. This lets the model use similarity between classes instead of fighting it. - Break a huge unlabelled training run into fixed-size sessions, such as 1M items per session, so it fits modest hardware and can be checkpointed. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (tree): gparse could first run masked-LM training on raw fables, tales and school texts, then be trained on treebanks, to shrink the genre gap. - **I2** (learn): every gparse or teacher-tier result could be paired with a perceptron-parser or rule baseline, which acts as a sanity reference alongside the negative controls. - **I3** (learn): per-label confusion tables for deprels or VerbNet roles would expose classes the model quietly collapses into a dominant neighbour. - **I4** (frames): semantic role or verb-class assignment could first choose a coarse VerbNet class family and then the fine class, keeping each decision explainable. - **I5** (learn): large silver-data or self-play training passes could run in resumable chunks, with evaluation between chunks. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works ## Cite Cite as: Moataz Mansour, Moustafa Tohamy, Zeyad Ezzat and Marwan Torki. 2020. Arabic Dialect Identification Using BERT Fine-Tuning. Proceedings of the Fifth Arabic Natural Language Processing Workshop. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{mansour2020arabic, title = {{Arabic Dialect Identification Using BERT Fine-Tuning}}, author = {Moataz Mansour, Moustafa Tohamy, Zeyad Ezzat and Marwan Torki}, booktitle = {Proceedings of the Fifth Arabic Natural Language Processing Workshop}, year = {2020}, url = {https://aclanthology.org/2020.wanlp-1.33/} } ``` - [← previous The Effect of Sociocultural Variables on Sarcasm Communication Online](https://md.research.elamur.ai/free/en/papers/oprea-2020-effect-sociocultural-variables-sarcasm-communication/) - [next → ARBML: Democratizing Arabic Natural Language Processing Tools](https://md.research.elamur.ai/free/en/papers/alyafeai-2020-arbml-democratizing-arabic-natural-language/) - [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/8/) ## References 5 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020. [NADI 2020: The First Nuanced Arabic Dialect Identification Shared Task](https://md.research.elamur.ai/free/en/papers/mageed-2020-nadi-2020-first-nuanced-arabic/). In Proceedings of the Fifth Arabic Natural Language Processing Workshop (WANLP 2020), Barcelona, Spain. [↗](https://doi.org/10.48550/arxiv.2010.11334) 2. Kathrein Abu Kwaik and Motaz Saad. 2019. [ArbDialectID at MADAR shared task 1: Language modelling and ensemble learning for fine grained Arabic dialect identification](https://md.research.elamur.ai/free/en/papers/kwaik-2019-arbdialectid-madar-shared-task-1/). In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 254–258, Florence, Italy, August. Association for Computational Linguistics. 3. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Houda+Bouamor%2C+Nizar+Habash%2C+Mohammad+Salameh%2C+Wajdi+Zaghouani%2C+Owen+Rambow%2C+Dana+Abdulrahim%2C+Ossama+Obeid%2C+Salam+Khalifa%2C+Fadhl+Eryani%2C+Alexander+Erdmann%2C+and+Kemal+Oflazer%2E+2018%2E+The+MADAR+Arabic+di) 4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). CoRR, abs/1810.04805. [↗](https://arxiv.org/abs/1810.04805) 5. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. [Kuisail at semeval-2020 task 12: Bert-cnn for offensive speech identification in social media](https://md.research.elamur.ai/free/en/papers/safaya-2020-kuisail-semeval-2020-task-12/). In Proceedings of the International Workshop on Semantic Evaluation (SemEval). *Updated 1 November 2026*