---
title: "Multi-dialect Arabic BERT for Country-level Dialect Identification"
description: "The paper describes the system by team Mawdoo3 AI that took first place in subtask 1 of… 26.78 F1. 29 references, cited by 33 papers here."
authors: ["Bashar Talafha", "Mohammad Ali", "Muhy Eddin Za'ter", "Haitham Seelawi", "Ibraheem Tuffaha", "Mostafa Samir", "Wael Farhan", "Hussein Al-Natsheh"]
year: 2020
venue: "Proceedings of the Fifth Arabic Natural Language Processing Workshop"
source: "https://aclanthology.org/2020.wanlp-1.10/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/talafha-2020-multi-dialect-arabic-bert-country/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/talafha-2020-multi-dialect-arabic-bert-country/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/) / Multi-dialect Arabic BERT for Country-level Dialect Identification

# Multi-dialect Arabic BERT for Country-level Dialect Identification

**Authors:** [Bashar Talafha](https://md.research.elamur.ai/free/en/authors/bashar-talafha/), Mohammad Ali, [Muhy Eddin Za'ter](https://md.research.elamur.ai/free/en/authors/muhy-eddin-za-ter/), [Haitham Seelawi](https://md.research.elamur.ai/free/en/authors/haitham-seelawi/), [Ibraheem Tuffaha](https://md.research.elamur.ai/free/en/authors/ibraheem-tuffaha/), [Mostafa Samir](https://md.research.elamur.ai/free/en/authors/mostafa-samir/), [Wael Farhan](https://md.research.elamur.ai/free/en/authors/wael-farhan/) and [Hussein Al-Natsheh](https://md.research.elamur.ai/free/en/authors/hussein-al-natsheh/) · **Year:** 2020 · **Venue:** [Proceedings of the Fifth Arabic Natural Language Processing Workshop](https://md.research.elamur.ai/free/en/venues/wanlp/) ([WANLP 2020](https://md.research.elamur.ai/free/en/venues/wanlp/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.wanlp-1.10/) · [PDF](https://aclanthology.org/2020.wanlp-1.10.pdf) · [OpenAlex](https://openalex.org/W3119989085)

**Citations:** 44 citations (OpenAlex, 2026-10-14) · [33](https://md.research.elamur.ai/free/en/papers/talafha-2020-multi-dialect-arabic-bert-country/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper describes the system by team Mawdoo3 AI that took first place in subtask 1 of the NADI shared task. That subtask asks a model to assign a tweet to one of 21 Arab countries according to its dialect. The task is hard because Arabic varieties lack a shared spelling standard, are morphologically rich, and neighbouring dialects overlap heavily.

The organizers supplied 21,000 labelled training tweets and IDs for 10M unlabelled tweets. The authors started from the public ArabicBERT model and continued its masked-language-model pre-training on the unlabelled tweets. They then fine-tuned it as a classifier over the [CLS] vector. Finally, they averaged the class probabilities of four fine-tuned runs that differed in maximum input length.

The resulting ensemble reached 26.78 F1 on the test set. Every classic machine-learning and embedding baseline they tried scored clearly lower. The adapted encoder is released publicly as Multi-dialect-Arabic-BERT. The work shows that cheap in-domain continued pre-training of an existing model pays off for fine-grained dialect classification.

## Key points
- **Data:** NADI subtask 1 provides 21,000 training tweets and 10,000 tweets split equally between dev and test, each labelled with a single country.
  - Egypt is the most frequent class and Bahrain the rarest.
  - The authors managed to fetch 97.7% of the 10M unlabelled tweets.
- **Model:** The base is ArabicBERT, trained on about 93 GB of Arabic text, with a 32,000-token sentencepiece vocabulary.
  - It was further pre-trained for 3 epochs on the tweets.
  - Fine-tuning used Adam with a learning rate of 3.75 × 10⁻⁵, batch size 16, and 3 epochs, with no preprocessing beyond the tokenizer.
- **Ensemble:** Four runs with maximum sequence lengths of 80, 90, 100 and 250 were combined by element-wise averaging of their softmax outputs.
  - The ensemble scored 27.58 F1 on dev and 26.78 F1 (42.86 accuracy) on test.
  - A single Multi-dialect-Arabic-BERT model scored 26 F1 on dev, and plain ArabicBERT scored 24.45.
- **Baselines:** Classic approaches, including MADAR-Mawdoo3, MADAR-Safina, MADAR-JUST and TF–IDF with SVM/LR/NB, peaked at about 17 F1 on dev.
  - Pooled FastText or AraVec embeddings peaked at 20.86 F1 on dev.
- **Rules and errors:** Hand-made lexicon rules raised dev F1 to 29.03 but nudged test F1 down to 26.77.
  - The confusion matrix shows bias toward large classes and mix-ups between neighbouring countries; for example, half of Bahrain dev tweets were predicted as UAE.

## Interesting ideas
- Instead of training a new model, they continued masked-LM training of an existing encoder on unlabelled text from the target genre, which gave a sizeable gain at low cost.
- The ensemble members differ only in one cheap setting (maximum input length), and their probability distributions are simply averaged, so diversity comes almost for free.
- Hand-crafted rules that improved the dev score but slightly hurt the test score show that rules tuned on a development set quietly overfit it.
- Examining the confusion matrix revealed tweets where the model was right and the gold label was wrong, including tweets that were actually MSA rather than any dialect.
- The MADAR-Safina baseline duplicates every word before extracting character n-grams so that patterns spanning a word's end and start (circumfixes) become visible.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tree): gparse's encoder could get a short continued pre-training pass on the raw genre it must parse (fables, school texts, word problems) before parser fine-tuning.
- **I2** (tree): several gparse runs with different truncation lengths or seeds could have their arc and label scores averaged, and the spread between runs could also serve as an explainable confidence signal.
- **I3** (learn): self-induced repair rules should be accepted only after a check on a separate held-out split, as a built-in negative control against dev-set overfitting.
- **I4** (learn): confident disagreements between witness parsers and a treebank can be routed to review as candidate annotation errors rather than counted as model errors.
- **I5** (tokens): a similar boundary-wrapping trick could enrich character n-gram features for guessing the tag and lemma of unknown words, capturing prefix–suffix combinations jointly.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works

## Cite

Cite as: Bashar Talafha, Mohammad Ali, Muhy Eddin Za'ter, Haitham Seelawi, Ibraheem Tuffaha, Mostafa Samir, Wael Farhan and Hussein Al-Natsheh. 2020. Multi-dialect Arabic BERT for Country-level Dialect Identification. Proceedings of the Fifth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{talafha2020multi,
  title = {{Multi-dialect Arabic BERT for Country-level Dialect Identification}},
  author = {Bashar Talafha, Mohammad Ali, Muhy Eddin Za'ter, Haitham Seelawi, Ibraheem Tuffaha, Mostafa Samir, Wael Farhan and Hussein Al-Natsheh},
  booktitle = {Proceedings of the Fifth Arabic Natural Language Processing Workshop},
  year = {2020},
  url = {https://aclanthology.org/2020.wanlp-1.10/}
}
```

- [← previous QADiscourse - Discourse Relations as QA Pairs: Representation, Crowdsourcing and Baselines](https://md.research.elamur.ai/free/en/papers/pyatkin-2020-qadiscourse-discourse-relations-as-qa/)
- [next → A survey on Recognizing Textual Entailment as an NLP Evaluation](https://md.research.elamur.ai/free/en/papers/poliak-2020-survey-recognizing-textual-entailment-as/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/)

## References

29 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Muhammad Abdul-Mageed, Hassan Alhuzali, and Mohamed Elaraby. 2018. [You tweet what you speak: A city-level dataset of arabic dialects](https://md.research.elamur.ai/free/en/papers/mageed-2018-you-tweet-what-you-speak/). In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
2. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020. [NADI 2020: The First Nuanced Arabic Dialect Identification Shared Task](https://md.research.elamur.ai/free/en/papers/mageed-2020-nadi-2020-first-nuanced-arabic/). In Proceedings of the Fifth Arabic Natural Language Processing Workshop (WANLP 2020), Barcelona, Spain. [↗](https://doi.org/10.48550/arxiv.2010.11334)
3. Mohamed Ali. 2018. [Character level convolutional neural network for arabic dialect identification](https://md.research.elamur.ai/free/en/papers/ali-2018-character-level-convolutional-neural-network/). In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018), pages 122–127.
4. Mohammed Matuq Ashi, Muazzam Ahmed Siddiqui, and Farrukh Nadeem. 2018. Pre-trained word embeddings for arabic aspect-based sentiment analysis of airline tweets. In International Conference on Advanced Intelligent Systems and Informatics, pages 241–251. Springer. [find](https://scholar.google.com/scholar?q=Mohammed+Matuq+Ashi%2C+Muazzam+Ahmed+Siddiqui%2C+and+Farrukh+Nadeem%2E+2018%2E+Pre%2Dtrained+word+embeddings+for+arabic+aspect%2Dbased+sentiment+analysis+of+airline+tweets%2E+In+International+Conference+on+Advanced)
5. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. [Enriching word vectors with subword information](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/). Transactions of the Association for Computational Linguistics, 5:135–146. [↗](https://doi.org/10.1162/tacl_a_00051)
6. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, et al. 2018. The madar arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). [find](https://scholar.google.com/scholar?q=Houda+Bouamor%2C+Nizar+Habash%2C+Mohammad+Salameh%2C+Wajdi+Zaghouani%2C+Owen+Rambow%2C+Dana+Abdulrahim%2C+Ossama+Obeid%2C+Salam+Khalifa%2C+Fadhl+Eryani%2C+Alexander+Erdmann%2C+et+al%2E+2018%2E+The+madar+arabic+dialect+corpus)
7. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. [The madar shared task on arabic fine-grained dialect identification](https://md.research.elamur.ai/free/en/papers/bouamor-2019-madar-shared-task-arabic-fine/). In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 199–207.
8. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. [↗](https://arxiv.org/abs/2003.10555)
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805)
10. Mohamed Elaraby and Muhammad Abdul-Mageed. 2018. [Deep models for arabic dialect identification on benchmarked data](https://md.research.elamur.ai/free/en/papers/elaraby-2018-deep-models-arabic-dialect-identification/). In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2018), pages 263–274.
11. Heba Elfardy and Mona Diab. 2013. [Sentence level dialect identification in arabic](https://md.research.elamur.ai/free/en/papers/elfardy-2013-sentence-level-dialect-identification-arabic/). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 456–461.
12. Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. [Scalable modified Kneser-Ney language model estimation](https://md.research.elamur.ai/free/en/papers/heafield-2013-scalable-modified-kneser-ney-language/). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 690–696, Sofia, Bulgaria, August. Association for Computational Linguistics.
13. Fei Huang. 2015. [Improved arabic dialect classification with social media data](https://md.research.elamur.ai/free/en/papers/huang-2015-improved-arabic-dialect-classification-social/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2118–2126.
14. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759. [↗](https://arxiv.org/abs/1607.01759)
15. Diederik P Kingma and Jimmy Ba. 2014. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980. [↗](https://arxiv.org/abs/1412.6980)
16. Taku Kudo and John Richardson. 2018. [Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/). arXiv preprint arXiv:1808.06226. [↗](https://arxiv.org/abs/1808.06226)
17. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. [Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). arXiv preprint arXiv:1301.3781. [↗](https://arxiv.org/abs/1301.3781)
18. Ahmad Ragab, Haitham Seelawi, Mostafa Samir, Abdelrahman Mattar, Hesham Al-Bataineh, Mohammad Zaghloul, Ahmad Mustafa, Bashar Talafha, Abed Alhakim Freihat, and Hussein Al-Natsheh. 2019. [Mawdoo3 ai at madar shared task: Arabic fine-grained dialect identification with ensemble learning](https://md.research.elamur.ai/free/en/papers/ragab-2019-mawdoo3-ai-madar-shared-task/). In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 244–248.
19. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. [Kuisail at semeval-2020 task 12: Bert-cnn for offensive speech identification in social media](https://md.research.elamur.ai/free/en/papers/safaya-2020-kuisail-semeval-2020-task-12/). In Proceedings of the International Workshop on Semantic Evaluation (SemEval).
20. Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018. [Fine-grained arabic dialect identification](https://md.research.elamur.ai/free/en/papers/salameh-2018-fine-grained-arabic-dialect-identification/). In Proceedings of the 27th International Conference on Computational Linguistics, pages 1332–1344.
21. Abu Bakr Soliman, Kareem Eissa, and Samhaa R El-Beltagy. 2017. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. [find](https://scholar.google.com/scholar?q=Abu+Bakr+Soliman%2C+Kareem+Eissa%2C+and+Samhaa+R+El%2DBeltagy%2E+2017%2E+Aravec%3A+A+set+of+arabic+word+embedding+models+for+use+in+arabic+nlp%2E+Procedia+Computer+Science%2C+117%3A256%E2%80%93265%2E)
22. Bashar Talafha, Ali Fadel, Mahmoud Al-Ayyoub, Yaser Jararweh, AL-Smadi Mohammad, and Patrick Juola. 2019a. [Team just at the madar shared task on arabic fine-grained dialect identification](https://md.research.elamur.ai/free/en/papers/talafha-2019-team-just-madar-shared-task/). In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 285–289. [↗](https://doi.org/10.60692/br9mc-4qp64)
23. Bashar Talafha, Wael Farhan, Ahmed Altakrouri, and Hussein Al-Natsheh. 2019b. [Mawdoo3 ai at madar shared task: Arabic tweet dialect identification](https://md.research.elamur.ai/free/en/papers/talafha-2019-mawdoo3-ai-madar-shared-task/). In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 239–243.
24. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in neural information processing systems, pages 5998–6008.
25. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). In Advances in neural information processing systems, pages 5753–5763. [↗](https://doi.org/10.48550/arxiv.1906.08237)
26. Wajdi Zaghouani and Anis Charfi. 2018. Arap-tweet: A large multi-dialect twitter corpus for gender, age and language variety identification. arXiv preprint arXiv:1808.07674. [↗](https://arxiv.org/abs/1808.07674)
27. Omar F Zaidan and Chris Callison-Burch. 2014. [Arabic dialect identification](https://md.research.elamur.ai/free/en/papers/zaidan-2014-arabic-dialect-identification/). Computational Linguistics, 40(1):171–202. [↗](https://doi.org/10.1162/coli_a_00169)
28. Chiyu Zhang and Muhammad Abdul-Mageed. 2019. [No army, no navy: Bert semi-supervised learning of arabic dialects](https://md.research.elamur.ai/free/en/papers/zhang-2019-no-army-no-navy-bert/). In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 279–284.
29. Vladimir Zolotov and David Kung. 2017. Analysis and optimization of fasttext linear text classifier. arXiv preprint arXiv:1702.05531. [↗](https://arxiv.org/abs/1702.05531)

*Updated 1 November 2026*