---
title: "Arabic dialect identification: An Arabic-BERT model with data"
description: "The paper describes the ArabicProcessors entry to the NADI 2020 shared task. 18 references, cited by 7 papers here."
authors: ["Kamel Gaanoun", "Imade Benelallam"]
year: 2020
venue: "Proceedings of the Fifth Arabic Natural Language Processing Workshop"
source: "https://aclanthology.org/2020.wanlp-1.28/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gaanoun-2020-arabic-dialect-identification-arabic-bert/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gaanoun-2020-arabic-dialect-identification-arabic-bert/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/8/) / Arabic dialect identification: An Arabic-BERT model with data augmentation and ensembling strategy

# Arabic dialect identification: An Arabic-BERT model with data augmentation and ensembling strategy

**Authors:** [Kamel Gaanoun](https://md.research.elamur.ai/free/en/authors/kamel-gaanoun/) and [Imade Benelallam](https://md.research.elamur.ai/free/en/authors/imade-benelallam/) · **Year:** 2020 · **Venue:** [Proceedings of the Fifth Arabic Natural Language Processing Workshop](https://md.research.elamur.ai/free/en/venues/wanlp/) ([WANLP 2020](https://md.research.elamur.ai/free/en/venues/wanlp/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.wanlp-1.28/) · [PDF](https://aclanthology.org/2020.wanlp-1.28.pdf)

**Citations:** [7](https://md.research.elamur.ai/free/en/papers/gaanoun-2020-arabic-dialect-identification-arabic-bert/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper describes the ArabicProcessors entry to the NADI 2020 shared task. The task is to identify the dialect of Arabic tweets at two levels: country (Subtask 1, 21 classes) and province (Subtask 2, 100 classes). The labelled data is small and skewed toward a few countries.

The team built a three-stage pipeline:
1. **Fine-tuning.** They fine-tuned Arabic-BERT, which beat AraBERT on their dev set.
2. **Data augmentation.** First, they reshuffled tweet fragments within each country. Second, they ran a staged semi-supervised loop over the organisers' Unlabeled-10M tweets, keeping only high-confidence pseudo-labels with per-class thresholds.
3. **Ensembling.** They combined the resulting models with hard voting and weighted soft voting, searching over model subsets.

The baseline was a classical XGBoost model at 15.41% macro F1 on dev; the pipeline reached about 25% on dev. On test it ranked 3rd in Subtask 1 (23.26% F1) and 2nd in Subtask 2 (5.75%). The work shows that a language-specific BERT plus careful pseudo-labelling beats classical classifiers on fine-grained dialect ID. The gains were largest for under-represented classes.

## Key points
- **Data:** TRAIN has 21,000 tweets and DEV 4,957; TEST has 5,000 unlabeled tweets. Of the 9,999,978 Unlabeled-10M tweet IDs, only 6,815,470 could still be fetched. Removing conflicting or non-dialectal duplicates cut 674 tweets from TRAIN.
- **Models on dev (Subtask 1):** Naive Bayes reached 8.58%, Logistic Regression 14.30%, XGBoost 15.41% and Arabic-BERT 23.45%. Fine-tuning used 3 epochs, batch size 80, learning rate 2e-5 and max length 128.
- **Semi-supervised loop:**
  - The unlabeled set was split into 5 parts.
  - Pseudo-labels were kept above the 99th softmax percentile for Egypt, Iraq and Saudi Arabia, above the 80th for other countries, and above the 90th for provinces.
  - In total, 1,056,984 tweets were added.
- **Ensembles (dev / test, Subtask 1):**
  - Basic hard voting: 24.15% / 22.52%.
  - Modified hard voting: 24.32% / 23.26%, the best on test.
  - Bayesian-weighted soft voting: 25.01% / 23.03%, the best on dev.
  - For Subtask 2, ensembling did not help.
- **Remaining weaknesses:** minority labels improved more than majority ones (+1.67 vs +1.10 points). Even so, Bahrain, Qatar and Kuwait were essentially never predicted, and MSA tweets remained a source of confusion.

## Interesting ideas
- Pseudo-label confidence thresholds can be set per class: very strict for classes the model already handles well, looser for rare ones, so augmentation flows mainly toward weak classes.
- Self-training can run chunk by chunk with a gate at each round. Each round trains one candidate from the base model and one from the current model, and keeps the result only if it beats the previous best on dev.
- Before training, drop identical texts that carry different labels, plus boilerplate produced by external services. Such items teach contradictions rather than signal.
- When combining several models, search every subset and tune vote weights, but note that the dev-optimal ensemble here did worse on test than a simpler one.
- Majority voting needs an explicit tie-break rule, because a library's default (here, the smallest label id) silently biases outcomes.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): When harvesting silver trees, lower the agreement bar for rare constructions or relations and keep it high for frequent ones, so silver data fills gaps rather than reinforcing what is already learned.
- **I2** (learn): Silver-data retraining rounds could adopt the same accept-only-if-better gate on a fixed dev set, which limits drift and makes each round's contribution auditable.
- **I3** (learn): Scan treebank and silver corpora for identical sentences with divergent analyses and for templated, machine-generated text, then quarantine them before training the parsers.
- **I4** (tree): When combining witness parsers such as the perceptron parser and gparse, judge any tuned combination on a held-out split it was not tuned on, to avoid overfitting the dev set.
- **I5** (tokens): When several taggers or lemmatizers vote, define and log a principled tie-break, such as the more frequent tag or the more confident witness, so the decision stays explainable.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works

🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Kamel Gaanoun and Imade Benelallam. 2020. Arabic dialect identification: An Arabic-BERT model with data augmentation and ensembling strategy. Proceedings of the Fifth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{gaanoun2020arabic,
  title = {{Arabic dialect identification: An Arabic-BERT model with data augmentation and ensembling strategy}},
  author = {Kamel Gaanoun and Imade Benelallam},
  booktitle = {Proceedings of the Fifth Arabic Natural Language Processing Workshop},
  year = {2020},
  url = {https://aclanthology.org/2020.wanlp-1.28/}
}
```

- [← previous An Online Readability Leveled Arabic Thesaurus](https://md.research.elamur.ai/free/en/papers/jiang-2020-online-readability-leveled-arabic-thesaurus/)
- [next → Arabic Opinion Mining Using a Hybrid Recommender System Approach](https://md.research.elamur.ai/free/en/papers/harrag-2020-arabic-opinion-mining-using-hybrid/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/8/)

## References

18 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abdul-Mageed, M., Zhang, C., Bouamor, H. & Habash, N. (2020), [NADI 2020: The First Nuanced Arabic Dialect Identification Shared Task](https://md.research.elamur.ai/free/en/papers/mageed-2020-nadi-2020-first-nuanced-arabic/), in ‘Proceedings of the Fifth Arabic Natural Language Processing Workshop (WANLP2020)’, Barcelona, Spain.
2. Antoun, W., Baly, F. & Hajj, H. (2020), [Arabert: Transformer-based model for arabic language understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/), in ‘LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020’, p. 9.
3. Aridhi, C., Hadhemi, A., Souissi, E. & Younes, J. (2017), Word-level identification of romanized tunisian dialect, pp. 170–175. [find](https://scholar.google.com/scholar?q=Aridhi%2C+C%2E%2C+Hadhemi%2C+A%2E%2C+Souissi%2C+E%2E+%26+Younes%2C+J%2E+%282017%29%2C+Word%2Dlevel+identification+of+romanized+tunisian+dialect%2C+pp%2E+170%E2%80%93175%2E)
4. Bouamor, H., Hassan, S. & Habash, N. (2019), [The MADAR shared task on Arabic fine-grained dialect identification](https://md.research.elamur.ai/free/en/papers/bouamor-2019-madar-shared-task-arabic-fine/), in ‘Proceedings of the Fourth Arabic Natural Language Processing Workshop’, Association for Computational Linguistics, Florence, Italy, pp. 199–207. https://www.aclweb.org/anthology/W19-4622 [↗](https://aclanthology.org/W19-4622/)
5. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. (2019), [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/), in ‘Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)’, Association for Computational Linguistics, Minneapolis, Minnesota, pp. 4171–4186. https://www.aclweb.org/anthology/N19-1423 [↗](https://aclanthology.org/N19-1423/)
6. El-Khair, I. A. (2016), ‘[1.5 billion words arabic corpus](https://md.research.elamur.ai/free/en/papers/el-khair-2016-1-5-billion-words-arabic/)’, ArXiv abs/1611.04033. [↗](https://arxiv.org/abs/1611.04033)
7. Elfardy, H. & Diab, M. (2013), [Sentence level dialect identification in arabic](https://md.research.elamur.ai/free/en/papers/elfardy-2013-sentence-level-dialect-identification-arabic/), Vol. 2, pp. 456–461.
8. Habash, N. Y. (2010), Introduction to Arabic Natural Language Processing, Morgan & Claypool. [https://ieeexplore.ieee.org/document/6813521](https://ieeexplore.ieee.org/document/6813521) [↗](https://ieeexplore.ieee.org/document/6813521)
9. ITU (2019), Measuring digital development, facts and figures, Technical report, International Telecommunication Union. [find](https://scholar.google.com/scholar?q=ITU+%282019%29%2C+Measuring+digital+development%2C+facts+and+figures%2C+Technical+report%2C+International+Telecommunication+Union%2E)
10. Kim, Y. (2014), [Convolutional neural networks for sentence classification](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/), in ‘Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)’, Association for Computational Linguistics, Doha, Qatar, pp. 1746–1751. https://www.aclweb.org/anthology/D14-1181 [↗](https://aclanthology.org/D14-1181/)
11. Lulu, L. & Elnagar, A. (2018), ‘Automatic arabic dialect classification using deep learning models’, Procedia Computer Science 142, 262–269. [find](https://scholar.google.com/scholar?q=Lulu%2C+L%2E+%26+Elnagar%2C+A%2E+%282018%29%2C+%E2%80%98Automatic+arabic+dialect+classification+using+deep+learning+models%E2%80%99%2C+Procedia+Computer+Science+142%2C+262%E2%80%93269%2E)
12. Safaya, A., Abdullatif, M. & Yuret, D. (2020), [Kuisail at semeval-2020 task 12: Bert-cnn for offensive speech identification in social media](https://md.research.elamur.ai/free/en/papers/safaya-2020-kuisail-semeval-2020-task-12/), in ‘Proceedings of the International Workshop on Semantic Evaluation (SemEval)’.
13. Salameh, M., Bouamor, H. & Habash, N. (2018), [Fine-grained Arabic dialect identification](https://md.research.elamur.ai/free/en/papers/salameh-2018-fine-grained-arabic-dialect-identification/), in ‘Proceedings of the 27th International Conference on Computational Linguistics’, Association for Computational Linguistics, Santa Fe, New Mexico, USA, pp. 1332–1344. https://www.aclweb.org/anthology/C18-1113 [↗](https://aclanthology.org/C18-1113/)
14. Shon, S., Ali, A., Samih, Y., Mubarak, H. & Glass, J. (2020), Adi17: A fine-grained arabic dialect identification dataset. [find](https://scholar.google.com/scholar?q=Shon%2C+S%2E%2C+Ali%2C+A%2E%2C+Samih%2C+Y%2E%2C+Mubarak%2C+H%2E+%26+Glass%2C+J%2E+%282020%29%2C+Adi17%3A+A+fine%2Dgrained+arabic+dialect+identification+dataset%2E)
15. Zaidan, O. & Callison-Burch, C. (2013), ‘[Arabic dialect identification](https://md.research.elamur.ai/free/en/papers/zaidan-2014-arabic-dialect-identification/)’, Computational Linguistics . [↗](https://doi.org/10.1162/coli_a_00169)
16. Zampieri, M., Malmasi, S., Ljubešic, N., Nakov, P., Ali, A., Tiedemann, J., Scherrer, Y. & Aepli, N. (2017), [Findings of the VárDial evaluation campaign 2017, in ‘Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial)’, Association for Computational Linguistics, Valencia, Spain, pp](https://md.research.elamur.ai/free/en/papers/zampieri-2017-findings-vardial-evaluation-campaign/). 1–15. https://www.aclweb.org/anthology/W17-1201 [↗](https://aclanthology.org/W17-1201/)
17. Zhang, C. & Abdul-Mageed, M. (2019), [No army, no navy: BERT semi-supervised learning of Arabic dialects](https://md.research.elamur.ai/free/en/papers/zhang-2019-no-army-no-navy-bert/), in ‘Proceedings of the Fourth Arabic Natural Language Processing Workshop’, Florence, Italy.
18. Zheng, S. & Yang, M. (2019), A New Method of Improving BERT for Text Classification, pp. 442–452. [find](https://scholar.google.com/scholar?q=Zheng%2C+S%2E+%26+Yang%2C+M%2E+%282019%29%2C+A+New+Method+of+Improving+BERT+for+Text+Classification%2C+pp%2E+442%E2%80%93452%2E)

*Updated 1 November 2026*