---
title: "BERT-based Multi-Task Model for Country and Province Level"
description: "This paper describes a system for the second NADI shared task. 18 references, cited by 14 papers here."
authors: ["Abdellah El Mekki", "Abdelkader El Mahdaouy", "Kabil Essefar", "Nabil El Mamoun", "Ismail Berrada", "Ahmed Khoumsi"]
year: 2021
venue: "Proceedings of the Sixth Arabic Natural Language Processing Workshop"
source: "https://aclanthology.org/2021.wanlp-1.31/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mekki-2021-bert-based-multi-task-model/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mekki-2021-bert-based-multi-task-model/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/6/) / BERT-based Multi-Task Model for Country and Province Level MSA and Dialectal Arabic Identification

# BERT-based Multi-Task Model for Country and Province Level MSA and Dialectal Arabic Identification

**Authors:** [Abdellah El Mekki](https://md.research.elamur.ai/free/en/authors/abdellah-el-mekki/), [Abdelkader El Mahdaouy](https://md.research.elamur.ai/free/en/authors/abdelkader-el-mahdaouy/), [Kabil Essefar](https://md.research.elamur.ai/free/en/authors/kabil-essefar/), [Nabil El Mamoun](https://md.research.elamur.ai/free/en/authors/nabil-el-mamoun/), [Ismail Berrada](https://md.research.elamur.ai/free/en/authors/ismail-berrada/) and [Ahmed Khoumsi](https://md.research.elamur.ai/free/en/authors/ahmed-khoumsi/) · **Year:** 2021 · **Venue:** [Proceedings of the Sixth Arabic Natural Language Processing Workshop](https://md.research.elamur.ai/free/en/venues/wanlp/) ([WANLP 2021](https://md.research.elamur.ai/free/en/venues/wanlp/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.wanlp-1.31/) · [PDF](https://aclanthology.org/2021.wanlp-1.31.pdf) · [OpenAlex](https://openalex.org/W3155312056)

**Citations:** 12 citations (OpenAlex, 2026-10-14) · [14](https://md.research.elamur.ai/free/en/papers/mekki-2021-bert-based-multi-task-model/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This paper describes a system for the second NADI shared task. The task asks a model to guess where an Arabic tweet comes from, at both country and province level, for Modern Standard Arabic (MSA) and for Dialectal Arabic (DA). The organizers released 31,000 labeled tweets covering 21 countries and 100 provinces. The authors fine-tune MARBERT, a BERT variant pretrained on Arabic Twitter, inside a multi-task network that predicts country and province together. One encoder is shared by both tasks. Each task has its own attention pooling over token embeddings, and its output is joined with the [CLS] vector before a small classifier. Training minimizes the sum of the two cross-entropy losses. Compared with a single-task model of the same design, the joint model scores higher on every subtask on both dev and test, though absolute scores stay modest, especially for provinces. Error analysis shows the main difficulties: rare countries, closely related Gulf dialects, and MSA, which carries little regional signal. The work suggests that predicting coarse and fine labels jointly helps fine-grained dialect identification.

## Key points
- NADI 2021 has four subtasks: 1.1 country-level MSA, 1.2 country-level DA, 2.1 province-level MSA and 2.2 province-level DA. The data split is 21,000 tweets for training, with 10,000 divided equally between dev and test.
- MARBERT was pretrained with the MLM objective only, on roughly 1 billion Arabic tweets. The authors chose it to stay close to the target domain.
- Training setup:
  - Adam optimizer with learning rate 1e-5, batch size 16 and 5 epochs.
  - Preprocessing copies MARBERT's pipeline: diacritics are removed and mentions are replaced with a USER token.
- Test macro-F1, multi-task vs single-task:
  - 21.47 vs 20.90 on subtask 1.1
  - 30.64 vs 29.07 on subtask 1.2
  - 5.35 vs 4.72 on subtask 2.1
  - 7.32 vs 5.30 on subtask 2.2
  - Test accuracy on country-level DA reached 50.30.
- Adding attention pooling on top of [CLS] helped both the single-task and the multi-task models, compared with classifying from [CLS] alone.

## Interesting ideas
- Each task gets its own attention pooling over a shared encoder, and the result is joined with the global [CLS] vector, so every head sees both shared and task-specific evidence.
- A coarse label (country) and a fine label (province) are trained together with a plain summed loss, so the easier label steadies learning of the sparse one.
- The single-task baseline uses exactly the same architecture minus the sharing, so any gain can be traced to multi-task learning alone.
- Errors are explained by two factors: how many training examples a class has, and how similar it is to its neighbors (Gulf dialects, low-signal MSA).
- The encoder is pretrained on the same genre as the target, and fine-tuning reuses the encoder's own preprocessing, which avoids a mismatch between pretraining and fine-tuning input.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tree): gparse could give its auxiliary heads (e.g. sentence-type or genre detection) their own attention pooling while keeping one shared transformer, without the heads interfering.
- **I2** (tokens): the en tagger could predict coarse UPOS jointly with fine morphological features, so that rare feature combinations borrow signal from the coarse tag.
- **I3** (learn): when testing a new training trick, compare against a control that differs only in that one component, so improvements stay explainable.
- **I4** (learn): parser and tagger confusions could be reported by label frequency and by closeness between labels, separating data scarcity from true ambiguity.
- **I5** (tree): gparse fine-tuning should apply exactly the text normalization its pretrained encoder saw, and prefer encoders pretrained on text close to the fables and school texts Mova reads.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Abdellah El Mekki, Abdelkader El Mahdaouy, Kabil Essefar, Nabil El Mamoun, Ismail Berrada and Ahmed Khoumsi. 2021. BERT-based Multi-Task Model for Country and Province Level MSA and Dialectal Arabic Identification. Proceedings of the Sixth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mekki2021bert,
  title = {{BERT-based Multi-Task Model for Country and Province Level MSA and Dialectal Arabic Identification}},
  author = {Abdellah {El Mekki}, Abdelkader {El Mahdaouy}, Kabil Essefar, Nabil {El Mamoun}, Ismail Berrada and Ahmed Khoumsi},
  booktitle = {Proceedings of the Sixth Arabic Natural Language Processing Workshop},
  year = {2021},
  url = {https://aclanthology.org/2021.wanlp-1.31/}
}
```

- [← previous Presuppositional exhaustification](https://md.research.elamur.ai/free/en/papers/bassi-2021-presuppositional-exhaustification/)
- [next → English-language dubbing: challenges and quality standards of an emerging localisation trend](https://md.research.elamur.ai/free/en/papers/miggiani-2021-english-language-dubbing-challenges-quality/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/6/)

## References

18 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. ## Conclusion In this paper, we introduced our submitted system to the second NADI shared task. We proposed an MTL model for joint country-level and province-Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020a. [ARBERT & MARBERT: Deep bidirectional transformers for arabic](https://md.research.elamur.ai/free/en/papers/abdul-mageed-2021-arbert-marbert-deep-bidirectional-transformers/). arXiv preprint arXiv:2101.01785. [↗](https://arxiv.org/abs/2101.01785)
2. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020b. [NADI 2020: The First Nuanced Arabic Dialect Identification Shared Task](https://md.research.elamur.ai/free/en/papers/mageed-2020-nadi-2020-first-nuanced-arabic/). In Proceedings of the Fifth Arabic Natural Language Processing Workshop (WANLP 2020), Barcelona, Spain. [↗](https://doi.org/10.48550/arxiv.2010.11334)
3. Nizar Habash, Mona Diab, and Owen Rambow. 2012. [Conventional orthography for dialectal Arabic](https://md.research.elamur.ai/free/en/papers/habash-2012-conventional-orthography-dialectal-arabic/). In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), pages 711–718, Istanbul, Turkey. European Languages Resources Association (ELRA).
4. Muhammad Abdul-Mageed, Chiyu Zhang, Abdel-Rahim Elmadany, Houda Bouamor, and Nizar Habash. 2021. [NADI 2021: The Second Nuanced Arabic Dialect Identification Shared Task](https://md.research.elamur.ai/free/en/papers/mageed-2021-nadi-2021-second-nuanced-arabic/). In Proceedings of the Sixth Arabic Natural Language Processing Workshop (WANLP 2021). [↗](https://doi.org/10.48550/arxiv.2103.08466)
5. Nizar Y Habash. 2010. Introduction to Arabic natural language processing, volume 3. Morgan & Claypool Publishers. [find](https://scholar.google.com/scholar?q=Nizar+Y+Habash%2E+2010%2E+Introduction+to+Arabic+natural+language+processing%2C+volume+3%2E+Morgan+%26+Claypool+Publishers%2E)
6. Muhammad Abdul-Mageed, Chiyu Zhang, Abdel-Rahim Elmadany, and Lyle Ungar. 2020c. Toward micro-dialect identification in diaglossic and code-switched environments. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5855–5876, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Muhammad+Abdul%2DMageed%2C+Chiyu+Zhang%2C+Abdel%2DRahim+Elmadany%2C+and+Lyle+Ungar%2E+2020c%2E+Toward+micro%2Ddialect+identification+in+diaglossic+and+code%2Dswitched+environments%2E+In+Proceedings+of+the+2020+Conference)
7. Kathrein Abu Kwaik and Motaz Saad. 2019. [ArbDi-alectID at MADAR shared task 1: Language modelling and ensemble learning for fine grained Arabic dialect identification](https://md.research.elamur.ai/free/en/papers/salameh-2018-fine-grained-arabic-dialect-identification/). In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 254–258, Florence, Italy. Association for Computational Linguistics.
8. Imad A. Al-Sughaiyer and Ibrahim A. Al-Kharashi. 2004. Arabic morphological analysis techniques: A comprehensive survey. Journal of the American Society for Information Science and Technology, 55(3):189–213. [↗](https://doi.org/10.1002/asi.10368)
9. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). CoRR, abs/1409.0473. [↗](https://arxiv.org/abs/1409.0473)
10. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. [The MADAR shared task on Arabic fine-grained dialect identification](https://md.research.elamur.ai/free/en/papers/bouamor-2019-madar-shared-task-arabic-fine/). In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 199–207.
11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423)
12. Abdellah El Mekki, Ahmed Alami, Hamza Alami, Ahmed Khoumsi, and Ismail Berrada. 2020. [Weighted combination of BERT and n-GRAM features for nuanced Arabic dialect identification](https://md.research.elamur.ai/free/en/papers/mekki-2020-weighted-combination-bert-n-gram/). In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 268–274, Barcelona, Spain (Online). Association for Computational Linguistics.
13. Diederik P. Kingma and Jimmy Ba. 2014. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). Cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015. [↗](https://arxiv.org/abs/1412.6980)
14. Karima Meftouh, Karima Abidi, Salima Harrat, and Kamel Smaili. 2019. [The SMarT classifier for Arabic fine-grained dialect identification](https://md.research.elamur.ai/free/en/papers/meftouh-2019-smart-classifier-arabic-fine-grained/). In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 259–263, Florence, Italy. Association for Computational Linguistics.
15. Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018. [Fine-grained Arabic dialect identification](https://md.research.elamur.ai/free/en/papers/salameh-2018-fine-grained-arabic-dialect-identification/). In Proceedings of the 27th International Conference on Computational Linguistics, pages 1332–1344, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
16. Bashar Talafha, Mohammad Ali, Muhy Eddin Za’ter, Haitham Seelawi, Ibraheem Tuffaha, Mostafa Samir, Wael Farhan, and Hussein Al-Natsheh. 2020. [Multi-dialect Arabic BERT for country-level dialect identification](https://md.research.elamur.ai/free/en/papers/talafha-2020-multi-dialect-arabic-bert-country/). In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 111–118, Barcelona, Spain (Online). Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2007.05612)
17. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. [Hierarchical attention networks for document classification](https://md.research.elamur.ai/free/en/papers/yang-2016-hierarchical-attention-networks-document-classification/). In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1480–1489, San Diego, California. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n16-1174)
18. Omar F. Zaidan and Chris Callison-Burch. 2014. [Arabic dialect identification](https://md.research.elamur.ai/free/en/papers/zaidan-2014-arabic-dialect-identification/). Computational Linguistics, 40(1):171–202. [↗](https://doi.org/10.1162/coli_a_00169)

*Updated 1 November 2026*