---
title: "A Fine-Grained Annotated Multi-Dialectal Arabic Corpus (2019)"
description: "The authors release ARAP-Tweet 2.0, a much larger and cleaner version of their earlier corpus. 24 references, cited by 6 papers here."
authors: ["Anis Charfi", "Wajdi Zaghouani", "Syed Hassan Mehdi", "Esraa Mohamed"]
year: 2019
venue: "Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019)"
source: "https://aclanthology.org/R19-1023/"
doi: "10.26615/978-954-452-056-4_023"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/charfi-2019-fine-grained-annotated-multi-dialectal/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/charfi-2019-fine-grained-annotated-multi-dialectal/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/7/) / A Fine-Grained Annotated Multi-Dialectal Arabic Corpus

# A Fine-Grained Annotated Multi-Dialectal Arabic Corpus

**Authors:** [Anis Charfi](https://md.research.elamur.ai/free/en/authors/anis-charfi/), [Wajdi Zaghouani](https://md.research.elamur.ai/free/en/authors/wajdi-zaghouani/), Syed Hassan Mehdi and [Esraa Mohamed](https://md.research.elamur.ai/free/en/authors/esraa-mohamed/) · **Year:** 2019 · **Venue:** [Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019)](https://md.research.elamur.ai/free/en/venues/ranlp/) ([RANLP 2019](https://md.research.elamur.ai/free/en/venues/ranlp/2019/)) · **Publisher:** [INCOMA Ltd.](https://md.research.elamur.ai/free/en/publishers/incoma-ltd/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/R19-1023/) · [PDF](https://aclanthology.org/R19-1023.pdf) · [DOI 10.26615/978-954-452-056-4\_023](https://doi.org/10.26615/978-954-452-056-4_023) · [OpenAlex](https://openalex.org/W2986081116)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · [6](https://md.research.elamur.ai/free/en/papers/charfi-2019-fine-grained-annotated-multi-dialectal/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Research on author profiling in Arabic has been limited because few resources exist for dialectal Arabic, and those that do rarely include speaker attributes such as age or gender. The authors release ARAP-Tweet 2.0, a much larger and cleaner version of their earlier corpus. It contains about 5 million tweets (about 50 million words) from roughly 3000 Twitter users across 17 countries, grouped into 15 regions.

Users were found in two ways: through tweet geolocation and through searches for words unique to a region. The pool was then expanded through the followers of those users. Every account was hand-labelled for dialect, gender and age group. Each region is strictly balanced on gender and on three age bands.

Unlike the first version, every user was annotated twice, so agreement was measured over the entire corpus rather than a sample. Average Cohen's Kappa was 0.99 for gender, 0.92 for dialect and 0.88 for age.

The resource supports profiling, dialect identification, authorship attribution, sentiment analysis and bot detection for under-resourced Arabic varieties.

## Key points
- **Scale:** ARAP-Tweet 2.0 has about 5.3M original tweets, with retweets excluded and about 10 words per tweet on average. It covers 15 regions with 198 users each, split 99 male and 99 female. This doubles the per-region user count of version 1.
- **Inclusion criteria:** a user needed between 100 and 3200 Arabic tweets. More than half of their tweets had to be in dialect rather than MSA. Users who mixed several dialects were replaced with users tweeting in a single dialect.
- **Gender cues:** annotators checked the username, the profile photo, and self-describing adjectives carrying the feminine ending Taa' Marbootah.
- **Age cues, in order of preference:**
  - birth years embedded in usernames (e.g., Omda1981m);
  - biographies, other social networks such as LinkedIn and Facebook, and keyword searches for self-reported age;
  - as a last resort, two independent photo-based guesses checked against Microsoft's how-old.net tool.
- **Agreement per region:** age agreement was weakest for Iraq (0.696), Sudan (0.734) and Tunisia (0.742).
- **Challenges:**
  - older users tend to write in MSA;
  - Maghrebi users often tweet in French;
  - many women hide their names and photos;
  - accounts went private or were closed, which the authors handled by periodic re-checking and replacement.

## Interesting ideas
- Speakers of a variety can be found by searching for a small seed list of marker words unique to that variety, then expanding through the social graph; this yields targeted data without an existing classifier.
- Inter-annotator agreement was computed on every item through full double annotation, not on a 10% sample, so the reliability figures describe the whole released data.
- Labels are inferred through an ordered cascade: exact explicit evidence first, then indirect clues, and finally two independent guesses plus an external tool, keeping the item only when they converge.
- Ambiguous items (mixed dialect, mostly MSA, unknown gender) are dropped and replaced with clean items from the same stratum, which keeps the strict balance intact rather than forcing a label.
- Speaker attributes are read from the writer's own self-descriptions, such as grammatical agreement on adjectives about oneself or a stated birth year or age.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): seed searches with dialect-marker words (e.g., Scots or AAVE forms) could gather dialectal English to stress-test the tokenizer, tagger and lemmatizer on non-standard morphology.
- **I2** (learn): agreement statistics among witness parsers could be reported over the full silver set, with per-stratum breakdowns showing where agreement drops, just as Iraq and Sudan stood out for age.
- **I3** (learn): LLM teacher tiers could follow the same pattern, with cheap explicit evidence first and costly guesses only as a fallback, and an item accepted only when independent sources agree.
- **I4** (learn): evaluation and negative-control sets could swap out unclear items for matched replacements, so class balance is never traded for label noise.
- **I5** (world): the world builder could attach attributes to the narrator entity from first-person statements like "I was born in 1981", while keeping track of which sentence licensed each attribute.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 539 works · 179,533 citations
- [Hamad bin Khalifa University](https://md.research.elamur.ai/free/en/universities/hamad-bin-khalifa-university/) — University, Doha, Qatar · 141 works · 14,175 citations
- [Carnegie Mellon University Qatar](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university-qatar/) — University, Doha, Qatar · 47 works · 2,773 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Anis Charfi, Wajdi Zaghouani, Syed Hassan Mehdi and Esraa Mohamed. 2019. A Fine-Grained Annotated Multi-Dialectal Arabic Corpus. Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{charfi2019fine,
  title = {{A Fine-Grained Annotated Multi-Dialectal Arabic Corpus}},
  author = {Anis Charfi, Wajdi Zaghouani, Syed Hassan Mehdi and Esraa Mohamed},
  booktitle = {Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019)},
  year = {2019},
  doi = {10.26615/978-954-452-056-4\_023},
  url = {https://doi.org/10.26615/978-954-452-056-4_023}
}
```

- [← previous Restoring Arabic vowels through omission-tolerant dictionary lookup](https://md.research.elamur.ai/free/en/papers/neme-2019-restoring-arabic-vowels-through-omission/)
- [next → Acquiring Annotated Data with Cross-lingual Explicitation for Implicit Discourse Relation Classification](https://md.research.elamur.ai/free/en/papers/shi-2019-acquiring-annotated-data-cross-lingual/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/7/)

## References

24 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Faisal Al-Shargi, Aidan Kaplan, Ramy Eskander, Nizar Habash, and Owen Rambow. 2016. [Morphologically annotated corpora and morphological analyzers for moroccan and sanaani yemeni arabic](https://md.research.elamur.ai/free/en/papers/shargi-2016-morphologically-annotated-corpora-morphological-analyzers/). In 10th Language Resources and Evaluation Conference (LREC 2016).
2. Nora Al-Twairesh, Rawan Al-Matham, Nora Madi, Nada Almugren, Al-Hanouf Al-Aljmi, Shahad Alshalan, Raghad Alshalan, Nafla Alrumayyan, Shams Al-Manea, Sumayah Bawazeer, et al. 2018. Suar: Towards building a corpus for the saudi dialect. Procedia computer science 142:72–82. [find](https://scholar.google.com/scholar?q=Nora+Al%2DTwairesh%2C+Rawan+Al%2DMatham%2C+Nora+Madi%2C+Nada+Almugren%2C+Al%2DHanouf+Al%2DAljmi%2C+Shahad+Alshalan%2C+Raghad+Alshalan%2C+Nafla+Alrumayyan%2C+Shams+Al%2DManea%2C+Sumayah+Bawazeer%2C+et+al%2E+2018%2E+Suar%3A+Towards+buildi)
3. Israa Alsarsour, Esraa Mohamed, Reem Suwaileh, and Tamer Elsayed. 2018. [Dart: A large dataset of dialectal arabic tweets](https://md.research.elamur.ai/free/en/papers/alsarsour-2018-dart-large-dataset-dialectal-arabic/). In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018).
4. Hamdy Mubarak and Kareem Darwish. 2014. [Using twitter to collect a multi-dialectal corpus of arabic](https://md.research.elamur.ai/free/en/papers/mubarak-2014-twitter-collect-multi-dialectal-corpus/). In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP). pages 1–7.
5. Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014. [A multidialectal parallel corpus of arabic](https://md.research.elamur.ai/free/en/papers/bouamor-2014-multidialectal-parallel-corpus-arabic/). In LREC. pages 1240–1245.
6. Martha Palmer, Olga Babko-Malaya, Ann Bies, Mona T Diab, Mohamed Maamouri, Aous Mansouri, and Wajdi Zaghouani. 2008. A pilot arabic propbank. In LREC. [↗](https://doi.org/10.63317/4s7vg7pvhnh7)
7. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, et al. 2018. The madar arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018). [find](https://scholar.google.com/scholar?q=Houda+Bouamor%2C+Nizar+Habash%2C+Mohammad+Salameh%2C+Wajdi+Zaghouani%2C+Owen+Rambow%2C+Dana+Abdulrahim%2C+Ossama+Obeid%2C+Salam+Khalifa%2C+Fadhl+Eryani%2C+Alexander+Erdmann%2C+et+al%2E+2018%2E+The+madar+arabic+dialect+corpus)
8. Arfath Pasha, Mohamed Al-Badrashiny, Mona T Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. [Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic](https://md.research.elamur.ai/free/en/papers/pasha-2014-madamira-fast-comprehensive-tool-morphological/). In LREC. volume 14, pages 1094–1101.
9. Mona Diab and Nizar Habash. 2007. Arabic dialect processing tutorial. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Tutorial Abstracts. Association for Computational Linguistics, pages 5–6. [find](https://scholar.google.com/scholar?q=Mona+Diab+and+Nizar+Habash%2E+2007%2E+Arabic+dialect+processing+tutorial%2E+In+Proceedings+of+the+Human+Language+Technology+Conference+of+the+NAACL%2C+Companion+Volume%3A+Tutorial+Abstracts%2E+Association+for+Com)
10. Paolo Rosso, Francisco M. Rangel Pardo, Bilal Ghanem, and Anis Charfi. 2018. ARAP: arabic author profiling project for cyber-security. Procesamiento del Lenguaje Natural 61:135–138. [↗](https://doi.org/10.26342/2018-61-16)
11. Clare R Voss, Stephen Tratz, Jamal Laoudi, and Douglas M Briesch. 2014. [Finding romanized arabic dialect in code-mixed tweets](https://md.research.elamur.ai/free/en/papers/voss-2014-finding-romanized-arabic-dialect-code/). In LREC. pages 2249– 2253.
12. Nizar Habash, Ryan Roth, Owen Rambow, Ramy Eskander, and Nadi Tomeh. 2013. [Morphological analysis and disambiguation for dialectal arabic](https://md.research.elamur.ai/free/en/papers/habash-2013-morphological-analysis-disambiguation-dialectal-arabic/). In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 426–432.
13. Wajdi Zaghouani and Anis Charfi. 2018a. Arap-tweet: A large multi-dialect twitter corpus for gender, age and language variety identification. arXiv preprint arXiv:1808.07674 . [↗](https://arxiv.org/abs/1808.07674)
14. Nizar Y Habash. 2010. Introduction to arabic natural language processing. Synthesis Lectures on Human Language Technologies 3(1):1–187. [↗](https://doi.org/10.1007/978-3-031-02139-8)
15. Wajdi Zaghouani and Anis Charfi. 2018b. [Guidelines and annotation framework for arabic author profiling](https://md.research.elamur.ai/free/en/papers/zaghouani-2018-guidelines-annotation-framework-arabic-author/). arXiv preprint arXiv:1808.07678 . [↗](https://arxiv.org/abs/1808.07678)
16. Mustafa Jarrar, Nizar Habash, Diyam Fuad Akra, and Nasser Zalmout. 2014. [Building a corpus for palestinian arabic: a preliminary study](https://md.research.elamur.ai/free/en/papers/jarrar-2014-building-corpus-palestinian-arabic-preliminary/) .
17. Wajdi Zaghouani, Mona Diab, Aous Mansouri, Sameer Pradhan, and Martha Palmer. 2010. The revised arabic propbank. In Proceedings of the fourth linguistic annotation workshop. Association for Computational Linguistics, pages 222–226. [find](https://scholar.google.com/scholar?q=Wajdi+Zaghouani%2C+Mona+Diab%2C+Aous+Mansouri%2C+Sameer+Pradhan%2C+and+Martha+Palmer%2E+2010%2E+The+revised+arabic+propbank%2E+In+Proceedings+of+the+fourth+linguistic+annotation+workshop%2E+Association+for+Computatio)
18. Salam Khalifa, Nizar Habash, Dana Abdulrahim, and Sara Hassan. 2016. [A large scale corpus of gulf arabic](https://md.research.elamur.ai/free/en/papers/khalifa-2016-large-scale-corpus-gulf-arabic/). arXiv preprint arXiv:1609.02960 . [↗](https://arxiv.org/abs/1609.02960)
19. Wajdi Zaghouani, Abdelati Hawwari, and Mona Diab. 2012. A pilot propbank annotation for quranic arabic. In Proceedings of the NAACL-HLT 2012 Workshop on Computational Linguistics for Literature. pages 78–83. [find](https://scholar.google.com/scholar?q=Wajdi+Zaghouani%2C+Abdelati+Hawwari%2C+and+Mona+Diab%2E+2012%2E+A+pilot+propbank+annotation+for+quranic+arabic%2E+In+Proceedings+of+the+NAACL%2DHLT+2012+Workshop+on+Computational+Linguistics+for+Literature%2E+pages)
20. Salam Khalifa, Nizar Habash, Fadhl Eryani, Ossama Obeid, Dana Abdulrahim, and Meera Al Kaabi. 2018. [A morphologically annotated corpus of emirati arabic](https://md.research.elamur.ai/free/en/papers/khalifa-2018-morphologically-annotated-corpus-emirati-arabic/). In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018).
21. Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar F Zaidan, and Chris Callison-Burch. 2012. [Machine translation of arabic dialects](https://md.research.elamur.ai/free/en/papers/zbib-2012-machine-translation-arabic-dialects/). In Proceedings of the NAACL 2012. pages 49–59.
22. Mohamed Maamouri, Ann Bies, Tim Buckwalter, Mona T Diab, Nizar Habash, Owen Rambow, and Dalila Tabessi. 2006. [Developing and using a pilot dialectal arabic treebank](https://md.research.elamur.ai/free/en/papers/maamouri-2006-developing-pilot-dialectal-arabic-treebank/). In LREC. pages 443–448.
23. Mohamed Maamouri, Ann Bies, Seth Kulick, Michael Ciul, Nizar Habash, and Ramy Eskander. 2014. Developing an egyptian arabic treebank: Impact of dialectal morphology on annotation and tool development. In LREC. pages 2348–2354. [find](https://scholar.google.com/scholar?q=Mohamed+Maamouri%2C+Ann+Bies%2C+Seth+Kulick%2C+Michael+Ciul%2C+Nizar+Habash%2C+and+Ramy+Eskander%2E+2014%2E+Developing+an+egyptian+arabic+treebank%3A+Impact+of+dialectal+morphology+on+annotation+and+tool+development%2E)
24. Mohamed Maamouri, Ann Bies, Seth Kulick, Wajdi Zaghouani, David Graff, and Michael Ciul. 2010. From speech to trees: Applying treebank annotation to arabic broadcast news. In LREC. [find](https://scholar.google.com/scholar?q=Mohamed+Maamouri%2C+Ann+Bies%2C+Seth+Kulick%2C+Wajdi+Zaghouani%2C+David+Graff%2C+and+Michael+Ciul%2E+2010%2E+From+speech+to+trees%3A+Applying+treebank+annotation+to+arabic+broadcast+news%2E+In+LREC%2E)

*Updated 1 November 2026*