--- title: "You Tweet What You Speak: A City-Level Dataset of Arabic" description: "Arabic is a family of varieties, and most of its dialects have little or no data that computational models… 1 reference, cited by 53 papers here." authors: ["Muhammad Abdul-Mageed", "Hassan Alhuzali", "Mohamed Elaraby"] year: 2018 venue: "Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)" source: "https://aclanthology.org/L18-1577/" licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)" licence_url: "https://creativecommons.org/licenses/by-nc/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/mageed-2018-you-tweet-what-you-speak/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mageed-2018-you-tweet-what-you-speak/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/5/) / You Tweet What You Speak: A City-Level Dataset of Arabic Dialects # You Tweet What You Speak: A City-Level Dataset of Arabic Dialects **Authors:** [Muhammad Abdul-Mageed](https://md.research.elamur.ai/free/en/authors/muhammad-abdul-mageed/), [Hassan Alhuzali](https://md.research.elamur.ai/free/en/authors/hassan-alhuzali/) and [Mohamed Elaraby](https://md.research.elamur.ai/free/en/authors/mohamed-elaraby/) · **Year:** 2018 · **Venue:** [Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)](https://md.research.elamur.ai/free/en/venues/lrec/) ([LREC 2018](https://md.research.elamur.ai/free/en/venues/lrec/2018/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/) **Links:** [Publisher's page](https://aclanthology.org/L18-1577/) · [PDF](https://aclanthology.org/L18-1577.pdf) · [OpenAlex](https://openalex.org/W2805089673) **Citations:** 59 citations (OpenAlex, 2026-10-14) · [53](https://md.research.elamur.ai/free/en/papers/mageed-2018-you-tweet-what-you-speak/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Arabic is a family of varieties, and most of its dialects have little or no data that computational models can use. Earlier dialect resources came mostly from blogs and news comments. They covered only a few varieties, such as Egyptian and Levantine, and usually labeled them by broad region. The authors start from more than a billion tweets they had collected in-house. These were gathered through geographic bounding boxes over Arab countries, without keyword seeds, over about five years (2013–2018). User-provided locations are resolved with geopy and OpenStreetMap Nominatim, and the resulting labels serve as a proxy for dialect. The final resource holds about 235 million tweets tagged at both city and country level, covering 29 cities in 10 Eastern Arab countries. Several of these countries, such as Jordan, Qatar, Yemen and the UAE, previously lacked large datasets. A pilot portion was checked by hand. To show the data is useful, the authors train word embeddings and compare their coverage with a public MSA vector model. Dialectal words, especially rare ones, turn out to be poorly covered by the MSA model. The dataset makes within-country, city-level dialect study possible. ## Key points - The dataset has 234,801,907 tweets from 159,605 users, spread over 29 cities in ten countries: the Gulf states (KSA, Kuwait, Qatar, the UAE and Oman), along with Yemen, Iraq, Jordan, Palestine and Egypt. - The data is very uneven. Riyadh alone contributes about 90.4M tweets, while Aqaba has roughly 57K. - Compared with Mubarak and Darwish (2014), who collected 123M tweets, the corpus is almost twice as large, spans about twice as many countries, and adds city-level labels. - Labels are assigned automatically with geopy and OpenStreetMap Nominatim, and a pilot section was manually verified for dialect. - The authors measured how well an MSA word-vector model (Zahran et al., 2015) covers words from the KSA and Yemen data: | Word frequency in the dialect data | Coverage in the MSA model | |---|---| | Most frequent | Close to 100% | | Random frequency | About 60–70% | | Least frequent | Around or below 50% | ## Interesting ideas - Free metadata (user location) is used as a cheap stand-in for an expensive label (dialect), and a small slice is then checked by hand to estimate how trustworthy the proxy is. - Lexical coverage of a standard-variety model is reported separately for frequent, random and rare words, which shows where the gap lies instead of hiding it in one average. - Labels are kept at the finest available granularity (city), and coarser levels (country, region) can always be derived from them later. - Crawling by geographic area rather than by seed keywords avoids topic bias and yields a more natural sample of everyday language. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): Silver labels derived from document provenance or metadata could be audited the same way, by manually verifying a small sample before training on the rest. - **I2** (tokens): Lexicon and lemmatizer coverage on a new genre or English dialect could be reported per frequency band, so long-tail failures become visible and targeted. - **I3** (tree): When converting treebanks between UD dialects or genres, tagging each sentence with fine-grained provenance would allow later analyses to group or split data without re-collecting it. - **I4** (learn): Unlabeled text for silver parsing could be sampled by source or region rather than by keyword queries, so evaluation and training data are not skewed toward chosen topics. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Pittsburgh](https://md.research.elamur.ai/free/en/universities/university-of-pittsburgh/) — University, Pittsburgh, United States · 65 works · 23,401 citations - [University of British Columbia](https://md.research.elamur.ai/free/en/universities/university-of-british-columbia/) — University, Vancouver, Canada · 104 works · 37,403 citations - [University of Manchester](https://md.research.elamur.ai/free/en/universities/university-of-manchester/) — University, Manchester, United Kingdom · 93 works · 45,138 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works 🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works ## Cite Cite as: Muhammad Abdul-Mageed, Hassan Alhuzali and Mohamed Elaraby. 2018. You Tweet What You Speak: A City-Level Dataset of Arabic Dialects. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{abdulmageed2018you, title = {{You Tweet What You Speak: A City-Level Dataset of Arabic Dialects}}, author = {Muhammad Abdul-Mageed, Hassan Alhuzali and Mohamed Elaraby}, booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)}, year = {2018}, doi = {10.63317/3wos9hprfv6b}, url = {https://doi.org/10.63317/3wos9hprfv6b} } ``` - [← previous Neural Models of Factuality](https://md.research.elamur.ai/free/en/papers/rudinger-2018-neural-models-factuality/) - [next → AI safety via debate](https://md.research.elamur.ai/free/en/papers/irving-2018-ai-safety-debate/) - [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/5/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Abdul-Mageed, M. M. and Herring, S. C. (2008). Arabic and english news coverage on aljazeera. net. [find](https://scholar.google.com/scholar?q=Abdul%2DMageed%2C+M%2E+M%2E+and+Herring%2C+S%2E+C%2E+%282008%29%2E+Arabic+and+english+news+coverage+on+aljazeera%2E+net%2E) *Updated 29 October 2026*