---
title: "Shami: A Corpus of Levantine Arabic Dialects (2018)"
description: "Arabic NLP tools are built mostly for Modern Standard Arabic (MSA), but everyday online writing uses… 31 references, cited by 39 papers here."
authors: ["Kathrein Abu Kwaik", "Motaz Saad", "Stergios Chatzikyriakidis", "Simon Dobnik"]
year: 2018
venue: "Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)"
source: "https://aclanthology.org/L18-1576/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kwaik-2018-shami-corpus-levantine-arabic-dialects/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kwaik-2018-shami-corpus-levantine-arabic-dialects/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/) / Shami: A Corpus of Levantine Arabic Dialects

# Shami: A Corpus of Levantine Arabic Dialects

**Authors:** [Kathrein Abu Kwaik](https://md.research.elamur.ai/free/en/authors/kathrein-abu-kwaik/), [Motaz Saad](https://md.research.elamur.ai/free/en/authors/motaz-saad/), [Stergios Chatzikyriakidis](https://md.research.elamur.ai/free/en/authors/stergios-chatzikyriakidis/) and [Simon Dobnik](https://md.research.elamur.ai/free/en/authors/simon-dobnik/) · **Year:** 2018 · **Venue:** [Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)](https://md.research.elamur.ai/free/en/venues/lrec/) ([LREC 2018](https://md.research.elamur.ai/free/en/venues/lrec/2018/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/L18-1576/) · [PDF](https://aclanthology.org/L18-1576.pdf) · [OpenAlex](https://openalex.org/W2805974756)

**Citations:** 56 citations (OpenAlex, 2026-10-14) · [39](https://md.research.elamur.ai/free/en/papers/kwaik-2018-shami-corpus-levantine-arabic-dialects/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Arabic NLP tools are built mostly for Modern Standard Arabic (MSA), but everyday online writing uses regional dialects that differ from MSA in vocabulary and usage. Earlier resources either merged all Levantine varieties into one class or covered only some of them. The authors built Shami (SDC), the first corpus that keeps the four Levantine dialects separate: Palestinian, Jordanian, Lebanese and Syrian. They gathered tweets automatically from public figures whose dialect is known, and also collected blogs and forum posts by hand. The text was cleaned with normalization rules designed to protect meaning, and native speakers filtered out MSA and non-Levantine material. To evaluate the corpus, they ran dialect identification with langid.py character n-gram models and scikit-learn Naive Bayes word models, then benchmarked the outcomes against two existing resources, PADIC and the Multi-dialect corpus. Natural SDC text proved harder to classify than those translated parallel corpora. Accuracy ranged from 52% for the four-way task to 90% for Jordanian versus Lebanese. The corpus matters because it shows how close these dialects really are in real-world writing.

## Key points
- SDC covers Palestinian, Jordanian, Lebanese and Syrian Arabic, with one text file per dialect plus a one-sentence-per-file layout for identification. It is released, together with the code and the raw pre-normalization version, at github.com/GU-CLASP/shami-corpus.
- Data sources:
  - Tweets collected through Tweepy from accounts of known Levantine public figures and by geographic location, with streams capped at 9,999 tweets each.
  - Manually harvested dialectal blogs, forum discussions and stories.
- Classifiers: langid.py with character 4- to 7-gram models, and scikit-learn Naive Bayes with word unigram and bigram models.
- Best result: Jordanian vs Lebanese with a word unigram model (0.90 accuracy). Worst result: the four-way task on the full SDC (52% accuracy). Pairs and triples that include Palestinian scored lower, since Palestinian sits between the others.
- In a small survey, native speakers could not confidently assign a dialect to sentences that lacked typical dialect keywords.

## Interesting ideas
- Before collapsing elongated letters, the authors first list words from standard-language text that legitimately contain repeated letters, so genuine doubles survive while expressive stretching is reduced.
- Normalization rules are applied only in specific positions (for example, Aleph only at the start of a word) rather than everywhere, because blanket rules merged words with different meanings.
- Asking native speakers to label keyword-free sentences shows that some distinctions are not recoverable from text at all, which separates task ambiguity from model weakness.
- Corpora built by translation exaggerate differences between varieties, so models look better on them than on natural text.
- Running pairwise and three-way classifications maps which varieties are confusable and which one acts as an in-between case.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): Build a list of real double-letter English words from the lexicon, then normalize "soooo" or "yesss" without damaging words like "committee" or "too".
- **I2** (tokens): Make spelling and case normalization rules position-aware and test each one against minimal pairs, so no rule erases a lexical contrast the parser needs.
- **I3** (learn): Add a human-ceiling check as a negative control, so failures on inherently ambiguous items are not counted as model errors.
- **I4** (learn): Evaluate parsers and readers on naturally occurring text, not only on crafted contrast sets, to avoid inflated scores.
- **I5** (tree): Use pairwise discrimination tests between English treebank varieties to decide which dialect conversions in UD actually need separate rules.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Gothenburg](https://md.research.elamur.ai/free/en/universities/university-of-gothenburg/) — University, Gothenburg, Sweden · 34 works · 1,625 citations
- [Islamic University of Gaza](https://md.research.elamur.ai/free/en/universities/islamic-university-of-gaza/) — University, Gaza, Palestine · 3 works · 109 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Kathrein Abu Kwaik, Motaz Saad, Stergios Chatzikyriakidis and Simon Dobnik. 2018. Shami: A Corpus of Levantine Arabic Dialects. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kwaik2018shami,
  title = {{Shami: A Corpus of Levantine Arabic Dialects}},
  author = {Kathrein Abu Kwaik, Motaz Saad, Stergios Chatzikyriakidis and Simon Dobnik},
  booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)},
  year = {2018},
  doi = {10.63317/4i96mxb3ymx8},
  url = {https://doi.org/10.63317/4i96mxb3ymx8}
}
```

- [← previous Questionable Answers in Question Answering Research: Reproducibility and Variability of Published Results](https://md.research.elamur.ai/free/en/papers/crane-2018-questionable-answers-question-answering-research/)
- [next → CogCompTime: A Tool for Understanding Time in Natural Language](https://md.research.elamur.ai/free/en/papers/ning-2018-cogcomptime-tool-understanding-time-natural/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/)

## References

31 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Adouane, W., Semmar, N., Johansson, R., and Bobicev, V. (2016). [Automatic Detection of Arabicized Berber and Arabic Varieties](https://md.research.elamur.ai/free/en/papers/adouane-2016-automatic-detection-arabicized-berber-arabic/). VarDial 3, page 63.
2. Al-Dubaee, S. A., Ahmad, N., Martinovic, J., and Snasel, V. (2010). Language Identification Using Wavelet Transform and Artificial Neural Network. In Computational Aspects of Social Networks (CASoN), 2010 International Conference on, pages 515–520. IEEE. [find](https://scholar.google.com/scholar?q=Al%2DDubaee%2C+S%2E+A%2E%2C+Ahmad%2C+N%2E%2C+Martinovic%2C+J%2E%2C+and+Snasel%2C+V%2E+%282010%29%2E+Language+Identification+Using+Wavelet+Transform+and+Artificial+Neural+Network%2E+In+Computational+Aspects+of+Social+Networks+%28CASoN%29%2C+)
3. Almeman, K. and Lee, M. (2013). Automatic Building of Arabic Multi Dialect Text Ccorpora by Bootstrapping Dialect Words. In Communications, Signal processing, and their Applications (ICCSPA), 2013 1st international conference on, pages 1–6. IEEE. [find](https://scholar.google.com/scholar?q=Almeman%2C+K%2E+and+Lee%2C+M%2E+%282013%29%2E+Automatic+Building+of+Arabic+Multi+Dialect+Text+Ccorpora+by+Bootstrapping+Dialect+Words%2E+In+Communications%2C+Signal+processing%2C+and+their+Applications+%28ICCSPA%29%2C+2013+1st)
4. Botha, G. R. et al. (2008). Text-based Language Identification for the South African Languages. Ph.D. thesis, University of Pretoria. [find](https://scholar.google.com/scholar?q=Botha%2C+G%2E+R%2E+et+al%2E+%282008%29%2E+Text%2Dbased+Language+Identification+for+the+South+African+Languages%2E+Ph%2ED%2E+thesis%2C+University+of+Pretoria%2E)
5. Bouamor, H., Habash, N., and Oflazer, K. (2014). [A Multidialectal Parallel Corpus of Arabic](https://md.research.elamur.ai/free/en/papers/bouamor-2014-multidialectal-parallel-corpus-arabic/). In LREC, pages 1240–1245.
6. Cavnar, W. B., Trenkle, J. M., et al. (1994). N-gram-based Text Categorization. Ann Arbor MI, 48113(2):161–175. [find](https://scholar.google.com/scholar?q=Cavnar%2C+W%2E+B%2E%2C+Trenkle%2C+J%2E+M%2E%2C+et+al%2E+%281994%29%2E+N%2Dgram%2Dbased+Text+Categorization%2E+Ann+Arbor+MI%2C+48113%282%29%3A161%E2%80%93175%2E)
7. Cotterell, R. and Callison-Burch, C. (2014). [A Multi-dialect, Multi-Genre Corpus of Informal Written Arabic](https://md.research.elamur.ai/free/en/papers/cotterell-2014-multi-dialect-multi-genre-corpus/). In LREC, pages 241–245.
8. Diab, M., Habash, N., Rambow, O., Altantawy, M., and Benajiba, Y. (2010). Colaba: Arabic Dialect Annotation and Processing. In Lrec workshop on semitic language processing, pages 66–74. [find](https://scholar.google.com/scholar?q=Diab%2C+M%2E%2C+Habash%2C+N%2E%2C+Rambow%2C+O%2E%2C+Altantawy%2C+M%2E%2C+and+Benajiba%2C+Y%2E+%282010%29%2E+Colaba%3A+Arabic+Dialect+Annotation+and+Processing%2E+In+Lrec+workshop+on+semitic+language+processing%2C+pages+66%E2%80%9374%2E)
9. Diab, M. (2009). Second Generation AMIRA Tools for Arabic Processing: Fast and Robust Tokenization, POS Tagging, and Base Phrase Chunking. In 2nd International Conference on Arabic Language Resources and Tools, volume 110. [find](https://scholar.google.com/scholar?q=Diab%2C+M%2E+%282009%29%2E+Second+Generation+AMIRA+Tools+for+Arabic+Processing%3A+Fast+and+Robust+Tokenization%2C+POS+Tagging%2C+and+Base+Phrase+Chunking%2E+In+2nd+International+Conference+on+Arabic+Language+Resources+)
10. Dunning, T. (1994). Statistical identification of Language. Computing Research Laboratory, New Mexico State University. [find](https://scholar.google.com/scholar?q=Dunning%2C+T%2E+%281994%29%2E+Statistical+identification+of+Language%2E+Computing+Research+Laboratory%2C+New+Mexico+State+University%2E)
11. Elfardy, H. and Diab, M. T. (2013). [Sentence Level Dialect Identification in Arabic](https://md.research.elamur.ai/free/en/papers/elfardy-2013-sentence-level-dialect-identification-arabic/). In ACL (2), pages 456–461.
12. Gonzalez-Dominguez, J., Lopez-Moreno, I., Sak, H., Gonzalez-Rodriguez, J., and Moreno, P. J. (2014). Automatic Language Identification Using Long Short-term Memory Recurrent Neural Networks. In Fifteenth Annual Conference of the International Speech Communication Association. [find](https://scholar.google.com/scholar?q=Gonzalez%2DDominguez%2C+J%2E%2C+Lopez%2DMoreno%2C+I%2E%2C+Sak%2C+H%2E%2C+Gonzalez%2DRodriguez%2C+J%2E%2C+and+Moreno%2C+P%2E+J%2E+%282014%29%2E+Automatic+Language+Identification+Using+Long+Short%2Dterm+Memory+Recurrent+Neural+Networks%2E+In+Fiftee)
13. Habash, N., Rambow, O., and Roth, R. (2009). MADA+ TOKAN: A Toolkit for Arabic Tokenization, Diacritization, Morphological Disambiguation, POS Tagging, Stemming and Lemmatization. In Proceedings of the 2nd international conference on Arabic language resources and tools (MEDAR), Cairo, Egypt, volume 41, page 62. [find](https://scholar.google.com/scholar?q=Habash%2C+N%2E%2C+Rambow%2C+O%2E%2C+and+Roth%2C+R%2E+%282009%29%2E+MADA%2B+TOKAN%3A+A+Toolkit+for+Arabic+Tokenization%2C+Diacritization%2C+Morphological+Disambiguation%2C+POS+Tagging%2C+Stemming+and+Lemmatization%2E+In+Proceedings+of+th)
14. Habash, N. Y. (2010). Introduction to Arabic Natural Language Processing. Synthesis Lectures on Human Language Technologies, 3(1):1–187. [↗](https://doi.org/10.1007/978-3-031-02139-8)
15. Jarrar, M., Habash, N., Akra, D., and Zalmout, N. (2014). [Building a Corpus for Palestinian Arabic: A Preliminary Study](https://md.research.elamur.ai/free/en/papers/jarrar-2014-building-corpus-palestinian-arabic-preliminary/). In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 18– 27.
16. Lui, M. and Baldwin, T. (2011). Cross-domain Feature Selection for Language Identification. In In Proceedings of 5th International Joint Conference on Natural Language Processing. Citeseer. [find](https://scholar.google.com/scholar?q=Lui%2C+M%2E+and+Baldwin%2C+T%2E+%282011%29%2E+Cross%2Ddomain+Feature+Selection+for+Language+Identification%2E+In+In+Proceedings+of+5th+International+Joint+Conference+on+Natural+Language+Processing%2E+Citeseer%2E)
17. Lui, M. and Baldwin, T. (2012). [Langid.py: An Off-the-shelf Language Identification Tool](https://md.research.elamur.ai/free/en/papers/lui-2012-langid-py-off-shelf-language/). In Proceedings of the ACL 2012 System Demonstrations, pages 25–30. Association for Computational Linguistics.
18. Maamouri, M. and Cieri, C. (2002). Resources for Arabic Natural Language Processing. In International Symposium on Processing Arabic, volume 1. [find](https://scholar.google.com/scholar?q=Maamouri%2C+M%2E+and+Cieri%2C+C%2E+%282002%29%2E+Resources+for+Arabic+Natural+Language+Processing%2E+In+International+Symposium+on+Processing+Arabic%2C+volume+1%2E)
19. Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S., and McClosky, D. (2014). [The Stanford Corenlp Natural Language Processing Toolkit](https://md.research.elamur.ai/free/en/papers/manning-2014-stanford-corenlp-natural-language-processing/). In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60.
20. McCandless, M. (2011). Accuracy and Performance of Google’s Compact Language Detector. [find](https://scholar.google.com/scholar?q=McCandless%2C+M%2E+%282011%29%2E+Accuracy+and+Performance+of+Google%E2%80%99s+Compact+Language+Detector%2E)
21. Meftouh, K., Harrat, S., Jamoussi, S., Abbas, M., and Smaili, K. (2015). [Machine Translation Experiments on PADIC: A Parallel Arabic Dialect Corpus](https://md.research.elamur.ai/free/en/papers/meftouh-2015-machine-translation-experiments-padic-parallel/). In The 29th Pacific Asia conference on language, information and computation.
22. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. (2011). Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12(Oct):2825–2830. [↗](https://doi.org/10.48550/arxiv.1201.0490)
23. Richardson, F. S. and Campbell, W. M. (2008). Language Recognition With Discriminative Keyword Selection. In Acoustics, Speech and Signal Processing, 2008. ICASSP 2008. IEEE International Conference on, pages 4145– 4148. IEEE. [find](https://scholar.google.com/scholar?q=Richardson%2C+F%2E+S%2E+and+Campbell%2C+W%2E+M%2E+%282008%29%2E+Language+Recognition+With+Discriminative+Keyword+Selection%2E+In+Acoustics%2C+Speech+and+Signal+Processing%2C+2008%2E+ICASSP+2008%2E+IEEE+International+Conference+o)
24. Sadat, F., Kazemi, F., and Farzindar, A. (2014). Automatic Identification of Arabic Dialects in Social Media. In Proceedings of the First International Workshop on Social Media Retrieval and Analysis, pages 35–40. ACM. [find](https://scholar.google.com/scholar?q=Sadat%2C+F%2E%2C+Kazemi%2C+F%2E%2C+and+Farzindar%2C+A%2E+%282014%29%2E+Automatic+Identification+of+Arabic+Dialects+in+Social+Media%2E+In+Proceedings+of+the+First+International+Workshop+on+Social+Media+Retrieval+and+Analysis%2C)
25. Salloum, W., Elfardy, H., Alamir-Salloum, L., Habash, N., and Diab, M. (2014). Sentence Level Dialect Identification for Machine Translation System Selection. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 772–778. [find](https://scholar.google.com/scholar?q=Salloum%2C+W%2E%2C+Elfardy%2C+H%2E%2C+Alamir%2DSalloum%2C+L%2E%2C+Habash%2C+N%2E%2C+and+Diab%2C+M%2E+%282014%29%2E+Sentence+Level+Dialect+Identification+for+Machine+Translation+System+Selection%2E+In+Proceedings+of+the+52nd+Annual+Meeting)
26. Selamat, A. (2011). Improved N-grams Approach for Web Page Language Identification. In Transactions on Computational Collective Intelligence V, pages 1–26. Springer. [find](https://scholar.google.com/scholar?q=Selamat%2C+A%2E+%282011%29%2E+Improved+N%2Dgrams+Approach+for+Web+Page+Language+Identification%2E+In+Transactions+on+Computational+Collective+Intelligence+V%2C+pages+1%E2%80%9326%2E+Springer%2E)
27. Shuyo, N. (2010). Language Detection Library for Java. Retrieved Jul, 7:2016. [find](https://scholar.google.com/scholar?q=Shuyo%2C+N%2E+%282010%29%2E+Language+Detection+Library+for+Java%2E+Retrieved+Jul%2C+7%3A2016%2E)
28. Yang, X. and Liang, W. (2010). An N-gram-and-Wikipedia Joint Approach to Natural Language Identification. In Universal Communication Symposium (IUCS), 2010 4th International, pages 332–339. IEEE. [find](https://scholar.google.com/scholar?q=Yang%2C+X%2E+and+Liang%2C+W%2E+%282010%29%2E+An+N%2Dgram%2Dand%2DWikipedia+Joint+Approach+to+Natural+Language+Identification%2E+In+Universal+Communication+Symposium+%28IUCS%29%2C+2010+4th+International%2C+pages+332%E2%80%93339%2E+IEEE%2E)
29. Zaidan, O. F. and Callison-Burch, C. (2011). [The Arabic Online Commentary Dataset: An Annotated Dataset of Informal Arabic with High Dialectal Content](https://md.research.elamur.ai/free/en/papers/zaidan-2011-arabic-online-commentary-dataset-annotated/). In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2, pages 37–41. Association for Computational Linguistics.
30. Zaidan, O. F. and Callison-Burch, C. (2014). [Arabic Dialect Identification](https://md.research.elamur.ai/free/en/papers/zaidan-2014-arabic-dialect-identification/). Computational Linguistics, 40(1):171–202. [↗](https://doi.org/10.1162/coli_a_00169)
31. Zbib, R., Malchiodi, E., Devlin, J., Stallard, D., Matsoukas, S., Schwartz, R., Makhoul, J., Zaidan, O. F., and Callison-Burch, C. (2012). [Machine Translation of Arabic Dialects](https://md.research.elamur.ai/free/en/papers/zbib-2012-machine-translation-arabic-dialects/). In Proceedings of the 2012 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 49–59. Association for Computational Linguistics.

*Updated 1 November 2026*