---
title: "Creating a Large Multi-Layered Representational Repository of"
description: "Code switching, where bilingual writers alternate between language varieties within a text or even… 38 references, cited by 7 papers here."
authors: ["Mona Diab", "Mahmoud Ghoneim", "Abdelati Hawwari", "Fahad AlGhamdi", "Nada AlMarwani", "Mohamed Al-Badrashiny"]
year: 2016
venue: "Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)"
source: "https://aclanthology.org/L16-1669/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/diab-2016-creating-large-multi-layered-representational/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/diab-2016-creating-large-multi-layered-representational/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/) / Creating a Large Multi-Layered Representational Repository of Linguistic Code Switched Arabic Data

# Creating a Large Multi-Layered Representational Repository of Linguistic Code Switched Arabic Data

**Authors:** [Mona Diab](https://md.research.elamur.ai/free/en/authors/mona-diab/), [Mahmoud Ghoneim](https://md.research.elamur.ai/free/en/authors/mahmoud-ghoneim/), [Abdelati Hawwari](https://md.research.elamur.ai/free/en/authors/abdelati-hawwari/), [Fahad AlGhamdi](https://md.research.elamur.ai/free/en/authors/fahad-alghamdi/), [Nada AlMarwani](https://md.research.elamur.ai/free/en/authors/nada-almarwani/) and [Mohamed Al-Badrashiny](https://md.research.elamur.ai/free/en/authors/mohamed-al-badrashiny/) · **Year:** 2016 · **Venue:** [Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)](https://md.research.elamur.ai/free/en/venues/lrec/) ([LREC 2016](https://md.research.elamur.ai/free/en/venues/lrec/2016/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/L16-1669/) · [PDF](https://aclanthology.org/L16-1669.pdf) · [OpenAlex](https://openalex.org/W2976765013)

**Citations:** 11 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/diab-2016-creating-large-multi-layered-representational/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Code switching, where bilingual writers alternate between language varieties within a text or even within a word, causes NLP tools built for a single language to fail. Arabic is a strong case: Modern Standard Arabic (MSA) and Egyptian Dialectal Arabic share much vocabulary, but many shared words carry different meanings. The authors built a large annotated code-switching repository covering three genres: tweets, online commentaries and discussion fora. They designed an XML annotation standard with document, word, switch-point and syntactic layers. They also wrote a generic coarse guideline and an Arabic-specific fine-grained one, each in a detailed in-lab form and a simplified crowdsourcing form. A web tool manages the annotation workflow. Each token receives one of 16 code-switching labels plus POS and orthographic-error tags, and the label depends on sentence context. The resource holds 886,252 tokens with an overall inter-annotator agreement of 93.1%. It addresses the shortage of consistent code-switching training data, which earlier work identified as a main reason tagger accuracy collapses on mixed text.

## Key points
- The repository contains 886,252 tokens of MSA / Egyptian DA code-switched text. Sources are the LDC Egyptian Arabic Treebanks parts 1–8 (ARZ), the Arabic Online Commentary dataset (AOC) and Twitter (TWT). Overall IAA is 93.1%.
- There are 16 CS labels. The linguistic ones are MSA, DA, Ambiguous, MA (MSA affix on a dialectal word), FW, MF (Arabic morphology on a foreign word or the reverse), NE and UNK. Non-linguistic labels cover Latin script, URLs, punctuation, numbers, diacritics, emoticons, sounds and Other.
- Each token also gets one of 14 POS tags (including MWE-Com and NE-Com for parts of multiword expressions and named entities) and an error tag, either Typo or Correct.
- The XML schema has four levels. The document, word and CS-point levels are fully filled; the syntactic level is only partly filled. Switch points are not marked by hand; they are derived wherever the word-level language tag changes.
- Annotation was done in-lab by three native annotators plus a lead. Each had a mandatory three-week training period, and the team met weekly. Crowdsourcing was used in parallel.

## Interesting ideas
- Shared words are labeled by the reading they have in context, and a separate Ambiguous tag covers cases the context cannot resolve, instead of forcing a guess.
- Hybrid words get their own tags (MA, MF), and in mixed words each morpheme is assigned a language separately.
- Spelling problems are kept in their own Typo/Correct layer, separate from the linguistic label.
- The same tagset has two guideline versions: a detailed one for expert gold annotation and a simplified one for crowd workers, each mapped onto the shared label set.
- The guidelines were revised over several rounds by analyzing where annotators disagreed.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): The tagger and lemmatizer could output an explicit "undecidable here" state with the reason, rather than a confident but arbitrary choice, which keeps the pipeline honest and explainable.
- **I2** (tokens): Morphology could record which part of a word comes from which source, for example a borrowed stem with native English inflection, so odd coinages are analyzed instead of becoming unknown tokens.
- **I3** (tokens): A separate typo flag lets the tagger record a misspelling or a bad split or merge without corrupting the POS or lemma decision.
- **I4** (learn): LLM teacher tiers could receive prompts at different levels of detail that all map to one label inventory, so cheap and expensive annotators produce comparable silver data.
- **I5** (learn): Disagreements among witness parsers could be grouped and inspected the same way, so that recurring conflicts lead to clearer conversion conventions or new repair rules.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [George Washington University](https://md.research.elamur.ai/free/en/universities/george-washington-university/) — University, Washington, United States · 51 works · 8,148 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Nada AlMarwani and Mohamed Al-Badrashiny. 2016. Creating a Large Multi-Layered Representational Repository of Linguistic Code Switched Arabic Data. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{diab2016creating,
  title = {{Creating a Large Multi-Layered Representational Repository of Linguistic Code Switched Arabic Data}},
  author = {Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Nada AlMarwani and Mohamed Al-Badrashiny},
  booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)},
  year = {2016},
  doi = {10.63317/2k45bgq8m7ru},
  url = {https://doi.org/10.63317/2k45bgq8m7ru}
}
```

- [← previous An Evaluation of Parser Robustness for Ungrammatical Sentences](https://md.research.elamur.ai/free/en/papers/hashemi-2016-evaluation-parser-robustness-ungrammatical-sentences/)
- [next → Discrimination between Similar Languages, Varieties and Dialects using CNN- and LSTM-based Deep Neural Networks](https://md.research.elamur.ai/free/en/papers/guggilla-2016-discrimination-between-similar-languages-varieties/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/)

## References

38 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Al-Badrashiny, M., Elfardy, H., & Diab, M. (2015). [AIDA2: A Hybrid Approach for Token and Sentence Level Dialect Identification in Arabic](https://md.research.elamur.ai/free/en/papers/badrashiny-2015-aida2-hybrid-approach-token-sentence/). CoNLL 2015, 42.
2. Al-Badrashiny, M., Pasha, A., Diab, M., Habash, N., Rambow, O., Salloum, W., Eskander, R. (2016). SPLIT: Smart Preprocessing (Quasi) Language Independent Tool. In the proceedings of the 10th International Conference on Language Resources and Evaluation (LREC'16). [find](https://scholar.google.com/scholar?q=Al%2DBadrashiny%2C+M%2E%2C+Pasha%2C+A%2E%2C+Diab%2C+M%2E%2C+Habash%2C+N%2E%2C+Rambow%2C+O%2E%2C+Salloum%2C+W%2E%2C+Eskander%2C+R%2E+%282016%29%2E+SPLIT%3A+Smart+Preprocessing+%28Quasi%29+Language+Independent+Tool%2E+In+the+proceedings+of+the+10th+Internati)
3. Albirini, A., Benmamoun, E., & Saadah, E. (2011).Grammatical features of Egyptian and Palestinian Arabic heritage speakers’ oral production. Studies in Second Language Acquisition, 33(02), 273-303. [find](https://scholar.google.com/scholar?q=Albirini%2C+A%2E%2C+Benmamoun%2C+E%2E%2C+%26+Saadah%2C+E%2E+%282011%29%2EGrammatical+features+of+Egyptian+and+Palestinian+Arabic+heritage+speakers%E2%80%99+oral+production%2E+Studies+in+Second+Language+Acquisition%2C+33%2802%29%2C+273%2D303%2E)
4. Bassiouney, R. (Ed.). (2010). Arabic and the media. Brill. [↗](https://doi.org/10.1163/ej.9789004182585.i-303)
5. Benajiba, Y., & Diab, M. (2010). A web application for dialectal arabic text annotation. In Proceedings of the LREC workshop for language resources (LRS) and human language technologies (HLT) for Semitic languages: Status, updates, and prospects. [find](https://scholar.google.com/scholar?q=Benajiba%2C+Y%2E%2C+%26+Diab%2C+M%2E+%282010%29%2E+A+web+application+for+dialectal+arabic+text+annotation%2E+In+Proceedings+of+the+LREC+workshop+for+language+resources+%28LRS%29+and+human+language+technologies+%28HLT%29+for+Semi)
6. Chan, B. H. S. (2003). Aspects of the Syntax, the Pragmatics, and the Production of Code-Switching. Peter Lang. [find](https://scholar.google.com/scholar?q=Chan%2C+B%2E+H%2E+S%2E+%282003%29%2E+Aspects+of+the+Syntax%2C+the+Pragmatics%2C+and+the+Production+of+Code%2DSwitching%2E+Peter+Lang%2E)
7. Chan, B. H. S. (2008). Code-switching, word order and the lexical/functional category distinction. Lingua, 118(6), 777-809. [↗](https://doi.org/10.1016/j.lingua.2007.05.004)
8. Cook, V. J. (1992). Evidence for multicompetence. Language learning, 42(4), 557-591. [↗](https://doi.org/10.1111/j.1467-1770.1992.tb01044.x)
9. Dashti, A. (2015). The role and status of the English language in Kuwait. English Today, 31(03), 28-33. [↗](https://doi.org/10.1017/s026607841500022x)
10. Dey, A., & Fung, P. (2014). A Hindi-English Code-Switching Corpus. In The 9th International Conference on Language Resources and Evaluation (LREC) (pp. 2410-2413). [find](https://scholar.google.com/scholar?q=Dey%2C+A%2E%2C+%26+Fung%2C+P%2E+%282014%29%2E+A+Hindi%2DEnglish+Code%2DSwitching+Corpus%2E+In+The+9th+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%29+%28pp%2E+2410%2D2413%29%2E)
11. Chiang, D., Diab, M. T., Habash, N., Rambow, O., & Shareef, S. (2006). [Parsing Arabic Dialects](https://md.research.elamur.ai/free/en/papers/chiang-2006-parsing-arabic-dialects/). In EACL. [↗](https://doi.org/10.7916/d85t3tzg)
12. Elfardy, H., & Diab, M. T. (2012). [Simplified guidelines for the creation of Large Scale Dialectal Arabic Annotations](https://md.research.elamur.ai/free/en/papers/elfardy-2012-simplified-guidelines-creation-large-scale/). In LREC (pp. 371-378).
13. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5), 378. [↗](https://doi.org/10.1037/h0031619)
14. Gelderen, E., & MacSwan, J. (2008). Interface conditions and code-switching: Pronouns, lexical DPs, and checking theory. Lingua, 118(6), 765-776. [↗](https://doi.org/10.1016/j.lingua.2007.05.003)
15. Grosjean, F. (1989). Neurolinguists, beware! The bilingual is not two monolinguals in one person. Brain and language, 36(1), 3-15. [find](https://scholar.google.com/scholar?q=Grosjean%2C+F%2E+%281989%29%2E+Neurolinguists%2C+beware%21+The+bilingual+is+not+two+monolinguals+in+one+person%2E+Brain+and+language%2C+36%281%29%2C+3%2D15%2E)
16. Lederberg, A. R., & Morales, C. (1985). Code switching by bilinguals: Evidence against a third grammar. Journal of Psycholinguistic Research, 14(2), 113-136. [↗](https://doi.org/10.1007/bf01067625)
17. Li, Y., Yu, Y., & Fung, P. (2012). A Mandarin-English Code-Switching Corpus. In LREC (pp. 2515-2519). [find](https://scholar.google.com/scholar?q=Li%2C+Y%2E%2C+Yu%2C+Y%2E%2C+%26+Fung%2C+P%2E+%282012%29%2E+A+Mandarin%2DEnglish+Code%2DSwitching+Corpus%2E+In+LREC+%28pp%2E+2515%2D2519%29%2E)
18. Lyu, D. C., Lyu, R. Y., Chiang, Y. C., & Hsu, C. N. (2006, May). Speech recognition on code-switching among the Chinese dialects. In Acoustics, Speech and Signal Processing, 2006. ICASSP 2006 Proceedings. 2006 IEEE International Conference on (Vol. 1, pp. I-I). IEEE. [find](https://scholar.google.com/scholar?q=Lyu%2C+D%2E+C%2E%2C+Lyu%2C+R%2E+Y%2E%2C+Chiang%2C+Y%2E+C%2E%2C+%26+Hsu%2C+C%2E+N%2E+%282006%2C+May%29%2E+Speech+recognition+on+code%2Dswitching+among+the+Chinese+dialects%2E+In+Acoustics%2C+Speech+and+Signal+Processing%2C+2006%2E+ICASSP+2006+Proceedi)
19. Maamouri, M., Bies, A., Kulick, S., Tabessi, D., & Krouna, S. (2012). Egyptian Arabic Treebank Pilot. [find](https://scholar.google.com/scholar?q=Maamouri%2C+M%2E%2C+Bies%2C+A%2E%2C+Kulick%2C+S%2E%2C+Tabessi%2C+D%2E%2C+%26+Krouna%2C+S%2E+%282012%29%2E+Egyptian+Arabic+Treebank+Pilot%2E)
20. Maharjan, S., Blair, E., Bethard, S., & Solorio, T. (2015, June). Developing Language-tagged Corpora for Code-switching Tweets. In The 9th Linguistic Annotation Workshop held in conjuncion with NAACL 2015 (p. 72). [find](https://scholar.google.com/scholar?q=Maharjan%2C+S%2E%2C+Blair%2C+E%2E%2C+Bethard%2C+S%2E%2C+%26+Solorio%2C+T%2E+%282015%2C+June%29%2E+Developing+Language%2Dtagged+Corpora+for+Code%2Dswitching+Tweets%2E+In+The+9th+Linguistic+Annotation+Workshop+held+in+conjuncion+with+NAACL+)
21. MacSwan, J. (1999). A minimalist approach to intrasentential codeswitching. New York and London: Garland, 1999. Language in Society, 30(02), 285-289. [find](https://scholar.google.com/scholar?q=MacSwan%2C+J%2E+%281999%29%2E+A+minimalist+approach+to+intrasentential+codeswitching%2E+New+York+and+London%3A+Garland%2C+1999%2E+Language+in+Society%2C+30%2802%29%2C+285%2D289%2E)
22. MacSwan, J. (2000). The architecture of the bilingual language faculty: Evidence from intrasentential code switching. Bilingualism: language and cognition, 3(01), 37-54. [↗](https://doi.org/10.1017/s1366728900000122)
23. MacSwan, J. (2005). Codeswitching and generative grammar: A critique of the MLF model and some remarks on “modified minimalism”. Bilingualism: language and cognition, 8(01), 1-22. [find](https://scholar.google.com/scholar?q=MacSwan%2C+J%2E+%282005%29%2E+Codeswitching+and+generative+grammar%3A+A+critique+of+the+MLF+model+and+some+remarks+on+%E2%80%9Cmodified+minimalism%E2%80%9D%2E+Bilingualism%3A+language+and+cognition%2C+8%2801%29%2C+1%2D22%2E)
24. MacSwan, J. (2005). Remarks on Jake, Myers-Scotton and Gross's response: There is no “matrix language”. Bilingualism: Language and cognition, 8(03), 277-284. [find](https://scholar.google.com/scholar?q=MacSwan%2C+J%2E+%282005%29%2E+Remarks+on+Jake%2C+Myers%2DScotton+and+Gross%27s+response%3A+There+is+no+%E2%80%9Cmatrix+language%E2%80%9D%2E+Bilingualism%3A+Language+and+cognition%2C+8%2803%29%2C+277%2D284%2E)
25. Myers-Scotton, C. (1993). Common and uncommon ground: Social and structural factors in codeswitching. Language in society, 22(04), 475-503. [↗](https://doi.org/10.1017/s0047404500017449)
26. Muysken, P. (2000). Bilingual speech: A typology of code-mixing (Vol. 11). Cambridge University Press. [find](https://scholar.google.com/scholar?q=Muysken%2C+P%2E+%282000%29%2E+Bilingual+speech%3A+A+typology+of+code%2Dmixing+%28Vol%2E+11%29%2E+Cambridge+University+Press%2E)
27. Redouane, R. (2005). Linguistic constraints on codeswitching and codemixing of bilingual Moroccan Arabic-French speakers in Canada. In ISB4: Proceedings of the 4th International Symposium on Bilingualism (pp. 1921-1933). [find](https://scholar.google.com/scholar?q=Redouane%2C+R%2E+%282005%29%2E+Linguistic+constraints+on+codeswitching+and+codemixing+of+bilingual+Moroccan+Arabic%2DFrench+speakers+in+Canada%2E+In+ISB4%3A+Proceedings+of+the+4th+International+Symposium+on+Bilingual)
28. Solorio, T., Blair, E., Maharjan, S., Bethard, S., Diab, M., Ghoneim, M., ... & Fung, P. (2014, October). [Overview for the first shared task on language identification in code-switched data](https://md.research.elamur.ai/free/en/papers/solorio-2014-overview-first-shared-task-language/). In Proceedings of The First Workshop on Computational Approaches to Code Switching (pp. 62-72).
29. Solorio, T., & Liu, Y. (2008, October). [Part-of-speech tagging for English-Spanish code-switched text](https://md.research.elamur.ai/free/en/papers/solorio-2008-part-speech-tagging-english-spanish/). In Proceedings of the Conference on Empirical Methods in Natural Language Processing (pp. 1051-1060). Association for Computational Linguistics.
30. Zaidan, O. F., & Callison-Burch, C. (2011, June). [The Arabic online commentary dataset: an annotated dataset of informal Arabic with high dialectal content](https://md.research.elamur.ai/free/en/papers/zaidan-2011-arabic-online-commentary-dataset-annotated/). In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers-Volume 2 (pp. 37-41). Association for Computational Linguistics. 11.
31. Language Resource References Linguistic Data Consortium. (2012). Egyptian Arabic Treebank DF Part 1 V2.0. Catalog No.: LDC2012E93. [find](https://scholar.google.com/scholar?q=Language+Resource+References+Linguistic+Data+Consortium%2E+%282012%29%2E+Egyptian+Arabic+Treebank+DF+Part+1+V2%2E0%2E+Catalog+No%2E%3A+LDC2012E93%2E)
32. Linguistic Data Consortium. (2012). Egyptian Arabic Treebank DF Part 2 V2.0. Catalog No.: LDC2012E98 [find](https://scholar.google.com/scholar?q=Linguistic+Data+Consortium%2E+%282012%29%2E+Egyptian+Arabic+Treebank+DF+Part+2+V2%2E0%2E+Catalog+No%2E%3A+LDC2012E98)
33. Linguistic Data Consortium. (2012). Egyptian Arabic Treebank DF Part 3 V2.0. Catalog No.: LDC2012E89 [find](https://scholar.google.com/scholar?q=Linguistic+Data+Consortium%2E+%282012%29%2E+Egyptian+Arabic+Treebank+DF+Part+3+V2%2E0%2E+Catalog+No%2E%3A+LDC2012E89)
34. Linguistic Data Consortium. (2012). Egyptian Arabic Treebank DF Part 4 V2.0. Catalog No.: LDC2012E99 [find](https://scholar.google.com/scholar?q=Linguistic+Data+Consortium%2E+%282012%29%2E+Egyptian+Arabic+Treebank+DF+Part+4+V2%2E0%2E+Catalog+No%2E%3A+LDC2012E99)
35. Linguistic Data Consortium. (2012). Egyptian Arabic Treebank DF Part 5 V2.0. Catalog No.: LDC2012E107 [find](https://scholar.google.com/scholar?q=Linguistic+Data+Consortium%2E+%282012%29%2E+Egyptian+Arabic+Treebank+DF+Part+5+V2%2E0%2E+Catalog+No%2E%3A+LDC2012E107)
36. Linguistic Data Consortium. (2012). Egyptian Arabic Treebank DF Part 6 V2.0. Catalog No.: LDC2012E125 [find](https://scholar.google.com/scholar?q=Linguistic+Data+Consortium%2E+%282012%29%2E+Egyptian+Arabic+Treebank+DF+Part+6+V2%2E0%2E+Catalog+No%2E%3A+LDC2012E125)
37. Linguistic Data Consortium. (2012). Egyptian Arabic Treebank DF Part 7 V2.0. Catalog No.: LDC2013E12 [find](https://scholar.google.com/scholar?q=Linguistic+Data+Consortium%2E+%282012%29%2E+Egyptian+Arabic+Treebank+DF+Part+7+V2%2E0%2E+Catalog+No%2E%3A+LDC2013E12)
38. Linguistic Data Consortium. (2013). Egyptian Arabic Treebank DF Part 8 V2.0. Catalog No.: LDC2013E21 [find](https://scholar.google.com/scholar?q=Linguistic+Data+Consortium%2E+%282013%29%2E+Egyptian+Arabic+Treebank+DF+Part+8+V2%2E0%2E+Catalog+No%2E%3A+LDC2013E21)

*Updated 1 November 2026*