---
title: "AIDA2 — paper by Al-Badrashiny et al. (2015)"
description: "The paper addresses dialect identification in Arabic text where Modern Standard Arabic (MSA) and… 23 references, cited by 16 papers here."
authors: ["Mohamed Al-Badrashiny", "Heba Elfardy", "Mona Diab"]
year: 2015
venue: "Proceedings of the Nineteenth Conference on Computational Natural Language Learning"
source: "https://aclanthology.org/K15-1005/"
doi: "10.18653/v1/k15-1005"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/badrashiny-2015-aida2-hybrid-approach-token-sentence/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/badrashiny-2015-aida2-hybrid-approach-token-sentence/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/page/3/) / AIDA2: A Hybrid Approach for Token and Sentence Level Dialect Identification in Arabic

# AIDA2: A Hybrid Approach for Token and Sentence Level Dialect Identification in Arabic

**Authors:** [Mohamed Al-Badrashiny](https://md.research.elamur.ai/free/en/authors/mohamed-al-badrashiny/), [Heba Elfardy](https://md.research.elamur.ai/free/en/authors/heba-elfardy/) and [Mona Diab](https://md.research.elamur.ai/free/en/authors/mona-diab/) · **Year:** 2015 · **Venue:** [Proceedings of the Nineteenth Conference on Computational Natural Language Learning](https://md.research.elamur.ai/free/en/venues/conll/) ([CoNLL 2015](https://md.research.elamur.ai/free/en/venues/conll/2015/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/K15-1005/) · [PDF](https://aclanthology.org/K15-1005.pdf) · [DOI 10.18653/v1/k15-1005](https://doi.org/10.18653/v1/k15-1005) · [OpenAlex](https://openalex.org/W2252222397)

**Citations:** 34 citations (OpenAlex, 2026-10-14) · [16](https://md.research.elamur.ai/free/en/papers/badrashiny-2015-aida2-hybrid-approach-token-sentence/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper addresses dialect identification in Arabic text where Modern Standard Arabic (MSA) and Egyptian Arabic (EDA) are mixed, sometimes within a single sentence. This is a common pattern in social media, and it hurts tools trained only on MSA. AIDA2, an improved successor of the authors' earlier AIDA system, works at two levels.

- **Token level:** each word gets one of six labels from the ShTk code-switching shared task. A CRF sequence labeler makes this decision using signals from MADAMIRA morphology, a dialect-tagged language model, the ModLex modality lexicon, a shallow named-entity module and sentence statistics.
- **Sentence level:** two independent classifiers model different aspects of the input. Each uses a feature set chosen heuristically. A decision tree then combines their labels and confidence scores to call the sentence mostly MSA or mostly EDA.

Both levels beat the prior state of the art. Token F-score rises from 86.8% to 90.6%, and sentence accuracy rises from 86.6% to 90.8%. Reliable identification matters because it can serve as a routing step before downstream Arabic tools such as machine translation.

## Key points
- The token tag set is lang1 (MSA), lang2 (EDA), ne, ambig, mixed and other, following the ShTk shared task (Solorio et al., 2014).
- The language model is a 5-gram model over D3-tokenized text. It is trained on 119K hand-annotated ShTk words plus 8M weblog words (4M MSA, 4M EDA), which are labeled automatically by the dialect of their source.
- MADAMIRA marks a token as MSA or EDA depending on which analyzer produced its analysis: SAMA gives MSA, CALIMA gives EDA, and no analysis gives unk.
- At sentence level, Comp-Cl keeps 11 selected features, such as msaPPL, edaPPL, diaPercent and modEDA. Abs-Cl uses only msaProb and edaProb. The decision-tree ensemble ("DT Ensemble") reaches 89.9% in 10-fold cross-validation.
- In an ablation, removing the modality features lowered Comp-Cl from 87.3% to 87.0%. The authors consider this notable because the feature is so rare.

## Interesting ideas
- The LM training data is labeled for free by source dialect, and each word's label is copied onto all its sub-word pieces. This yields large silver data without manual annotation.
- Which morphological analyzer succeeds on a word (SAMA vs CALIMA) becomes a feature in its own right. The history of how a word was analyzed is treated as evidence about the word.
- An "isMixed" flag fires when the prefix, stem and suffix of one word receive different dialect decisions, catching words that blend two varieties.
- Instead of a majority vote, a decision tree is trained on the labels and confidence scores of two independent classifiers, so it learns when to trust which model.
- The authors test whether each indicator works in both directions. CALIMA counts flag EDA in 96.5% of EDA sentences, but SAMA counts exceed CALIMA counts in only 23.6% of MSA sentences, so samaWords was dropped.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Silver training data could inherit labels such as register or genre from document provenance, with the label propagated down to every token of a sentence.
- **I2** (tokens): The en tagger could record which lexicon or rule produced each lemma and expose that origin as an explainable signal for informal or nonstandard words.
- **I3** (tokens): A similar check could flag hybrid forms, such as foreign stems with English affixes, so they are not forced into a single tag.
- **I4** (learn): Witness parsers could be combined by a small learned arbiter over their per-arc confidences, which stays inspectable when it is a shallow tree.
- **I5** (learn): Evaluation could routinely check whether a diagnostic feature discriminates both classes, acting as a negative control against cues that only seem symmetric.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Columbia University](https://md.research.elamur.ai/free/en/universities/columbia-university/) — University, New York, United States · 254 works · 56,747 citations
- [George Washington University](https://md.research.elamur.ai/free/en/universities/george-washington-university/) — University, Washington, United States · 51 works · 8,148 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works

## Cite

Cite as: Mohamed Al-Badrashiny, Heba Elfardy and Mona Diab. 2015. AIDA2: A Hybrid Approach for Token and Sentence Level Dialect Identification in Arabic. Proceedings of the Nineteenth Conference on Computational Natural Language Learning.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{albadrashiny2015aida2,
  title = {{AIDA2: A Hybrid Approach for Token and Sentence Level Dialect Identification in Arabic}},
  author = {Mohamed Al-Badrashiny, Heba Elfardy and Mona Diab},
  booktitle = {Proceedings of the Nineteenth Conference on Computational Natural Language Learning},
  year = {2015},
  doi = {10.18653/v1/k15-1005},
  url = {https://doi.org/10.18653/v1/k15-1005}
}
```

- [← previous Transition-Based Syntactic Linearization](https://md.research.elamur.ai/free/en/papers/liu-2015-transition-based-syntactic-linearization/)
- [next → Discriminative Syntax-Based Word Ordering for Text Generation](https://md.research.elamur.ai/free/en/papers/zhang-2015-discriminative-syntax-based-word-ordering/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/page/3/)

## References

23 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rania Al-Sabbagh, Jana Diesner, and Roxana Girju. 2013. [Using the semantic-syntactic interface for reliable arabic modality annotation](https://md.research.elamur.ai/free/en/papers/sabbagh-2013-semantic-syntactic-interface-reliable-arabic/). In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 410–418, Nagoya, Japan, October. Asian Federation of Natural Language Processing.
2. Waleed Ammar, Chris Dyer, and Noah A Smith. 2014. Conditional random field autoencoders for unsupervised structured prediction. In Z. Ghahramani, M. Welling, C. Cortes, N.D. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 3311–3319. Curran Associates, Inc. [↗](https://doi.org/10.1184/r1/6473297)
3. Yassine Benajiba, Paolo Rosso, and Jos Miguel Bene-druiz. 2007. Anersys: An arabic named entity recognition system based on maximum entropy. In In Proceedings of CICLing-2007. [find](https://scholar.google.com/scholar?q=Yassine+Benajiba%2C+Paolo+Rosso%2C+and+Jos+Miguel+Bene%2Ddruiz%2E+2007%2E+Anersys%3A+An+arabic+named+entity+recognition+system+based+on+maximum+entropy%2E+In+In+Proceedings+of+CICLing%2D2007%2E)
4. Fadi Biadsy, Julia Hirschberg, and Nizar Habash. 2009. [Spoken arabic dialect identification using phonotactic modeling](https://md.research.elamur.ai/free/en/papers/biadsy-2009-spoken-arabic-dialect-identification-phonotactic/). In Proceedings of the Workshop on Computational Approaches to Semitic Languages at the meeting of the European Association for Computational Linguistics (EACL), Athens, Greece.
5. Gokul Chittaranjan, Yogarshi Vyas, Kalika Bali, and Monojit Choudhury, 2014. Proceedings of the First Workshop on Computational Approaches to Code Switching, chapter Word-level Language Identification using CRF: Code-switching Shared Task Report of MSR India System, pages 73–79. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Gokul+Chittaranjan%2C+Yogarshi+Vyas%2C+Kalika+Bali%2C+and+Monojit+Choudhury%2C+2014%2E+Proceedings+of+the+First+Workshop+on+Computational+Approaches+to+Code+Switching%2C+chapter+Word%2Dlevel+Language+Identification)
6. Ryan Cotterell and Chris Callison-Burch. 2014a. [A multi-dialect, multi-genre corpus of informal written arabic](https://md.research.elamur.ai/free/en/papers/cotterell-2014-multi-dialect-multi-genre-corpus/). In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykjavik, Iceland, may. European Language Resources Association (ELRA).
7. Ryan Cotterell and Chris Callison-Burch. 2014b. [A multi-dialect, multi-genre corpus of informal written arabic](https://md.research.elamur.ai/free/en/papers/cotterell-2014-multi-dialect-multi-genre-corpus/). In Proceedings of the Language Resources and Evaluation Conference (LREC).
8. Heba Elfardy and Mona Diab. 2013. [Sentence-Level Dialect Identification in Arabic](https://md.research.elamur.ai/free/en/papers/elfardy-2013-sentence-level-dialect-identification-arabic/). In Proceedings of ACL2013, Sofia, Bulgaria, August.
9. Heba Elfardy, Mohamed Al-Badrashiny, and Mona Diab, 2014. Proceedings of the First Workshop on Computational Approaches to Code Switching, chapter [AIDA: Identifying Code Switching in Informal Arabic Text](https://md.research.elamur.ai/free/en/papers/elfardy-2014-aida-identifying-code-switching-informal/), pages 94–101. Association for Computational Linguistics.
10. Jr. Forney, G.D. 1973. The viterbi algorithm. Proceedings of the IEEE, 61(3):268–278, March. [↗](https://doi.org/10.1109/proc.1973.9030)
11. Nizar Habash and Fatiha Sadat. 2006. [Arabic preprocessing schemes for statistical machine translation](https://md.research.elamur.ai/free/en/papers/habash-2006-arabic-preprocessing-schemes-statistical-machine/). Proceedings of the 7th Meeting of the North Amer-icanChapter of the Association for Computational Linguistics/Human Language Technologies Conference (HLT-NAACL06).
12. Nizar Habash, Ramy Eskander, and AbdelAti Hawwari. 2012. [A Morphological Analyzer for Egyptian Arabic](https://md.research.elamur.ai/free/en/papers/habash-2012-morphological-analyzer-egyptian-arabic/). In NAACL-HLT 2012 Workshop on Computational Morphology and Phonology (SIGMORPHON2012), pages 1–9.
13. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H Witten. 2009. The weka data mining software: an update. ACM SIGKDD Explorations Newsletter, 11(1). [↗](https://doi.org/10.1145/1656274.1656278)
14. Naman Jain and Ahmad Riyaz Bhat, 2014. Proceedings of the First Workshop on Computational Approaches to Code Switching, chapter Language Identification in Code-Switching Scenario, pages 87–93. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Naman+Jain+and+Ahmad+Riyaz+Bhat%2C+2014%2E+Proceedings+of+the+First+Workshop+on+Computational+Approaches+to+Code+Switching%2C+chapter+Language+Identification+in+Code%2DSwitching+Scenario%2C+pages+87%E2%80%9393%2E+Associa)
15. Levi King, Eric Baucom, Timur Gilmanov, Sandra Kübler, Daniel Whyatt, Wolfgang Maier, and Paul Rodrigues. 2014. The iucl+ system: Word-level language identification via extended markov models. In Proceedings of the First Workshop on Computational Approaches to Code-Switching. EMNLP 2014, Conference on Empirical Methods in Natural Language Processing, Doha, Qatar. [find](https://scholar.google.com/scholar?q=Levi+King%2C+Eric+Baucom%2C+Timur+Gilmanov%2C+Sandra+Ku%CC%88bler%2C+Daniel+Whyatt%2C+Wolfgang+Maier%2C+and+Paul+Rodrigues%2E+2014%2E+The+iucl%2B+system%3A+Word%2Dlevel+language+identification+via+extended+markov+models%2E+In+Pro)
16. Chu-Cheng Lin, Waleed Ammar, Lori Levin, and Chris Dyer, 2014. Proceedings of the First Workshop on Computational Approaches to Code Switching, chapter The CMU Submission for the Shared Task on Language Identification in Code-Switched Data, pages 80–86. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chu%2DCheng+Lin%2C+Waleed+Ammar%2C+Lori+Levin%2C+and+Chris+Dyer%2C+2014%2E+Proceedings+of+the+First+Workshop+on+Computational+Approaches+to+Code+Switching%2C+chapter+The+CMU+Submission+for+the+Shared+Task+on+Langua)
17. Mohamed Maamouri, Dave Graff, Basma Bouziri, Sondos Krouna, Ann Bies, and Seth Kulick. 2010. Ldc standard arabic morphological analyzer (sama) version 3.1. [↗](https://doi.org/10.35111/wgjk-zy44)
18. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan M. Roth. 2014. [MADAMIRA: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Arabic](https://md.research.elamur.ai/free/en/papers/pasha-2014-madamira-fast-comprehensive-tool-morphological/). In Proceedings of LREC, Reykjavik, Iceland.
19. M.A.A. Rashwan, M.A.S.A.A. Al-Badrashiny, M. Attia, S.M. Abdou, and A. Rafea. 2011. A stochastic arabic diacritizer based on a hybrid of factorized and unfactorized textual features. Audio, Speech, and Language Processing, IEEE Transactions on, 19(1):166–175, Jan. [↗](https://doi.org/10.1109/tasl.2010.2045240)
20. Wael Salloum, Heba Elfardy, Linda Alamir-Salloum, Nizar Habash, and Mona Diab. 2014. Sentence level dialect identification for machine translation system selection. In Proceedings of the annual meeting of the Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Wael+Salloum%2C+Heba+Elfardy%2C+Linda+Alamir%2DSalloum%2C+Nizar+Habash%2C+and+Mona+Diab%2E+2014%2E+Sentence+level+dialect+identification+for+machine+translation+system+selection%2E+In+Proceedings+of+the+annual+meetin)
21. Fei Sha and Fernando Pereira. 2003. Shallow parsing with conditional random fields. pages 213–220. [find](https://scholar.google.com/scholar?q=Fei+Sha+and+Fernando+Pereira%2E+2003%2E+Shallow+parsing+with+conditional+random+fields%2E+pages+213%E2%80%93220%2E)
22. Thamar Solorio, Elizabeth Blair, Suraj Maharjan, Steve Bethard, Mona Diab, Mahmoud Gonheim, Abdelati Hawwari, Fahad AlGhamdi, Julia Hirshberg, Alison Chang, , and Pascale Fung. 2014. [Overview for the first shared task on language identification in code-switched data](https://md.research.elamur.ai/free/en/papers/solorio-2014-overview-first-shared-task-language/). In In Proceedings of the First Workshop on Computational Approaches to Code-Switching. EMNLP 2014, Conference on Empirical Methods in Natural Language Processing, October, 2014, Doha, Qatar.
23. Omar F Zaidan and Chris Callison-Burch. 2011. [The arabic online commentary dataset: an annotated dataset of informal arabic with high dialectal content](https://md.research.elamur.ai/free/en/papers/zaidan-2011-arabic-online-commentary-dataset-annotated/). In ACL 2011.

*Updated 1 November 2026*