--- title: "Finding Romanized Arabic Dialect in Code-Mixed Tweets (2014)" description: "The work offers a cheap way to bootstrap language ID and lexicons for unstandardized, code-mixed… 8 references, cited by 22 papers here." authors: ["Clare Voss", "Stephen Tratz", "Jamal Laoudi", "Douglas Briesch"] year: 2014 venue: "Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)" source: "https://aclanthology.org/L14-1086/" licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)" licence_url: "https://creativecommons.org/licenses/by-nc/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/voss-2014-finding-romanized-arabic-dialect-code/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/voss-2014-finding-romanized-arabic-dialect-code/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/page/2/) / Finding Romanized Arabic Dialect in Code-Mixed Tweets # Finding Romanized Arabic Dialect in Code-Mixed Tweets **Authors:** [Clare Voss](https://md.research.elamur.ai/free/en/authors/clare-voss/), [Stephen Tratz](https://md.research.elamur.ai/free/en/authors/stephen-tratz/), [Jamal Laoudi](https://md.research.elamur.ai/free/en/authors/jamal-laoudi/) and [Douglas Briesch](https://md.research.elamur.ai/free/en/authors/douglas-briesch/) · **Year:** 2014 · **Venue:** [Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)](https://md.research.elamur.ai/free/en/venues/lrec/) ([LREC 2014](https://md.research.elamur.ai/free/en/venues/lrec/2014/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/) **Links:** [Publisher's page](https://aclanthology.org/L14-1086/) · [PDF](https://aclanthology.org/L14-1086.pdf) · [OpenAlex](https://openalex.org/W2250868350) **Citations:** 38 citations (OpenAlex, 2026-10-14) · [22](https://md.research.elamur.ai/free/en/papers/voss-2014-finding-romanized-arabic-dialect-code/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper addresses spotting Moroccan Arabic (Darija) when it is typed in Latin letters on Twitter and mixed with French and English. Earlier Arabic dialect identification work covered only Arabic script. Because no labeled romanized Darija corpus existed, the authors gathered tweet conversations by searching for a dozen Darija-specific Roman-script words. They clustered about 40K unlabeled tweets with LDA and used the tweets most strongly associated with each language-dominated topic as noisy automatic labels. A Maximum Entropy token classifier was trained on these labels and checked against expert-annotated dev and test sets built in their DATool. On the test set, recall reached 93.2% for Darija, but Darija precision was only about 0.76, because many French tokens were tagged as Darija. Working from the annotations, the authors also built a lexicon that groups spelling variants of the same Darija word and described the recurring variant patterns. A later test set from entirely new users showed similar behavior. The work offers a cheap way to bootstrap language ID and lexicons for unstandardized, code-mixed dialects. ## Key points - **Data:** - 40,628 unlabeled tweets were used for LDA. - Dev and test sets have 800 gold tweets each. - A "New" test set has 106 tweets whose users do not appear in the original data. - The collection was seeded with 12 Darija tokens and drawn from the social networks of 8 individuals. - **Silver training labels:** - A 5-topic LDA model gave two Darija topics plus one topic each for English, French, and Arabic script. - The authors took the top 500 tweets from each Darija topic, the top 1000 each for English and French, and the top 300 for Arabic script, for 3,300 tweets in total. - Punctuation, @users, laughter sounds, hashtags, numbers, emoticons and URLs were labeled by heuristics. - **Classifier:** Maximum Entropy, using character 1–3-gram features, LDA topic ranking, and a context window of three tokens on each side. - **Test results:** - Precision: Darija 0.760, French 0.950, English 0.955. - Recall: Darija 0.932, French 0.834, English 0.901 (the abstract's figures for English and French appear swapped relative to the table). - On the dev set, 363 French tokens were mislabeled as Darija. - On the New set, Darija recall dropped to 0.850. - **Lexicon:** - Six spelling-variation patterns were identified: letter repetition, French- vs English-style spellings, dropped vowels, similar-sounding substitutions (e.g., k/9), apostrophes and accents, and truncated endings. - The New set contributed 78 new surface forms, 25 of which were variants of existing entries. ## Interesting ideas - Unsupervised topic clusters can be turned into training labels by taking only the highest-ranked members of each cluster, accepting some noise to get a labeled set almost for free. - The authors capped how many examples came from each class, and took fewer from a noisy topic, so that no class dominated the training data. - They added a test set drawn from users completely disjoint from the original data, which exposed a drop in robustness that the dev and test sets hid. - Non-linguistic tokens such as handles, laughter, emoticons and URLs get their own heuristic categories instead of being forced into a language label. - Surface forms are grouped into a single lexical entry using a small set of named spelling-variation rules, such as letter repetition and vowel dropping. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): Silver data for rare constructions could be seeded the same way, keeping only the most confident cluster members before witness-parser agreement filters them further. - **I2** (learn): When assembling silver treebanks or teacher-labeled sets, quotas per phenomenon would keep frequent patterns from swamping the rare ones. - **I3** (learn): Evaluation could include held-out sets split by source or author, not just by sentence, as a stronger check on generalization. - **I4** (tokens): The tokenizer and tagger could assign explicit categories to such tokens, so the parser handles social-media noise predictably and explainably. - **I5** (tokens): Explicit, inspectable normalization rules (e.g., collapsing "sooooo" to "so") could map noisy English spellings to lemmas while recording which rule fired. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [DEVCOM Army Research Laboratory](https://md.research.elamur.ai/free/en/labs/devcom-army-research-laboratory/) — Research institute, Adelphi, United States · 14 works · 2,292 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (2): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works ## Cite Cite as: Clare Voss, Stephen Tratz, Jamal Laoudi and Douglas Briesch. 2014. Finding Romanized Arabic Dialect in Code-Mixed Tweets. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{voss2014finding, title = {{Finding Romanized Arabic Dialect in Code-Mixed Tweets}}, author = {Clare Voss, Stephen Tratz, Jamal Laoudi and Douglas Briesch}, booktitle = {Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)}, year = {2014}, doi = {10.63317/2swsts7u3oij}, url = {https://doi.org/10.63317/2swsts7u3oij} } ``` - [← previous Detecting Disagreement in Conversations using Pseudo-Monologic Rhetorical Structure](https://md.research.elamur.ai/free/en/papers/allen-2014-detecting-disagreement-conversations-pseudo-monologic/) - [next → Summarizing Online Forum Discussions – Can Dialog Acts of Individual Messages Help?](https://md.research.elamur.ai/free/en/papers/bhatia-2014-summarizing-online-forum-discussions-can/) - [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/page/2/) ## References 8 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Abdennebi, E. H. and Bowman, S., editors. (2011). Moroccan Arabic Verb Dictionary: English – Moroccan Arabic. Artisanal Treasures, 2nd edition. [find](https://scholar.google.com/scholar?q=Abdennebi%2C+E%2E+H%2E+and+Bowman%2C+S%2E%2C+editors%2E+%282011%29%2E+Moroccan+Arabic+Verb+Dictionary%3A+English+%E2%80%93+Moroccan+Arabic%2E+Artisanal+Treasures%2C+2nd+edition%2E) 2. Berger, A. L., Pietra, V. J. D., and Pietra, S. A. D. (1996). [A maximum entropy approach to natural language processing](https://md.research.elamur.ai/free/en/papers/berger-1996-maximum-entropy-approach-natural-language/). Computational linguistics, 22(1):39–71. [↗](https://doi.org/10.5555/234285.234289) 3. Blei, D. M., Ng, A. Y., and Jordan, M. I. (2003). [Latent dirichlet allocation](https://md.research.elamur.ai/free/en/papers/blei-2003-latent-dirichlet-allocation/). the Journal of machine Learning research, 3:993–1022. 4. Elfardy, H. and Diab, M. (2012). [Simplified guidelines for the creation of large-scale dialectal arabic annotations](https://md.research.elamur.ai/free/en/papers/elfardy-2012-simplified-guidelines-creation-large-scale/). Proceedings of LREC. 5. Habash, N., Diab, M., and Rambow, O. (2012). Conventional orthography for dialectal arab. Proceedings of LREC. [find](https://scholar.google.com/scholar?q=Habash%2C+N%2E%2C+Diab%2C+M%2E%2C+and+Rambow%2C+O%2E+%282012%29%2E+Conventional+orthography+for+dialectal+arab%2E+Proceedings+of+LREC%2E) 6. Harrell, R. S. and Sobelman, H., editors. (1966). A Dictionary of Moroccan Arabic. Georgetown University Press, Washington, D.C. [find](https://scholar.google.com/scholar?q=Harrell%2C+R%2E+S%2E+and+Sobelman%2C+H%2E%2C+editors%2E+%281966%29%2E+A+Dictionary+of+Moroccan+Arabic%2E+Georgetown+University+Press%2C+Washington%2C+D%2EC%2E) 7. Harrell, R. S. (1962). A Short Reference Grammar of Moroccan Arabic. Georgetown University Press, Washington, D.C. [find](https://scholar.google.com/scholar?q=Harrell%2C+R%2E+S%2E+%281962%29%2E+A+Short+Reference+Grammar+of+Moroccan+Arabic%2E+Georgetown+University+Press%2C+Washington%2C+D%2EC%2E) 8. Tratz, S., Briesch, D., Laoudi, J., and Voss, C. (2013). [Tweet conversation annotation tool with a focus on an arabic dialect, moroccan darija](https://md.research.elamur.ai/free/en/papers/tratz-2013-tweet-conversation-annotation-tool-focus/). In Proceedings of the 7th Linguistic Annotation Workshop & Interoperability with Discourse. ACL. *Updated 1 November 2026*