---
title: "Romanized Berber and Romanized Arabic Automatic Language"
description: "Off-the-shelf language identifiers do not recognise Romanized Arabic (RA, often… 95.91% accuracy. 17 references, cited by 8 papers here."
authors: ["Wafia Adouane", "Nasredine Semmar", "Richard Johansson"]
year: 2016
venue: "Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3)"
source: "https://aclanthology.org/W16-4807/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/adouane-2016-romanized-berber-romanized-arabic-automatic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/adouane-2016-romanized-berber-romanized-arabic-automatic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/) / Romanized Berber and Romanized Arabic Automatic Language Identification Using Machine Learning

# Romanized Berber and Romanized Arabic Automatic Language Identification Using Machine Learning

**Authors:** [Wafia Adouane](https://md.research.elamur.ai/free/en/authors/wafia-adouane/), [Nasredine Semmar](https://md.research.elamur.ai/free/en/authors/nasredine-semmar/) and [Richard Johansson](https://md.research.elamur.ai/free/en/authors/richard-johansson/) · **Year:** 2016 · **Venue:** [Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3)](https://md.research.elamur.ai/free/en/venues/coling/) ([COLING 2016](https://md.research.elamur.ai/free/en/venues/coling/2016/)) · **Publisher:** [International Committee on Computational Linguistics (ICCL)](https://md.research.elamur.ai/free/en/publishers/international-committee-on-computational-linguistics-iccl/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/W16-4807/) · [PDF](https://aclanthology.org/W16-4807.pdf) · [OpenAlex](https://openalex.org/W2785919452)

**Citations:** 8 citations (OpenAlex, 2026-10-14) · [8](https://md.research.elamur.ai/free/en/papers/adouane-2016-romanized-berber-romanized-arabic-automatic/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Off-the-shelf language identifiers do not recognise Romanized Arabic (RA, often called Arabizi) or Romanized Berber (RB). Both are under-resourced, have no fixed spelling, mix in other languages, and share many look-alike words because they coexist in North Africa. The authors treat document-level identification of RA and RB as supervised classification. They first built the missing data: about 20,000 RA documents from many Arab dialects and 7,000 RB documents covering six Berber varieties, collected from YouTube comments, forums, news sites, Twitter and Facebook and checked by native speakers. They then trained SVM classifiers on character and word n-grams, optionally adding compiled language-specific lexicons, and compared them with Cavnar's n-gram classifier as a baseline. The task included several other Latin-script languages as distractors, and inputs were short texts capped at 140 characters. The best setup combined character 5-grams with lexicon entries, reaching F-scores of 99.75% for RB and 97.77% for RA. The work matters because it supplies the first usable identifier and corpora for these informal languages, a necessary first step before any further processing.

## Key points
- Corpora: RA has 20,000 documents (144,535 words) and RB has 7,000 documents (31,274 words). Documents run from roughly 2 to 236 words and were gathered from content published around 2013–2016. RB covers Kabyle, Tachelhit, Tarifit, Tachawit, Tachenwit and Tamzabit.
- Setup: six classes (RA, RB, EN, FR, ML, RP), an SVM built with scikit-learn, and Cavnar's classifier as the baseline. Most experiments limit text to 140 characters.
- Character unigram+trigram features gave a macro F-score of 98.75%. Character 5-grams plus lexicons raised accuracy to 99.02% and macro F to 99.00%, although RA's F-score dropped slightly (from 98.24% to 97.77%).
- Word unigrams scored 95.91% accuracy, rising to 97.50% when lexicon entries were added. Longer word n-grams performed worse because of sparsity.
- Remaining errors come mainly from false friends (especially RA vs RP) and from code-mixing (RA vs FR), almost always in documents under 10 words.

## Interesting ideas
- Pairing sub-word character n-grams with small hand-built lists of language-specific words lets the classifier both tolerate wildly varying spellings and anchor its decision on known, unambiguous markers.
- Training against deliberately confusable neighbour classes (French, English and other Romanized languages) makes the evaluation measure real discrimination rather than easy separation.
- Errors were broken down by document length, which showed that almost all failures sit in very short inputs; the authors capped test texts on the assumption that longer ones are easier.
- The paper highlights false friends, where identical surface strings carry different meanings in related languages, as the main failure source.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): A cheap "is this span English?" gate before tagging and lemmatization could use character n-grams plus a lexicon of foreign markers, so that code-switched or borrowed chunks are flagged rather than force-analysed.
- **I2** (learn): Evaluation sets could include near-miss distractor inputs as negative controls, such as non-English or garbled sentences, to check that components refuse or flag them instead of producing confident trees.
- **I3** (learn): Reporting parser and tagger scores stratified by sentence length would expose where short, context-poor inputs break things and steer repair-rule induction there.
- **I4** (world): When grounding words in WordNet/Wikidata entities, surface forms known to be ambiguous across languages or registers could be tagged so that the world builder asks for more context before committing to a sense.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Gothenburg](https://md.research.elamur.ai/free/en/universities/university-of-gothenburg/) — University, Gothenburg, Sweden · 34 works · 1,625 citations
- [Commissariat à l'Énergie Atomique et aux Énergies Alternatives](https://md.research.elamur.ai/free/en/government/commissariat-a-l-energie-atomique-et-aux-energies-alternatives/) — Government body, Paris, France · 11 works · 416 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Wafia Adouane, Nasredine Semmar and Richard Johansson. 2016. Romanized Berber and Romanized Arabic Automatic Language Identification Using Machine Learning. Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{adouane2016romanized,
  title = {{Romanized Berber and Romanized Arabic Automatic Language Identification Using Machine Learning}},
  author = {Wafia Adouane, Nasredine Semmar and Richard Johansson},
  booktitle = {Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3)},
  year = {2016},
  url = {https://aclanthology.org/W16-4807/}
}
```

- [← previous Exploiting Arabic Diacritization for High Quality Automatic Annotation](https://md.research.elamur.ai/free/en/papers/habash-2016-exploiting-arabic-diacritization-high-quality/)
- [next → The DialogBank](https://md.research.elamur.ai/free/en/papers/bunt-2016-dialogbank/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/)

## References

17 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Cyril Goutte, Serge Léger and Marine Carpuat. 2014. The NRC System for Discriminating Similar Languages. In the Proceedings of the VarDial Workshop. [find](https://scholar.google.com/scholar?q=Cyril+Goutte%2C+Serge+L%C3%A9ger+and+Marine+Carpuat%2E+2014%2E+The+NRC+System+for+Discriminating+Similar+Languages%2E+In+the+Proceedings+of+the+VarDial+Workshop%2E)
2. David Sankoff. 1998. The production of code-mixed discourse. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and the 17th International Conference on Computational Linguistics, New Brunswick, NJ: ACL Press, pages 8–21. [find](https://scholar.google.com/scholar?q=David+Sankoff%2E+1998%2E+The+production+of+code%2Dmixed+discourse%2E+In+Proceedings+of+the+36th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+17th+International+Conference+on+Computa)
3. Eirlys Davies, Abdelâli Bentahila and Jonathan Owens. 2013. Codeswitching and related issues involving Arabic. Oxford Handbook of Arabic Linguistics, Sociolinguistics, pages 326–348. [↗](https://doi.org/10.1093/oxfordhb/9780199764136.013.0014)
4. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake VanderPlas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot and Edouard Duchesnay. 2011. Scikit-learn: Machine learning in Python. Machine learning in Python. Journal of Machine Learning Research, 12, pages 2825–2830. [↗](https://doi.org/10.48550/arxiv.1201.0490)
5. Fatma Zohra Chelali, Khadidja Sadeddine and Amar Djeradi. 2015. Speaker identification system using LPC-Application on Berber language. HDSKD journal, 1(2):29–46. [find](https://scholar.google.com/scholar?q=Fatma+Zohra+Chelali%2C+Khadidja+Sadeddine+and+Amar+Djeradi%2E+2015%2E+Speaker+identification+system+using+LPC%2DApplication+on+Berber+language%2E+HDSKD+journal%2C+1%282%29%3A29%E2%80%9346%2E)
6. Houda Saâdane. 2015. Le traitement automatique de l’arabe dialectalisé: aspects méthodologiques et algorithmiques. PhD thesis, Université Grenoble Alpes. [↗](https://doi.org/10.70675/97a91053z9a18z42b5z93f7z9a6599dc3ad4)
7. Kareem Darwish. 2014. [Arabizi Detection and Conversion to Arabic](https://md.research.elamur.ai/free/en/papers/darwish-2014-arabizi-detection-conversion-arabic/). In the Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP).Doha, Qatar, pages 217–224. [↗](https://doi.org/10.48550/arxiv.1306.6755)
8. Lameen Souag. 2004. [Writing Berber Languages:](https://md.research.elamur.ai/free/en/papers/badrashiny-2014-automatic-transliteration-romanized-dialectal-arabic/) [http://goo.gl/ooA4uZ](http://goo.gl/ooA4uZ). Retrieved on April 8th, 2016. a quick summary. L. Souag. Archived from Mohamed Al-Badrashiny, Ramy Eskander, Nizar Habash and Owen Rambow. 2014. Automatic Transliteration of Romanized Dialectal Arabic. In the Proceedings of the Eighteenth Conference on Computational Language Learning, Baltimore, Maryland USA, pages 30–38. [↗](http://goo.gl/ooA4uZ)
9. Marco Lui and Timothy Baldwin. 2012. langid. py: An Off-the-shelf Language Identification Tool. In Proceedings of the ACL. — [langid.py: An Off-the-shelf Language Identification Tool](https://md.research.elamur.ai/free/en/papers/lui-2012-langid-py-off-shelf-language/)
10. Mona Diab, Nizar Habash, Owen Rambow, Mohamed Altantawy and Yassine Benajiba. 2010. COLABA: Arabic dialect annotation and processing. In Proceedings of the LREC Workshop on Semitic Language Processing, pages 66–74. [find](https://scholar.google.com/scholar?q=Mona+Diab%2C+Nizar+Habash%2C+Owen+Rambow%2C+Mohamed+Altantawy+and+Yassine+Benajiba%2E+2010%2E+COLABA%3A+Arabic+dialect+annotation+and+processing%2E+In+Proceedings+of+the+LREC+Workshop+on+Semitic+Language+Processing)
11. Omar F. Zaidan and Chris Callison-Burch. 2011. [The Arabic online commentary dataset: an annotated dataset of informal Arabic with high dialectal content](https://md.research.elamur.ai/free/en/papers/zaidan-2011-arabic-online-commentary-dataset-annotated/). In the Proceedings of the Association for Computational Linguistics (ACL), pages 37–41.
12. Peter Behnstadt and Manfred Woidich. 2013. Diactology. In the Oxford Handbook of Arabic Linguistics. [find](https://scholar.google.com/scholar?q=Peter+Behnstadt+and+Manfred+Woidich%2E+2013%2E+Diactology%2E+In+the+Oxford+Handbook+of+Arabic+Linguistics%2E)
13. Ramy Eskander, Mohamed Al-Badrashiny, Nizar Habash and Owen Rambow. 2014. [Foreign Words and the Automatic Processing of Arabic Social Media Text Written in Roman Script](https://md.research.elamur.ai/free/en/papers/eskander-2014-foreign-words-automatic-processing-arabic/). In the Proceedings of The First Workshop on Computational Approaches to Code Switching. Doha, Qatar, pages 1–12.
14. Ramzi Halimouche, Hocine Teffahi and Leila Falek. 2014. Detecting Sentences Types in Berber Language. International Conference on Multimedia Computing and Systems (ICMCS), pages 197–200. [find](https://scholar.google.com/scholar?q=Ramzi+Halimouche%2C+Hocine+Teffahi+and+Leila+Falek%2E+2014%2E+Detecting+Sentences+Types+in+Berber+Language%2E+International+Conference+on+Multimedia+Computing+and+Systems+%28ICMCS%29%2C+pages+197%E2%80%93200%2E)
15. Shervin Malmasi, Eshrag Refaee and Mark Dras. 2015. Arabic Dialect Identification using a Parallel Multidialectal Corpus. In the Proceedings of the 14th Conference of the Pacific Association for Computational Linguistics (PACLING 2015), pages 209–217, Bali, Indonesia. [find](https://scholar.google.com/scholar?q=Shervin+Malmasi%2C+Eshrag+Refaee+and+Mark+Dras%2E+2015%2E+Arabic+Dialect+Identification+using+a+Parallel+Multidialectal+Corpus%2E+In+the+Proceedings+of+the+14th+Conference+of+the+Pacific+Association+for+Compu)
16. Wafia Adouane. 2016. Automatic Detection of Under-resourced Languages: The case of Arabic Short Texts. Master’s thesis, University of Gothenburg. [find](https://scholar.google.com/scholar?q=Wafia+Adouane%2E+2016%2E+Automatic+Detection+of+Under%2Dresourced+Languages%3A+The+case+of+Arabic+Short+Texts%2E+Master%E2%80%99s+thesis%2C+University+of+Gothenburg%2E)
17. William B. Cavnar and John M. Trenkle. 1994. N-Gram-Based Text Categorization. In the Proceedings of SDAIR-94, 3rd Annual Symposium on Document Analysis and Information Retrieval, University of Nevada, Las Vegas. [find](https://scholar.google.com/scholar?q=William+B%2E+Cavnar+and+John+M%2E+Trenkle%2E+1994%2E+N%2DGram%2DBased+Text+Categorization%2E+In+the+Proceedings+of+SDAIR%2D94%2C+3rd+Annual+Symposium+on+Document+Analysis+and+Information+Retrieval%2C+University+of+Nevada)

*Updated 1 November 2026*