--- title: "Arabizi Identification in Twitter Data — paper by Tobaili (2016)" description: "Many sentiment studies on Arabic simply drop such text. 2 references, cited by 8 papers here." authors: ["Taha Tobaili"] year: 2016 venue: "Proceedings of the ACL 2016 Student Research Workshop" source: "https://aclanthology.org/P16-3008/" doi: "10.18653/v1/p16-3008" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/tobaili-2016-arabizi-identification-twitter-data/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tobaili-2016-arabizi-identification-twitter-data/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/) / Arabizi Identification in Twitter Data # Arabizi Identification in Twitter Data **Authors:** [Taha Tobaili](https://md.research.elamur.ai/free/en/authors/taha-tobaili/) · **Year:** 2016 · **Venue:** [Proceedings of the ACL 2016 Student Research Workshop](https://md.research.elamur.ai/free/en/venues/acl-student-research-workshop/) ([ACL Student Research Workshop 2016](https://md.research.elamur.ai/free/en/venues/acl-student-research-workshop/2016/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/P16-3008/) · [PDF](https://aclanthology.org/P16-3008.pdf) · [DOI 10.18653/v1/p16-3008](https://doi.org/10.18653/v1/p16-3008) · [OpenAlex](https://openalex.org/W2507475033) **Citations:** 7 citations (OpenAlex, 2026-10-14) · [8](https://md.research.elamur.ai/free/en/papers/tobaili-2016-arabizi-identification-twitter-data/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper tackles a gap in Arabic social-media processing: telling apart tweets in Arabizi (dialectal Arabic typed in Latin letters, with no fixed spelling) from English and other Latin-script tweets. Many sentiment studies on Arabic simply drop such text. The author streamed geolocated tweets from Lebanon and Egypt and set aside the Latin-script portion. After cleaning out noise and spam, one native speaker hand-labelled 5,000 tweets per country. These labels served two purposes: estimating how common Arabizi is, and training an SVM. The classifier does not use the text directly. Its inputs are Langdetect's outputs, including the spurious languages it reports, their number and their confidence scores, plus Twitter's own language tag and word-occurrence counts. It reached an average identification accuracy of 94.5%. Arabizi made up roughly 5% of all tweets in each country, which is less than earlier studies of private messaging found but still a large volume. The work is a first step toward sentiment analysis that includes Arabizi instead of discarding it. ## Key points - Corpus sizes: 60,364 tweets from Lebanon and 249,149 from Egypt. After preprocessing, the non-Arabic parts shrank from 32,024 to 21,878 tweets (Lebanon) and from 74,328 to 36,970 tweets (Egypt). - The annotated samples contain 465 Arabizi tweets for Lebanon and 955 for Egypt. They are published on project-rbz.com. - Arabizi share: - 9.3% of non-Arabic tweets in Lebanon and 19% in Egypt; - overall, 4.9% of all tweets in Lebanon and 5.7% in Egypt. - Method: an SVM over Langdetect detections, the Twitter API language tag and word counts, evaluated with 10-fold cross-validation. It was tested on both imbalanced data and data balanced by undersampling English. - Main error sources: - Lebanese tweets mixing English and Arabizi, which Langdetect tends to label as English; - misdetected Far-Eastern-language tweets. ## Interesting ideas - An off-the-shelf language detector has never seen Arabizi, so on such input it produces scattered, low-confidence guesses among unrelated languages; the paper turns this confusion pattern into a signal for an unseen language. - Tweets are labelled by their dominant language only when the Arabizi words include real content words (nouns, verbs) rather than just connectors or stop words, which gives a principled rule for code-switched text. - Elongated words and laughter strings are stripped because repeated letters mislead the detector, a cheap normalisation applied before the main tool. - The classifier is evaluated on both the natural skewed distribution and an undersampled balanced version, which separates real discrimination ability from majority-class effects. - The error analysis breaks failures down by input type (mixed-language tweets versus Far-Eastern scripts) and links each failure group to a concrete feature fix. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (tokens): A tagger's or lemmatizer's own uncertainty profile (many competing analyses, low scores) could flag spans of foreign or garbled text before they reach parsing. - **I2** (learn): When building silver data, sentences could be included or excluded by whether their content words are English, not merely their function words. - **I3** (tokens): Collapsing letter repetitions and filtering laughter tokens before morphology would keep informal input from producing bogus lemmas. - **I4** (learn): Reporting scores on both natural and rebalanced test sets would expose whether a component merely follows frequent labels. - **I5** (learn): Grouping parser or reader failures by input category would point repair-rule induction at specific, nameable weaknesses. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works ## Cite Cite as: Taha Tobaili. 2016. Arabizi Identification in Twitter Data. Proceedings of the ACL 2016 Student Research Workshop. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{tobaili2016arabizi, title = {{Arabizi Identification in Twitter Data}}, author = {Taha Tobaili}, booktitle = {Proceedings of the ACL 2016 Student Research Workshop}, year = {2016}, doi = {10.18653/v1/p16-3008}, url = {https://doi.org/10.18653/v1/p16-3008} } ``` - [← previous The DialogBank](https://md.research.elamur.ai/free/en/papers/bunt-2016-dialogbank/) - [next → Classifying ASR Transcriptions According to Arabic Dialect](https://md.research.elamur.ai/free/en/papers/hanani-2016-classifying-asr-transcriptions-according-arabic/) - [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/) ## References 2 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Year Location Participants Data Size of Data Arabizi English Arabic (Keong et al., 2015) [find](https://scholar.google.com/scholar?q=Year+Location+Participants+Data+Size+of+Data+Arabizi+English+Arabic+%28Keong+et+al%2E%2C+2015%29) 2. (Bies et al., 2014) Malysia [find](https://scholar.google.com/scholar?q=%28Bies+et+al%2E%2C+2014%29+Malysia) *Updated 30 October 2026*