---
title: "An Arabizi-English social media statistical machine translation"
description: "The paper addresses translating Arabizi (Arabic written informally in Latin letters and digits) into English. 19 references, cited by 7 papers here."
authors: ["Jonathan May", "Yassine Benjira", "Abdessamad Echihabi"]
year: 2014
venue: "Proceedings of the 11th Conference of the Association for Machine Translation in the Americas: MT Researchers Track"
source: "https://aclanthology.org/2014.amta-researchers.25/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/may-2014-arabizi-english-social-media-statistical/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/may-2014-arabizi-english-social-media-statistical/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/page/3/) / An Arabizi-English social media statistical machine translation system

# An Arabizi-English social media statistical machine translation system

**Authors:** [Jonathan May](https://md.research.elamur.ai/free/en/authors/jonathan-may/), Yassine Benjira and [Abdessamad Echihabi](https://md.research.elamur.ai/free/en/authors/abdessamad-echihabi/) · **Year:** 2014 · **Venue:** [Proceedings of the 11th Conference of the Association for Machine Translation in the Americas: MT Researchers Track](https://md.research.elamur.ai/free/en/venues/amta/) ([AMTA 2014](https://md.research.elamur.ai/free/en/venues/amta/2014/)) · **Publisher:** [Association for Machine Translation in the Americas](https://md.research.elamur.ai/free/en/publishers/association-for-machine-translation-in-the-americas/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/2014.amta-researchers.25/) · [PDF](https://aclanthology.org/2014.amta-researchers.25.pdf) · [OpenAlex](https://openalex.org/W3187000794)

**Citations:** 13 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/may-2014-arabizi-english-social-media-statistical/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper addresses translating Arabizi (Arabic written informally in Latin letters and digits) into English. No Arabizi–English parallel corpus exists, and mainstream tools barely handle this script. The authors keep an existing Arabic→English phrase-based system trained on social media data and put a deromanization module in front of it. That module is a character-level weighted finite-state transducer (wFST) that maps Arabizi to Arabic script and is rescored by a character 5-gram Arabic language model. They compare three ways of building the transducer table: written by an expert, induced from aligned word pairs with SMT tools, and a semi-automatic hybrid. Because the whole pipeline passes weighted lattices between modules, uncertain tokens such as "men" (English word or Arabizi for "from") stay ambiguous until the translation step chooses. Evaluated on Egyptian social-media test sets, the system's translation quality comes reasonably close to the ceiling set by gold deromanization. It also matches a specialised deromanizer, 3arrib, that relies on a morphological analyzer. The work matters because it makes large volumes of romanized dialect text machine-translatable without any new parallel data.

## Key points
- The final deromanizer wFST has 962 states and 1550 arcs. It is reweighted by a character-based 5-gram LM trained on 5.4m Arabic words. Characters were chosen over words so that out-of-vocabulary names would not be "corrected" into known words.
- The expert table had 83 character-sequence pairs. The automatic table came from 863 Arabizi/Arabic word pairs aligned with GIZA (reordering disabled) and phrase-style extraction of spans up to 4 characters, giving 3138 unique pairs. The hybrid kept only automatic pairs whose Arabizi side is shorter than 3 characters and combined them with manual entries.
- There are two test sets: Test 1 has 7,794 segments, of which 78.2% of source tokens are deromanizable; Test 2 has 27,901 segments, with 97.7% deromanizable.
- Translation BLEU in Table 4, as far as the extracted text can be read:
  - Without deromanization: 7.7 on Test 1 and 3.7 on Test 2.
  - With the semi-automatic + LM deromanizer: 15.3 and 13.4.
  - With 3arrib: 15.7 and 14.8.
  - With reference deromanization: 18.4 and 17.9.
- The gold Arabic for the test sets was produced by post-editing 3arrib output. This inflates 3arrib's intrinsic transliteration scores.

## Interesting ideas
- Instead of a hard classifier deciding whether a token is Arabizi or English, both readings go downstream as a weighted lattice, and the context-rich translator picks the path.
- A character-level transliteration table can be learned by treating word pairs as "sentences" and characters as "words", then reusing standard alignment and phrase extraction.
- The hybrid design keeps only short automatic patterns and adds human-written entries such as vowel deletions, which the alignment method cannot learn. This trades coverage against noise from spurious long pairs.
- A character LM was preferred over a word LM specifically to avoid overcorrecting unknown names into familiar words.
- Scoring the pipeline against an oracle run (perfect deromanization) and a no-preprocessing run brackets how much any module improvement can still buy.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): The tokenizer and tagger could emit lattices for ambiguous tokens such as code-switches, slang spellings or typos, letting the parser or frames stage settle them with an explainable record of the choice.
- **I2** (learn): The same trick could induce spelling-normalization or morphology rewrite rules from small sets of noisy/clean word pairs, yielding inspectable rules rather than opaque models.
- **I3** (learn): Self-induced repair rules could be pruned by length or support and then merged with hand rules that cover phenomena the induction method structurally misses.
- **I4** (tokens): Normalization of informal English should use sub-word scoring so that rare proper names reach the entity layer intact.
- **I5** (learn): Each stage could be evaluated with gold-input and null-module controls to measure its real headroom and contribution to final answers.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (3): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 1,087 works · [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Jonathan May, Yassine Benjira and Abdessamad Echihabi. 2014. An Arabizi-English social media statistical machine translation system. Proceedings of the 11th Conference of the Association for Machine Translation in the Americas: MT Researchers Track.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{may2014arabizi,
  title = {{An Arabizi-English social media statistical machine translation system}},
  author = {Jonathan May, Yassine Benjira and Abdessamad Echihabi},
  booktitle = {Proceedings of the 11th Conference of the Association for Machine Translation in the Americas: MT Researchers Track},
  year = {2014},
  url = {https://aclanthology.org/2014.amta-researchers.25/}
}
```

- [← previous A New Parsing Algorithm for Combinatory Categorial Grammar](https://md.research.elamur.ai/free/en/papers/kuhlmann-2014-new-parsing-algorithm-combinatory-categorial/)
- [next → Arabic Language Text Classification Using Dependency Syntax-Based Feature Selection](https://md.research.elamur.ai/free/en/papers/haralambous-2014-arabic-language-text-classification-using/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/page/3/)

## References

19 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Al-Badrashiny, M., Eskander, R., Habash, N., and Rambow, O. (2014). [Automatic transliteration of romanized dialectal Arabic](https://md.research.elamur.ai/free/en/papers/badrashiny-2014-automatic-transliteration-romanized-dialectal-arabic/). In Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pages 30–38, Ann Arbor, Michigan. Association for Computational Linguistics.
2. Al-Onaizan, Y. and Knight, K. (2002). [Machine transliteration of names in Arabic texts](https://md.research.elamur.ai/free/en/papers/onaizan-2002-machine-transliteration-names-arabic-texts/). In Proceedings of the ACL-02 Workshop on Computational Approaches to Semitic Languages, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
3. Allauzen, C., Riley, M., Schalkwyk, J., Skut, W., and Mohri, M. (2007). OpenFst: A general and efficient weighted finite-state transducer library. In Proceedings of the Ninth International Conference on Implementation and Application of Automata, (CIAA 2007), volume 4783 of Lecture Notes in Computer Science, pages 11–23. Springer. [http://www.openfst.org](http://www.openfst.org). [↗](http://www.openfst.org)
4. Carter, S., Tsagkias, M., and Weerkamp, W. (2011). Twitter hashtags: Joint translation and clustering. In Proceedings of the 3rd International Conference on Web Science. [find](https://scholar.google.com/scholar?q=Carter%2C+S%2E%2C+Tsagkias%2C+M%2E%2C+and+Weerkamp%2C+W%2E+%282011%29%2E+Twitter+hashtags%3A+Joint+translation+and+clustering%2E+In+Proceedings+of+the+3rd+International+Conference+on+Web+Science%2E)
5. Chalabi, A. and Gerges, H. (2012). [Romanized Arabic transliteration](https://md.research.elamur.ai/free/en/papers/chalabi-2012-romanized-arabic-transliteration/). In Proceedings of the Second Workshop on Advances in Text Input Methods, pages 89–96, Mumbai, India.
6. The COLING 2012 Organizing Committee. [find](https://scholar.google.com/scholar?q=The+COLING+2012+Organizing+Committee%2E)
7. Darwish, K. (2013). [Arabizi detection and conversion to Arabic](https://md.research.elamur.ai/free/en/papers/darwish-2014-arabizi-detection-conversion-arabic/). arXiv:1306.6755 [cs.CL], arXiv. http://arxiv.org/abs/1306.6755. [↗](https://arxiv.org/abs/1306.6755)
8. Dyer, C., Lopez, A., Ganitkevitch, J., Weese, J., Ture, F., Blunsom, P., Setiawan, H., Eidelman, V., and Resnik, P. (2010). [cdec: A decoder, alignment, and learning framework for finite-state and context-free translation models](https://md.research.elamur.ai/free/en/papers/dyer-2010-cdec-decoder-alignment-learning-framework/). In Proceedings of the ACL 2010 System Demonstrations, pages 7–12, Uppsala, Sweden. Association for Computational Linguistics.
9. Irvine, A., Callison-Burch, C., and Klementiev, A. (2010). Transliterating from all languages. In Proceedings of the 9th Conference of the Association for Machine Translation in the Americas, Denver, Colorado, USA. [find](https://scholar.google.com/scholar?q=Irvine%2C+A%2E%2C+Callison%2DBurch%2C+C%2E%2C+and+Klementiev%2C+A%2E+%282010%29%2E+Transliterating+from+all+languages%2E+In+Proceedings+of+the+9th+Conference+of+the+Association+for+Machine+Translation+in+the+Americas%2C+Denver%2C+)
10. Irvine, A., Weese, J., and Callison-Burch, C. (2012). Processing informal, romanized Pakistani text messages. In Proceedings of the Second Workshop on Language in Social Media, pages 75–78, Montréal, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Irvine%2C+A%2E%2C+Weese%2C+J%2E%2C+and+Callison%2DBurch%2C+C%2E+%282012%29%2E+Processing+informal%2C+romanized+Pakistani+text+messages%2E+In+Proceedings+of+the+Second+Workshop+on+Language+in+Social+Media%2C+pages+75%E2%80%9378%2C+Montre%CC%81al%2C)
11. Jiang, J., Way, A., and Haque, R. (2012). Translating user-generated content in the social networking space. In Proceedings of AMTA-2012, the Tenth Biennial Conference of the Association for Machine Translation in the Americas. [find](https://scholar.google.com/scholar?q=Jiang%2C+J%2E%2C+Way%2C+A%2E%2C+and+Haque%2C+R%2E+%282012%29%2E+Translating+user%2Dgenerated+content+in+the+social+networking+space%2E+In+Proceedings+of+AMTA%2D2012%2C+the+Tenth+Biennial+Conference+of+the+Association+for+Machine+T)
12. Johnson, H., Martin, J., Foster, G., and Kuhn, R. (2007). [Improving translation quality by discarding most of the phrasetable](https://md.research.elamur.ai/free/en/papers/johnson-2007-improving-translation-quality-discarding-most/). In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 967–975, Prague, Czech Republic. Association for Computational Linguistics.
13. Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007). Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, ACL ’07, pages 177–180, Stroudsburg, PA, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Koehn%2C+P%2E%2C+Hoang%2C+H%2E%2C+Birch%2C+A%2E%2C+Callison%2DBurch%2C+C%2E%2C+Federico%2C+M%2E%2C+Bertoldi%2C+N%2E%2C+Cowan%2C+B%2E%2C+Shen%2C+W%2E%2C+Moran%2C+C%2E%2C+Zens%2C+R%2E%2C+Dyer%2C+C%2E%2C+Bojar%2C+O%2E%2C+Constantin%2C+A%2E%2C+and+Herbst%2C+E%2E+%282007%29%2E+Moses%3A+Open+sourc)
14. Koehn, P., Och, F. J., and Marcu, D. (2003). Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1, NAACL ’03, pages 48–54, Stroudsburg, PA, USA. Association for Computational Linguistics. [↗](https://doi.org/10.21236/ada461156)
15. Lee, Y. K., Haghighi, A., and Barzilay, R. (2011). Modeling syntactic context improves morphological segmentation. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning, pages 1–9, Portland, Oregon, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Lee%2C+Y%2E+K%2E%2C+Haghighi%2C+A%2E%2C+and+Barzilay%2C+R%2E+%282011%29%2E+Modeling+syntactic+context+improves+morphological+segmentation%2E+In+Proceedings+of+the+Fifteenth+Conference+on+Computational+Natural+Language+Learning)
16. Mohri, M. (1997). [Finite-state transducers in language and speech processing](https://md.research.elamur.ai/free/en/papers/mohri-1997-finite-state-transducers-language-speech/). Computational Linguistics, 23(2):269–311.
17. Och, F. J. and Ney, H. (2003). A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. [↗](https://doi.org/10.1162/089120103321337421)
18. Pennell, D. and Liu, Y. (2011). A character-level machine translation approach for normalization of sms abbreviations. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 974–982, Chiang Mai, Thailand. Asian Federation of Natural Language Processing. [find](https://scholar.google.com/scholar?q=Pennell%2C+D%2E+and+Liu%2C+Y%2E+%282011%29%2E+A+character%2Dlevel+machine+translation+approach+for+normalization+of+sms+abbreviations%2E+In+Proceedings+of+5th+International+Joint+Conference+on+Natural+Language+Processi)
19. Post, M., Ganitkevitch, J., Orland, L., Weese, J., Cao, Y., and Callison-Burch, C. (2013). Joshua 5.0: Sparser, better, faster, server. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 206–212, Sofia, Bulgaria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Post%2C+M%2E%2C+Ganitkevitch%2C+J%2E%2C+Orland%2C+L%2E%2C+Weese%2C+J%2E%2C+Cao%2C+Y%2E%2C+and+Callison%2DBurch%2C+C%2E+%282013%29%2E+Joshua+5%2E0%3A+Sparser%2C+better%2C+faster%2C+server%2E+In+Proceedings+of+the+Eighth+Workshop+on+Statistical+Machine+Tra)

*Updated 1 November 2026*