---
title: "Arabic Dialect Handling in Hybrid Machine Translation (2010)"
description: "A dialect analysis model proposes candidate MSA words and phrases, using hand-written character… 13 references, cited by 38 papers here."
authors: ["Hassan Sawaf"]
year: 2010
venue: "Proceedings of the 9th Conference of the Association for Machine Translation in the Americas: Research Papers"
source: "https://aclanthology.org/2010.amta-papers.5/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sawaf-2010-arabic-dialect-handling-hybrid-machine/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sawaf-2010-arabic-dialect-handling-hybrid-machine/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2010](https://md.research.elamur.ai/free/en/papers/2010/) / Arabic Dialect Handling in Hybrid Machine Translation

# Arabic Dialect Handling in Hybrid Machine Translation

**Authors:** [Hassan Sawaf](https://md.research.elamur.ai/free/en/authors/hassan-sawaf/) · **Year:** 2010 · **Venue:** [Proceedings of the 9th Conference of the Association for Machine Translation in the Americas: Research Papers](https://md.research.elamur.ai/free/en/venues/amta/) ([AMTA 2010](https://md.research.elamur.ai/free/en/venues/amta/2010/)) · **Publisher:** [Association for Machine Translation in the Americas](https://md.research.elamur.ai/free/en/publishers/association-for-machine-translation-in-the-americas/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/2010.amta-papers.5/) · [PDF](https://aclanthology.org/2010.amta-papers.5.pdf) · [OpenAlex](https://openalex.org/W3170253630)

**Citations:** 65 citations (OpenAlex, 2026-10-14) · [38](https://md.research.elamur.ai/free/en/papers/sawaf-2010-arabic-dialect-handling-hybrid-machine/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper addresses a weakness of Arabic machine translation: systems trained on Modern Standard Arabic (MSA) struggle with dialectal, spontaneous and noisy input from broadcast transcripts and web text. Sawaf adds a dialect normalization stage to a hybrid MT (HMT) system. The hybrid core combines a Lexical Functional Grammar (LFG) rule-based engine with a phrase-based statistical decoder in a log-linear framework, where c-structure, f-structure, POS and semantic attributes all act as features.

The normalizer treats dialect-to-MSA conversion as a mostly monotone translation with local reordering. It has two parts. A dialect analysis model proposes candidate MSA words and phrases, using hand-written character rules plus rules learned with GIZA++ from bi-dialectal parallel corpora. An MSA language model then scores those candidates in context.

Normalization improved BLEU for the rule-based, statistical and hybrid systems alike. The gain was largest when both training and test data were normalized: about 1 point absolute on broadcast data and about 2 points on web data. The work shows that bringing nonstandard input into the variety a model was trained on is a cheap, model-agnostic gain.

## Key points
- **Dialect normalization BLEU gains (Table 3):**
  - LFG: 18.1% → 19.5%
  - SMT: 35.4% → 36.4%
  - HMT: 37.3% → 38.5%
- **Train/test normalization (Table 4, HMT):**
  - Broadcast: 35.5% baseline, 35.9% with test-only normalization, 36.4% with train and test normalized.
  - Web (NIST MT08 WB portion): 39.9% baseline, 40.5% with test-only normalization, 42.1% with train and test normalized.
- **Normalizer training data:** bi-dialectal corpora of 271K sentences (3.1M words) for broadcast and 1.6M sentences (28M words) for web. Dev sets have 5K and 8K sentences. A separate model is trained per dialect.
- **Broadcast test set:** collected by a provider in the UAE. It mixes newscasts, interviews, films and TV shows so that nearly all regional dialects appear. Table 5 gives an Iraqi Colloquial Arabic example.
- **Morphological segmentation:** shrinks a vocabulary of more than 600K words by up to 70%, to 256K morpheme units, which reduces unknown words.

## Interesting ideas
- Rather than teaching the main system every dialect, rewrite nonstandard input into the standard variety first, using a translation-style search with a strong standard-language model as judge. This isolates dialect variation in one swappable module.
- Running the normalizer over the training corpus as well as the test input gave a much bigger gain than normalizing test input alone, because the model's statistics and its inputs then live in the same space.
- The rewrite rules are induced from a corpus in which the same sentence appears in two varieties, using ordinary word alignment and phrase extraction.
- Hybridization is framed as smoothing: hand rules should win where training statistics are sparse or long-distance structure matters, and statistics should win where data are dense.
- The author notes that a strong target language model can silently drop low-frequency but critical words such as negations. Losing such a word damages "informativeness" far more than its BLEU cost suggests.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): a pre-tagging normalizer could map informal or regional English spellings to standard forms, keeping each rewrite logged so the step stays explainable.
- **I2** (learn): any normalization applied at inference should also be run over treebanks and silver data before training, with a test-only run kept as a control.
- **I3** (learn): pairs of the same sentence in two varieties or registers could be aligned to induce repair or conversion rules automatically.
- **I4** (tree): rule-derived signals could enter the perceptron parser as weighted features that dominate only for rare constructions.
- **I5** (back): the text → graph → text compressor and the generator should explicitly check that negation and key content nodes survive the round trip.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 1,105 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works

## Cite

Cite as: Hassan Sawaf. 2010. Arabic Dialect Handling in Hybrid Machine Translation. Proceedings of the 9th Conference of the Association for Machine Translation in the Americas: Research Papers.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sawaf2010arabic,
  title = {{Arabic Dialect Handling in Hybrid Machine Translation}},
  author = {Hassan Sawaf},
  booktitle = {Proceedings of the 9th Conference of the Association for Machine Translation in the Americas: Research Papers},
  year = {2010},
  url = {https://aclanthology.org/2010.amta-papers.5/}
}
```

- [← previous Discourse Constraints for Document Compression](https://md.research.elamur.ai/free/en/papers/clarke-2010-discourse-constraints-document-compression/)
- [next → Simplified Feature Set for Arabic Named Entity Recognition](https://md.research.elamur.ai/free/en/papers/hamid-2010-simplified-feature-set-arabic-named/)
- [↑ back to papers of 2010](https://md.research.elamur.ai/free/en/papers/2010/)

## References

13 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Y. Al-Onaizan, Jan Curin, M. Jahr, K. Knight, J. Lafferty, I. D. Melamed, F. J. Och, D. Purdy, N. A. Smith, and D. Yarowsky. 1999. "Statistical Machine Translation: Final Report," Johns Hopkins University 1999 Summer Workshop on Language Engineering, Center for Language and Speech Processing, Baltimore, MD. [find](https://scholar.google.com/scholar?q=Y%2E+Al%2DOnaizan%2C+Jan+Curin%2C+M%2E+Jahr%2C+K%2E+Knight%2C+J%2E+Lafferty%2C+I%2E+D%2E+Melamed%2C+F%2E+J%2E+Och%2C+D%2E+Purdy%2C+N%2E+A%2E+Smith%2C+and+D%2E+Yarowsky%2E+1999%2E+%22Statistical+Machine+Translation%3A+Final+Report%2C%22+Johns+Hopkins+Univer)
2. P. Brown, J. Cocke, S. Della Pietra, V. Della Pietra, F. Jelinek, J. Lafferty, R. Mercer, and P. Roosin. 1990. [A Statistical Approach to Machine Translation](https://md.research.elamur.ai/free/en/papers/brown-1990-statistical-approach-machine-translation/). Computational Linguistics, 16, pp. 79–85. Cambridge, MA. [↗](https://doi.org/10.5555/92858.92860)
3. P. Brown, S. Della Pietra, V. Della Pietra, and R. Mercer. 1993. The Mathematics of Statistical Machine Translation: Parameter Estimation. Computational Linguistics, 19(2), pp. 263–311. Cambridge, MA. [find](https://scholar.google.com/scholar?q=P%2E+Brown%2C+S%2E+Della+Pietra%2C+V%2E+Della+Pietra%2C+and+R%2E+Mercer%2E+1993%2E+The+Mathematics+of+Statistical+Machine+Translation%3A+Parameter+Estimation%2E+Computational+Linguistics%2C+19%282%29%2C+pp%2E+263%E2%80%93311%2E+Cambridge%2C+MA%2E)
4. E. Charniak, K. Knight, and K. Yamada. 2003. Syntax-Based Language Models for Statistical Machine Translation. In Proceedings of MT Summit IX, pp. 23–27. New Orleans, LA. [find](https://scholar.google.com/scholar?q=E%2E+Charniak%2C+K%2E+Knight%2C+and+K%2E+Yamada%2E+2003%2E+Syntax%2DBased+Language+Models+for+Statistical+Machine+Translation%2E+In+Proceedings+of+MT+Summit+IX%2C+pp%2E+23%E2%80%9327%2E+New+Orleans%2C+LA%2E)
5. R. Kaplan and J. Bresnan. 1982. Lexical-Functional Grammar: A Formal System for Grammatical Representation. In The Mental Representation of Grammatical Relations, pp. 173–281. MIT Press. [find](https://scholar.google.com/scholar?q=R%2E+Kaplan+and+J%2E+Bresnan%2E+1982%2E+Lexical%2DFunctional+Grammar%3A+A+Formal+System+for+Grammatical+Representation%2E+In+The+Mental+Representation+of+Grammatical+Relations%2C+pp%2E+173%E2%80%93281%2E+MIT+Press%2E)
6. R. Kneser, and H. Ney. 1995. Improved backing-off for m-gram language modeling. Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1. pp. 181–184. [find](https://scholar.google.com/scholar?q=R%2E+Kneser%2C+and+H%2E+Ney%2E+1995%2E+Improved+backing%2Doff+for+m%2Dgram+language+modeling%2E+Proceedings+of+the+IEEE+International+Conference+on+Acoustics%2C+Speech%2C+and+Signal+Processing%2C+volume+1%2E+pp%2E+181%E2%80%93184%2E)
7. P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar, A Constantin, E. Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translation, Annual Meeting of the Association for Computational Linguistics (ACL), demonstration session, Prague, Czech Republic. [find](https://scholar.google.com/scholar?q=P%2E+Koehn%2C+H%2E+Hoang%2C+A%2E+Birch%2C+C%2E+Callison%2DBurch%2C+M%2E+Federico%2C+N%2E+Bertoldi%2C+B%2E+Cowan%2C+W%2E+Shen%2C+C%2E+Moran%2C+R%2E+Zens%2C+C%2E+Dyer%2C+O%2E+Bojar%2C+A+Constantin%2C+E%2E+Herbst%2E+2007%2E+Moses%3A+Open+Source+Toolkit+for+Statis)
8. S. Köpru and J. Miller. 2009. [A Unification Based Approach to the Morphological Analysis and Generation of Arabic](https://md.research.elamur.ai/free/en/papers/kopru-2009-unification-based-approach-morphological-analysis/). CAASL3: Proc. of the 3rd Workshop on Computational Approaches to Arabic-script based Languages. Ottawa, ON, Canada.
9. F. J. Och, and H. Ney. 2002. [Discriminative Training and Maximum Entropy Models for Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/och-2002-discriminative-training-maximum-entropy-models/). In Proceedings of the Ninth Machine Translation Summit, pp. 295–302. New Orleans, LA.
10. F. J. Och, and H. Ney. 2004. The Alignment Template Approach to Statistical Machine Translation. In Computational Linguistics, 30(4), pp. 417–449. Cambridge, MA. [↗](https://doi.org/10.1162/0891201042544884)
11. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. 2002. [BLEU: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for the Computational Linguistics, pp. 311–318. Philadelphia, PA.
12. H. Sawaf, K. Schütz, and H. Ney. 2000. On the Use of Grammar-Based Language Models for Statistical Machine Translation. In Proceedings of the Sixth International Workshop on Parsing Technologies (IWPT), pp. 231–241. Trento, Italy. [find](https://scholar.google.com/scholar?q=H%2E+Sawaf%2C+K%2E+Sch%C3%BCtz%2C+and+H%2E+Ney%2E+2000%2E+On+the+Use+of+Grammar%2DBased+Language+Models+for+Statistical+Machine+Translation%2E+In+Proceedings+of+the+Sixth+International+Workshop+on+Parsing+Technologies+%28IWPT)
13. M. Shihadah, P. Roochnik. 1998. Lexical-Functional Grammar as a Computational-Linguistic Underpinning to Arabic Machine Translation. In Proceedings of the 6th International Conference and Exhibition on Multi-lingual Computing. Cambridge, UK. [find](https://scholar.google.com/scholar?q=M%2E+Shihadah%2C+P%2E+Roochnik%2E+1998%2E+Lexical%2DFunctional+Grammar+as+a+Computational%2DLinguistic+Underpinning+to+Arabic+Machine+Translation%2E+In+Proceedings+of+the+6th+International+Conference+and+Exhibition+o)

*Updated 1 November 2026*