--- title: "Arabic Preprocessing Schemes for Statistical — Habash 2006" description: "Arabic words can carry several attached clitics (conjunctions, particles, the article, pronouns) and are often spelled inconsistently." authors: ["Nizar Habash", "Fatiha Sadat"] year: 2006 venue: "Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers" source: "https://aclanthology.org/N06-2013/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/habash-2006-arabic-preprocessing-schemes-statistical-machine/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/habash-2006-arabic-preprocessing-schemes-statistical-machine/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2006](https://md.research.elamur.ai/free/en/papers/2006/) / Arabic Preprocessing Schemes for Statistical Machine Translation # Arabic Preprocessing Schemes for Statistical Machine Translation **Authors:** [Nizar Habash](https://md.research.elamur.ai/free/en/authors/nizar-habash/), [Fatiha Sadat](https://md.research.elamur.ai/free/en/authors/fatiha-sadat/) · **Year:** 2006 · **Venue:** Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/N06-2013/) · [PDF](https://aclanthology.org/N06-2013.pdf) · [OpenAlex](https://openalex.org/W2135161317) **Citations:** 253 citations (openalex-snapshot · 2026-10-14) · [105](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary Arabic words can carry several attached clitics (conjunctions, particles, the article, pronouns) and are often spelled inconsistently. This makes word-level input sparse and ambiguous for statistical MT. Habash and Sadat test how different word-level preprocessing choices affect Arabic-to-English phrase-based SMT. They separate a *scheme* (what the output looks like) from a *technique* (how it is produced). They cross six schemes (ST, D1, D2, D3, MR, EN) with three techniques of rising linguistic cost: REGEX, BAMA and MADA. Each combination is trained on 1%, 10% and 100% of a roughly 5-million-word LDC parallel corpus using the Portage system. Results are scored with BLEU-4 on NIST MT04 (mixed genres) and MT05 (news only). With the full data, splitting off only proclitics (D2) works best. With little data, the English-like EN scheme with MADA disambiguation gives large gains over plain tokenization. Preprocessing matters most when data is scarce or when the test genre differs from training. This gives practical guidance for any morphologically rich source language. ## Key points - **Schemes:** - ST: punctuation/number splitting and diacritic removal. - D1: splits conjunctions w+, f+. - D2: also splits particles l+, k+, b+, s+. - D3: also splits Al+ and pronominal enclitics. - MR: stem plus affix morphemes. - EN: lexemes with English-like POS tags and an explicit token for the dropped subject. - **Techniques:** greedy REGEX matching (unusable for MR and EN); BAMA, taking the first of its sorted analyses; and MADA, which picks among BAMA analyses using classifiers over 10 morphological dimensions. - **Setup:** - Portage with IBM models 1 and 2 and phrases of up to 8 words. - Trigram SRILM language model built from the English side of the parallel data, 116M words of English Gigaword and 128M words of the UN corpus. - Weight tuning on 200 sentences of NIST MT03. - **Small data (1%) on MT04:** EN+MADA reaches 17.5 BLEU versus 9.4 for ST. D3+MADA reaches 16.5. - **Full data:** D2+MADA scores 37.1 on MT04, against 34.6 for ST, which is significant. On MT05 it scores 38.6 against 37.8, which is not significant. Heavier splitting (D3, MR, EN) falls below ST at this size. - **Significance thresholds:** about 1.1 BLEU at 1% training and 1.7 BLEU at larger sizes, estimated by bootstrap resampling. ## Interesting ideas - **Separate the output scheme from the method that produces it.** Testing each combination shows whether gains come from the representation itself or from better analysis. This design transfers to tokenization studies for Ukrainian or any other inflected language. - **Regenerate the word after removing a clitic, rather than just trimming characters.** Regeneration normalizes the remaining stem, so one noun does not end up spread across several surface forms. That avoids adding new sparsity while trying to remove it. - **Evaluate along a learning curve.** The best segmentation granularity changes with corpus size. Aggressive splitting helps low-resource settings but hurts once data is plentiful, so a single data size can give misleading advice. - **Use two test sets that differ in genre from each other.** One matches the training domain and one mixes genres. This exposes robustness effects of preprocessing that an in-domain test alone would hide. - **Report per-condition significance thresholds from bootstrap resampling.** This makes clear which BLEU differences are trustworthy, especially when many small configurations are compared. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) ## Cited by Cited by 105 papers of our library cite it (of the 140884 whose bibliographies we read); 15 of them have a page here, 90 not yet. - [Better Arabic Parsing: Baselines, Evaluations, and Analysis](https://md.research.elamur.ai/free/en/papers/aclbulk-2010-green-better-arabic-parsing-baselines-evaluations/) - [Arabic Morphological Analyzer with Agglutinative Affix Morphemes and Fusional Concatenation Rules](https://md.research.elamur.ai/free/en/papers/aclbulk-2012-zaraket-arabic-morphological-analyzer-agglutinative-affix/) - [Challenging Language-Dependent Segmentation for Arabic: An Application to Machine Translation and Part-of-Speech Tagging](https://md.research.elamur.ai/free/en/papers/aclbulk-2017-sajjad-challenging-language-dependent-segmentation-arabic/) - [An Arabic Morphological Analyzer and Generator with Copious Features](https://md.research.elamur.ai/free/en/papers/aclbulk-2018-taji-arabic-morphological-analyzer-generator-copious/) - [Neural Coreference Resolution for Arabic](https://md.research.elamur.ai/free/en/papers/aloraini-2020-neural-coreference-resolution-arabic/) - [Optimizing Chinese Word Segmentation for Machine Translation Performance](https://md.research.elamur.ai/free/en/papers/chang-2008-optimizing-chinese-word-segmentation-machine/) - [Generalizing Word Lattice Translation](https://md.research.elamur.ai/free/en/papers/dyer-2008-generalizing-word-lattice-translation/) - [Automatic Language Identification in Texts: A Survey](https://md.research.elamur.ai/free/en/papers/jauhiainen-2019-automatic-language-identification-texts/) - [Probabilistic Modelling of Morphologically Rich Languages](https://md.research.elamur.ai/free/en/papers/langstream-2015-botha-probabilistic-modelling-morphologically-rich-languages/) - [Arabic Multi-Dialect Segmentation: bi-LSTM-CRF vs. SVM](https://md.research.elamur.ai/free/en/papers/langstream-2017-eldesouki-arabic-multi-dialect-segmentation-bi/) - [Machine Translation between English and Syriac (East Syriac Dialect) using Statistical Machine Learning](https://md.research.elamur.ai/free/en/papers/langstream-2026-sliwa-machine-translation-between-english-syriac/) - [Machine Translation between Hebrew and Arabic: Needs, Challenges and Preliminary Solutions](https://md.research.elamur.ai/free/en/papers/lingbulk-2010-shilon-machine-translation-between-hebrew-arabic/) - [Hebrew Morphological Preprocessing for Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/lingbulk-2012-singh-hebrew-morphological-preprocessing-statistical-machine/) - [Interlinear Glosses as a Multilingual Pivot for Machine Translation: An Updated Study on Turkish with Restricted Resources](https://md.research.elamur.ai/free/en/papers/ozer-2026-interlinear-glosses-as-multilingual-pivot/) - [Machine Translation of Arabic Dialects](https://md.research.elamur.ai/free/en/papers/zbib-2012-machine-translation-arabic-dialects/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{habash2006arabic, title = {{Arabic Preprocessing Schemes for Statistical Machine Translation}}, author = {Habash, Nizar and Sadat, Fatiha}, booktitle = {Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers}, year = {2006}, doi = {10.3115/1614049.1614062}, url = {https://doi.org/10.3115/1614049.1614062} } ``` - [← previous Annotating Information Structure in a Corpus of Spoken Danish](https://md.research.elamur.ai/free/en/papers/paggio-2006-annotating-information-structure-corpus-spoken/) - [next → Automatic classification of citation function](https://md.research.elamur.ai/free/en/papers/teufel-2006-automatic-classification-citation-function/) - [↑ back to papers of 2006](https://md.research.elamur.ai/free/en/papers/2006/) ## References 13 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. T. Buckwalter. 2002. Buckwalter Arabic Morphological Analyzer. Linguistic Data Consortium. (LDC2002L49). [find](https://scholar.google.com/scholar?q=T%2E+Buckwalter%2E+2002%2E+Buckwalter+Arabic+Morphological+Analyzer%2E+Linguistic+Data+Consortium%2E+%28LDC2002L49%29%2E) 2. S. Goldwater and D. McClosky. 2005. Improving Statistical MT through Morphological Analysis. In Proc. of the Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=S%2E+Goldwater+and+D%2E+McClosky%2E+2005%2E+Improving+Statistical+MT+through+Morphological+Analysis%2E+In+Proc%2E+of+the+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E) 3. N. Habash. 2004. Large Scale Lexeme Based Arabic Morphological Generation. In Proc. of Traitement Automatique du Langage Naturel. [find](https://scholar.google.com/scholar?q=N%2E+Habash%2E+2004%2E+Large+Scale+Lexeme+Based+Arabic+Morphological+Generation%2E+In+Proc%2E+of+Traitement+Automatique+du+Langage+Naturel%2E) 4. N. Habash and O. Rambow. 2005. Tokenization, Morphological Analysis, and Part-of-Speech Tagging for Arabic in One Fell Swoop. In Proc. of the Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=N%2E+Habash+and+O%2E+Rambow%2E+2005%2E+Tokenization%2C+Morphological+Analysis%2C+and+Part%2Dof%2DSpeech+Tagging+for+Arabic+in+One+Fell+Swoop%2E+In+Proc%2E+of+the+Association+for+Computational+Linguistics+%28ACL%29%2E) 5. P. Koehn. 2004a. Pharaoh: a Beam Search Decoder for Phrase-based Statistical Machine Translation Models. In Proc. of the Association for Machine Translation in the Americas. [find](https://scholar.google.com/scholar?q=P%2E+Koehn%2E+2004a%2E+Pharaoh%3A+a+Beam+Search+Decoder+for+Phrase%2Dbased+Statistical+Machine+Translation+Models%2E+In+Proc%2E+of+the+Association+for+Machine+Translation+in+the+Americas%2E) 6. P. Koehn. 2004b. Statistical Significance Tests For Machine Translation Evaluation. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=P%2E+Koehn%2E+2004b%2E+Statistical+Significance+Tests+For+Machine+Translation+Evaluation%2E+In+Proc%2E+of+EMNLP%2E) 7. Y. Lee. 2004. Morphological Analysis for Statistical Machine Translation. In Proc. of the North American Chapter of ACL. [find](https://scholar.google.com/scholar?q=Y%2E+Lee%2E+2004%2E+Morphological+Analysis+for+Statistical+Machine+Translation%2E+In+Proc%2E+of+the+North+American+Chapter+of+ACL%2E) 8. F. Och. 2003. Minimum Error Rate Training in Statistical Machine Translation. In Proc. of ACL. [find](https://scholar.google.com/scholar?q=F%2E+Och%2E+2003%2E+Minimum+Error+Rate+Training+in+Statistical+Machine+Translation%2E+In+Proc%2E+of+ACL%2E) 9. K. Papineni, S. Roukos, T. Ward, and W. Zhu. 2001. Bleu: a Method for Automatic Evaluation of Machine Translation. Technical Report RC22176(W0109-022), IBM Research. [find](https://scholar.google.com/scholar?q=K%2E+Papineni%2C+S%2E+Roukos%2C+T%2E+Ward%2C+and+W%2E+Zhu%2E+2001%2E+Bleu%3A+a+Method+for+Automatic+Evaluation+of+Machine+Translation%2E+Technical+Report+RC22176%28W0109%2D022%29%2C+IBM+Research%2E) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/) 10. M. Popović and H. Ney. 2004. Towards the Use of Word Stems and Suffixes for Statistical Machine Translation. In Proc. of the Conference on Language Resources and Evaluation. [find](https://scholar.google.com/scholar?q=M%2E+Popovic%CC%81+and+H%2E+Ney%2E+2004%2E+Towards+the+Use+of+Word+Stems+and+Suffixes+for+Statistical+Machine+Translation%2E+In+Proc%2E+of+the+Conference+on+Language+Resources+and+Evaluation%2E) 11. S. Nießen and H. Ney. 2004. Statistical Machine Translation with Scarce Resources Using Morpho-syntactic Information. Computational Linguistics, 30(2). [find](https://scholar.google.com/scholar?q=S%2E+Nie%C3%9Fen+and+H%2E+Ney%2E+2004%2E+Statistical+Machine+Translation+with+Scarce+Resources+Using+Morpho%2Dsyntactic+Information%2E+Computational+Linguistics%2C+30%282%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/en-2004-statistical-machine-translation-scarce-resources/) 12. F. Sadat, H. Johnson, A. Agbago, G. Foster, R. Kuhn, J. Martin, and A. Tikuisis. 2005. Portage: A Phrase-based Machine Translation System. In Proc. of ACL Workshop on Building and Using Parallel Texts. [find](https://scholar.google.com/scholar?q=F%2E+Sadat%2C+H%2E+Johnson%2C+A%2E+Agbago%2C+G%2E+Foster%2C+R%2E+Kuhn%2C+J%2E+Martin%2C+and+A%2E+Tikuisis%2E+2005%2E+Portage%3A+A+Phrase%2Dbased+Machine+Translation+System%2E+In+Proc%2E+of+ACL+Workshop+on+Building+and+Using+Parallel+Texts) 13. Andreas Stolcke. 2002. Srilm - An Extensible Language Modeling Toolkit. In Proc. of International Conference on Spoken Language Processing. [find](https://scholar.google.com/scholar?q=Andreas+Stolcke%2E+2002%2E+Srilm+%2D+An+Extensible+Language+Modeling+Toolkit%2E+In+Proc%2E+of+International+Conference+on+Spoken+Language+Processing%2E)