---
title: "Arabic-Segmentation Combination Strategies for Statistical"
description: "Arabic SMT is hard because words carry attached clitics and written text lacks short vowels. 54.7 BLEU. 17 references."
authors: ["Saab Mansour", "Hermann Ney"]
year: 2012
venue: "Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)"
source: "https://aclanthology.org/L12-1279/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mansour-2012-arabic-segmentation-combination-strategies-statistical/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mansour-2012-arabic-segmentation-combination-strategies-statistical/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2012](https://md.research.elamur.ai/free/en/papers/2012/page/2/) / Arabic-Segmentation Combination Strategies for Statistical Machine Translation

# Arabic-Segmentation Combination Strategies for Statistical Machine Translation

**Authors:** [Saab Mansour](https://md.research.elamur.ai/free/en/authors/saab-mansour/) and [Hermann Ney](https://md.research.elamur.ai/free/en/authors/hermann-ney/) · **Year:** 2012 · **Venue:** [Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)](https://md.research.elamur.ai/free/en/venues/lrec/) ([LREC 2012](https://md.research.elamur.ai/free/en/venues/lrec/2012/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/L12-1279/) · [PDF](https://aclanthology.org/L12-1279.pdf) · [OpenAlex](https://openalex.org/W2251960914)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Arabic SMT is hard because words carry attached clitics and written text lacks short vowels. One Arabic token often maps to several English words, which hurts IBM-style alignment and raises OOV rates. Segmentation is known to help, but earlier work never compared segmenters under identical conditions. The authors collected existing tools and built new ones: a rule-based FST, statistical SVM, CRF and HMM models, and several MADA schemes. They trained the segmenters on the Arabic Treebank. They then measured each one through the same phrase-based SMT system on the small IWSLT 2010 BTEC task and on a newswire subset of NIST 2009 Arabic-English.

Any segmentation clearly beat plain tokenization. The best segmenters, however, were statistically indistinguishable from one another. The authors then applied consensus system combination to the translation outputs. Merging outputs from different segmentation methods worked better than merging several schemes from one tool (MADA), and it gave significant gains over the best single system. Mixing both kinds added a small further gain. The lesson is that diversity among methods is worth more than variation within one model.

## Key points
- Segmenters were trained on Arabic Treebank Part 1 v3.0, which has about 150,000 news tokens segmented in the ATB scheme. Some models were restricted by the BAMA v1.0 lexicon, which has about 80,000 entries.
- The SVM segmenter (YAMCHA) tags each character with labels such as B-PRE1, B-WRD, I-WRD and B-SUFF. It uses a ±5 character window and the 5 previous tag decisions. A separate binary SVM step normalizes the feminine marker.
- For NIST, the UN and ISI data were dropped, leaving about 300K sentence pairs and 5M words (roughly 6% of the full corpus). This kept genres consistent at only a small cost in quality.
- On IWSLT08, the best single system reached about 54.7 BLEU. MADA scheme combination gave +1.0 BLEU, methods combination (SEGS-ALL) gave +2.8 BLEU and −2.2 TER, and ALL added another +0.6 BLEU.
- The combination pipeline follows Leusch and Ney (2010):
  - GIZA++ pairwise alignment of the hypotheses;
  - reordering to a skeleton hypothesis;
  - a lattice rescored with system weights and a language model.
- Significance was checked with bootstrap resampling (Koehn, 2004).
- Lexicon-based and lexicon-free statistical models did not differ significantly.

## Interesting ideas
- The best combination gains come from merging systems that differ in method, not from settings of one model, because their errors are less correlated.
- Preprocessing tools are judged by their effect on the final task under fixed conditions, not by intrinsic accuracy alone.
- Bootstrap significance testing showed that the leading segmenters were effectively tied, which prevents chasing noise.
- Clitic splitting is framed as character-level tagging with prefix, stem and suffix begin/inside labels plus a local context window.
- Several outputs are aligned to one chosen skeleton output, and a consensus is then picked by weighted voting.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Silver-data witnesses should be architecturally different (perceptron parser vs. gparse vs. an external parser), not variants of a single parser, so that their agreement actually means something.
- **I2** (tokens): Compare tokenizer and lemma variants by how much they change downstream LAS and answer accuracy on the same test sets.
- **I3** (learn): Report bootstrap confidence bounds when comparing parser or reader versions, and treat small leaderboard gaps as ties unless they are significant.
- **I4** (tokens): A similar character-tagging scheme could handle English splits such as possessive 's and n't contractions, with each split decision traceable to its features.
- **I5** (tree): Merge competing UD parses by aligning them to a primary parse and voting per token on head and label, recording which parsers backed each arc for explainability.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [RWTH Aachen University](https://md.research.elamur.ai/free/en/universities/rwth-aachen-university/) — University, Aachen, Germany · 18 works · 4,791 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 958 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 1,484 works

## Cite

Cite as: Saab Mansour and Hermann Ney. 2012. Arabic-Segmentation Combination Strategies for Statistical Machine Translation. Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mansour2012arabic,
  title = {{Arabic-Segmentation Combination Strategies for Statistical Machine Translation}},
  author = {Saab Mansour and Hermann Ney},
  booktitle = {Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)},
  year = {2012},
  doi = {10.63317/5jg3g5hi6v5d},
  url = {https://doi.org/10.63317/5jg3g5hi6v5d}
}
```

- [← previous Arabic Dialects of Central Asia: Internal Development Tendencies of Arabic Language Material](https://md.research.elamur.ai/free/en/papers/chikovani-2026-arabic-dialects-central-asia-internal/)
- [next → Building a ‘New’ Semitic Language: Lexical Enrichment in Modern Assyrian and Its Problems](https://md.research.elamur.ai/free/en/papers/gasparini-2026-building-new-semitic-language-lexical/)
- [↑ back to papers of 2012](https://md.research.elamur.ai/free/en/papers/2012/page/2/)

## References

17 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Roy Bar-haim, Khalil Sima’an, and Yoad Winter. 2008. Part-of-speech tagging of modern hebrew text. Nat. Lang. Eng., 14(2):223–251. [↗](https://doi.org/10.1017/s135132490700455x)
2. Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The Mathematics of Statistical Machine Translation: Parameter Estimation. Computational Linguistics, 19(2):263–311, June. [find](https://scholar.google.com/scholar?q=Peter+F%2E+Brown%2C+Stephen+A%2E+Della+Pietra%2C+Vincent+J%2E+Della+Pietra%2C+and+Robert+L%2E+Mercer%2E+1993%2E+The+Mathematics+of+Statistical+Machine+Translation%3A+Parameter+Estimation%2E+Computational+Linguistics%2C+19%282%29)
3. Pi-Chuan Chang, Michel Galley, and Christopher D. Manning. 2008. [Optimizing chinese word segmentation for machine translation performance](https://md.research.elamur.ai/free/en/papers/chang-2008-optimizing-chinese-word-segmentation-machine/). In StatMT ’08: Proceedings of the Third Workshop on Statistical Machine Translation, pages 224–232, Morristown, NJ, USA. Association for Computational Linguistics.
4. Mona Diab, Kadri Hacioglu, and Daniel Jurafsky. 2004. [Automatic Tagging of Arabic Text: From Raw Text to Base Phrase Chunks](https://md.research.elamur.ai/free/en/papers/diab-2004-automatic-tagging-arabic-text-raw/). In Daniel Marcu Susan Dumais and Salim Roukos, editors, HLT-NAACL 2004: Short Papers, pages 149–152, Boston, Massachusetts, USA, May 2 - May 7. Association for Computational Linguistics.
5. Anas El Isbihani, Shahram Khadivi, Oliver Bender, and Hermann Ney. 2006. Morpho-syntactic Arabic Preprocessing for Arabic to English Statistical Machine Translation. In Proceedings on the Workshop on Statistical Machine Translation, pages 15–22, New York City, June. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Anas+El+Isbihani%2C+Shahram+Khadivi%2C+Oliver+Bender%2C+and+Hermann+Ney%2E+2006%2E+Morpho%2Dsyntactic+Arabic+Preprocessing+for+Arabic+to+English+Statistical+Machine+Translation%2E+In+Proceedings+on+the+Workshop+on+)
6. Nizar Habash and Owen Rambow. 2005. [Arabic tokenization, part-of-speech tagging and morphological disambiguation in one fell swoop](https://md.research.elamur.ai/free/en/papers/habash-2005-arabic-tokenization-part-speech-tagging/). In ACL ’05: Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 573–580, Morristown, NJ, USA. Association for Computational Linguistics.
7. Philipp Koehn. 2004. [Statistical Significance Tests for Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/koehn-2004-statistical-significance-tests-machine-translation/). In Proc. of the Conf. on Empirical Methods for Natural Language Processing (EMNLP), pages 388–395, Barcelona, Spain, July.
8. John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In International Conference on Machine Learning, pages 282–289. [find](https://scholar.google.com/scholar?q=John+Lafferty%2C+Andrew+McCallum%2C+and+Fernando+Pereira%2E+2001%2E+Conditional+random+fields%3A+Probabilistic+models+for+segmenting+and+labeling+sequence+data%2E+In+International+Conference+on+Machine+Learning%2C+)
9. Gregor Leusch and Hermann Ney. 2010. The rwth system combination system for wmt 2010. In ACL 2010 Joint Fifth Workshop on Statistical Machine Translation and Metrics MATR, Uppsala, Sweden, July. [find](https://scholar.google.com/scholar?q=Gregor+Leusch+and+Hermann+Ney%2E+2010%2E+The+rwth+system+combination+system+for+wmt+2010%2E+In+ACL+2010+Joint+Fifth+Workshop+on+Statistical+Machine+Translation+and+Metrics+MATR%2C+Uppsala%2C+Sweden%2C+July%2E)
10. Saib Manour, Khalil Sima’an, and Yoad Winter. 2007. [Smoothing a lexicon-based pos tagger for arabic and hebrew](https://md.research.elamur.ai/free/en/papers/manour-2007-smoothing-lexicon-based-pos-tagger/). In Proceedings of the 2007 Workshop on Computational Approaches to Semitic Languages: Common Issues and Resources, pages 97–103, Prague, Czech Republic, June. Association for Computational Linguistics.
11. Saab Mansour. 2010. Morphtagger: Hmm-based arabic segmentation for statistical machine translation. In International Workshop on Spoken Language Translation, pages 321–327, Paris, France, December. [find](https://scholar.google.com/scholar?q=Saab+Mansour%2E+2010%2E+Morphtagger%3A+Hmm%2Dbased+arabic+segmentation+for+statistical+machine+translation%2E+In+International+Workshop+on+Spoken+Language+Translation%2C+pages+321%E2%80%93327%2C+Paris%2C+France%2C+December%2E)
12. Michael Paul, Marcello Federico, and Sebastian Stücker. 2010. Overview of the IWSLT 2010 Evaluation Campaign. In Marcello Federico, Ian Lane, Michael Paul, and François Yvon, editors, Proceedings of the seventh International Workshop on Spoken Language Translation (IWSLT), pages 3–27. [↗](https://doi.org/10.5445/ir/1000030232)
13. Fuchun Peng, Fangfang Feng, and Andrew McCallum. 2004. [Chinese segmentation and new word detection using conditional random fields](https://md.research.elamur.ai/free/en/papers/peng-2004-chinese-segmentation-new-word-detection/). In COLING ’04: Proceedings of the 20th international conference on Computational Linguistics, page 562, Morristown, NJ, USA. Association for Computational Linguistics.
14. Ryan Roth, Owen Rambow, Nizar Habash, Mona Diab, and Cynthia Rudin. 2008. [Arabic morphological tagging, diacritization, and lemmatization using lexeme models and feature ranking](https://md.research.elamur.ai/free/en/papers/roth-2008-arabic-morphological-tagging-diacritization-lemmatization/). In Proceedings of ACL-08: HLT, Short Papers, pages 117–120, Columbus, Ohio, June. Association for Computational Linguistics.
15. Fatiha Sadat and Nizar Habash. 2006. Combination of Preprocessing Schemes for Statistical MT. In Proc. of the 44th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1–8, Sydney, Australia, July. [find](https://scholar.google.com/scholar?q=Fatiha+Sadat+and+Nizar+Habash%2E+2006%2E+Combination+of+Preprocessing+Schemes+for+Statistical+MT%2E+In+Proc%2E+of+the+44th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL%29%2C+pages+1%E2%80%938%2C+Syd)
16. Nikolaos Trogkanis and Charles Elkan. 2010. Conditional random fields for word hyphenation. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 366–374, Uppsala, Sweden, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nikolaos+Trogkanis+and+Charles+Elkan%2E+2010%2E+Conditional+random+fields+for+word+hyphenation%2E+In+Proceedings+of+the+48th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+366%E2%80%93374%2C+U)
17. Richard Zens and Hermann Ney. 2008. Improvements in Dynamic Programming Beam Search for Phrase-based Statistical Machine Translation. In International Workshop on Spoken Language Translation, pages 195–205, Honolulu, Hawaii, October. [find](https://scholar.google.com/scholar?q=Richard+Zens+and+Hermann+Ney%2E+2008%2E+Improvements+in+Dynamic+Programming+Beam+Search+for+Phrase%2Dbased+Statistical+Machine+Translation%2E+In+International+Workshop+on+Spoken+Language+Translation%2C+pages+19)

*Updated 24 October 2026*