---
title: "Normalization for Automated Metrics: English and Arabic Speech"
description: "The authors argue that Arabic properties break the assumptions behind these metrics. 14 references, cited by 3 papers here."
authors: ["Sherri Condon", "Gregory A. Sanders", "Dan Parvaz", "Alan Rubenstein", "Christy Doran", "John Aberdeen", "Beatrice Oshika"]
year: 2009
venue: "Proceedings of Machine Translation Summit XII: Papers"
source: "https://aclanthology.org/2009.mtsummit-papers.3/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/condon-2009-normalization-automated-metrics-english-arabic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/condon-2009-normalization-automated-metrics-english-arabic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2009](https://md.research.elamur.ai/free/en/papers/2009/page/2/) / Normalization for Automated Metrics: English and Arabic Speech Translation

# Normalization for Automated Metrics: English and Arabic Speech Translation

**Authors:** [Sherri Condon](https://md.research.elamur.ai/free/en/authors/sherri-condon/), Gregory A. Sanders, [Dan Parvaz](https://md.research.elamur.ai/free/en/authors/dan-parvaz/), Alan Rubenstein, [Christy Doran](https://md.research.elamur.ai/free/en/authors/christy-doran/), [John Aberdeen](https://md.research.elamur.ai/free/en/authors/john-aberdeen/) and Beatrice Oshika · **Year:** 2009 · **Venue:** [Proceedings of Machine Translation Summit XII: Papers](https://md.research.elamur.ai/free/en/venues/mtsummit/) ([Machine Translation Summit (MT Summit) 2009](https://md.research.elamur.ai/free/en/venues/mtsummit/2009/)) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/2009.mtsummit-papers.3/) · [PDF](https://aclanthology.org/2009.mtsummit-papers.3.pdf) · [OpenAlex](https://openalex.org/W3198653489)

**Citations:** 8 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/condon-2009-normalization-automated-metrics-english-arabic/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
In the DARPA TRANSTAC speech translation evaluations, automated metrics such as BLEU, TER and METEOR tracked bilingual human judgments well for output in English but poorly for output in Iraqi Arabic. They also gave English→Arabic lower scores than Arabic→English, even though human judges rated English→Arabic translations as better. The authors argue that Arabic properties break the assumptions behind these metrics. Those properties are clitics attached to words, root-and-pattern morphology, unwritten short vowels, variable spelling and diacritics, and freer word order. To test this, they normalized Arabic text before scoring. Norm1 is a rule-based Perl script that collapses orthographic variants. Norm2 variants use the light10 light stemmer to either split off affixes and function words or remove them. They then correlated BLEU with two human measures: Likert semantic adequacy and a content-word transfer score. Removing function morphemes and scoring only unigrams gave the strongest agreement with human judges. The paper shows that metric validity for morphologically rich targets depends heavily on preprocessing, and that the choice of n-gram order also matters.

## Key points
- At the system level, BLEU, TER, METEOR and human scores correlated above 0.9, but correlations were much weaker on finer-grained units. They were also weaker for translation into Arabic.
- The human judgments came from five bilingual judges per evaluation. Semantic adequacy was rated per utterance, first on a four-point scale and later on a seven-point, equally spaced scale.
- For the correlation analysis, 13 dialog excerpts supplied a 1431-word English set of 109 utterances and a 1085-word Iraqi Arabic set of 96 utterances. With 3 systems, this gave 39 data points for Pearson's R.
- The normalizations were cumulative:
  - Norm1 handles seven types of orthographic variation.
  - Norm2a separates light10-identified affixes and function words.
  - Norm2b deletes them.
- In Table 3, BLEU_1 with Norm2b correlated about 0.75 with Likert adequacy, compared with about 0.50 for BLEU_1 with Norm2. BLEU_4 was always weaker. Speech-recognition WER correlated negatively with the human scores (about −0.63 and −0.67). The two human measures agreed most strongly with each other (0.86).

## Interesting ideas
- Build two variants of the same normalization, one that splits off function morphemes and one that drops them, and see which tracks human judgment better; this reveals that judges weight content words over grammatical glue.
- Unigram-only matching serves as a proxy for adequacy that ignores word order, which avoids penalizing legitimate reorderings missing from the references.
- A metric that looks fine at the system level can still be weak per item, so validate it against humans at the finest feasible granularity.
- Check whether each content word's meaning was carried over, as a human metric alongside a holistic adequacy rating; the paper found the two agreed strongly.
- Tokenization assumptions built into a metric silently decide what counts as a match, so clitic handling must happen before scoring.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Apply a split-vs-drop test to our evaluation scripts to find out whether a metric's errors come from function words or from content lemmas.
- **I2** (back): Score the text → graph → text compressor's round trips with content-lemma unigram overlap, so valid paraphrases with different word order are not counted as failures.
- **I3** (learn): Validate our evaluation metrics on per-sentence or per-excerpt slices, not only on corpus totals, before trusting them as training signals.
- **I4** (frames): Add a per-argument check that predicates and semantic-role fillers survive a transformation, giving an explainable alternative to a single overall score.
- **I5** (tokens): Split English contractions and possessive 's consistently in both system output and gold data, so mismatches reflect real errors rather than tokenizer disagreement.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (5): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 1,087 works · [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 480 works

## Cite

Cite as: Sherri Condon, Gregory A. Sanders, Dan Parvaz, Alan Rubenstein, Christy Doran, John Aberdeen and Beatrice Oshika. 2009. Normalization for Automated Metrics: English and Arabic Speech Translation. Proceedings of Machine Translation Summit XII: Papers.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{condon2009normalization,
  title = {{Normalization for Automated Metrics: English and Arabic Speech Translation}},
  author = {Sherri Condon, Gregory A. Sanders, Dan Parvaz, Alan Rubenstein, Christy Doran, John Aberdeen and Beatrice Oshika},
  booktitle = {Proceedings of Machine Translation Summit XII: Papers},
  year = {2009},
  url = {https://aclanthology.org/2009.mtsummit-papers.3/}
}
```

- [← previous Improvements To Monolingual English Word Sense Disambiguation](https://md.research.elamur.ai/free/en/papers/guo-2009-improvements-monolingual-english-word-sense/)
- [next → Adjective Density as a Text Formality Characteristic for Automatic Text Classification: A Study Based on the British National Corpus](https://md.research.elamur.ai/free/en/papers/fang-2009-adjective-density-as-text-formality/)
- [↑ back to papers of 2009](https://md.research.elamur.ai/free/en/papers/2009/page/2/)

## References

14 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Satanjeev Banerjee and Alon Lavie. 2005. [METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/). Proceedings of the ACL 2005 Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization, pages 65-73.
2. Niladri Chatterjee, Anish Johnson, and Madhav Krishna. 2007. Some Improvements over the BLEU Metric for Measuring Translation Quality for Hindi. Proceedings of the International Conference on Computing: Theory and Applications 2007, pages 485-90. [find](https://scholar.google.com/scholar?q=Niladri+Chatterjee%2C+Anish+Johnson%2C+and+Madhav+Krishna%2E+2007%2E+Some+Improvements+over+the+BLEU+Metric+for+Measuring+Translation+Quality+for+Hindi%2E+Proceedings+of+the+International+Conference+on+Computin)
3. Christopher Culy and Susanne Riehemann. 2003. The Limits of N-gram Translation Evaluation Metrics. Proceedings of the MT Summit IX, AMTA, pp. 71-79. [find](https://scholar.google.com/scholar?q=Christopher+Culy+and+Susanne+Riehemann%2E+2003%2E+The+Limits+of+N%2Dgram+Translation+Evaluation+Metrics%2E+Proceedings+of+the+MT+Summit+IX%2C+AMTA%2C+pp%2E+71%2D79%2E)
4. Sherri Condon, Jon Phillips, Christy Doran, John Aberdeen, Dan Parvaz, Beatrice Oshika, Greg Sanders, and Craig Schlenoff. 2008. Applying Automated Metrics to Speech Translation Dialogs. Proceedings of LREC-2008. [find](https://scholar.google.com/scholar?q=Sherri+Condon%2C+Jon+Phillips%2C+Christy+Doran%2C+John+Aberdeen%2C+Dan+Parvaz%2C+Beatrice+Oshika%2C+Greg+Sanders%2C+and+Craig+Schlenoff%2E+2008%2E+Applying+Automated+Metrics+to+Speech+Translation+Dialogs%2E+Proceedings+o)
5. Nizar Habash, Abdelhadi Soudi, and Tim Buckwalter. 2007. On Arabic transliteration. In Abdelhadi Soudi, Antal van den Bosch, and Günter Neumann, editors, Arabic Computational Morphology: Knowledge-based and empirical methods, volume 38 of Text, Speech and Language Technology, Springer Verlag. [↗](https://doi.org/10.1007/978-1-4020-6046-5_2)
6. Leah S. Larkey, Lisa Ballesteros and Margaret E. Connell. 2007. Light stemming for Arabic information retrieval. In Abdelhadi Soudi, Antal van den Bosch, and Günter Neumann, editors, Arabic Computational Morphology: Knowledge-based and empirical method, volume 38 of Text, Speech and Language Technology, Springer Verlag. [↗](https://doi.org/10.1007/978-1-4020-6046-5_12)
7. Alon Lavie, Kenji Sagae, and Shyamsundar Jayaraman. 2004. The Significance of Recall in Automatic Metrics for MT Evaluation. Proceedings of the 6th Conference of the Association for Machine Translation in the Americas (AMTA-2004), pages 134–143. [find](https://scholar.google.com/scholar?q=Alon+Lavie%2C+Kenji+Sagae%2C+and+Shyamsundar+Jayaraman%2E+2004%2E+The+Significance+of+Recall+in+Automatic+Metrics+for+MT+Evaluation%2E+Proceedings+of+the+6th+Conference+of+the+Association+for+Machine+Translatio)
8. Karolina Owczarzak, Josef van Genabith, and Andy Way. 2007. Dependency-Based Automatic Evaluation for Machine Translation. Proceedings of HLT-NAACL 2007 AMTA Workshop on Syntax and Structure in Statistical Translation, pages 80-87. [find](https://scholar.google.com/scholar?q=Karolina+Owczarzak%2C+Josef+van+Genabith%2C+and+Andy+Way%2E+2007%2E+Dependency%2DBased+Automatic+Evaluation+for+Machine+Translation%2E+Proceedings+of+HLT%2DNAACL+2007+AMTA+Workshop+on+Syntax+and+Structure+in+Statis)
9. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: A method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). Proceedings of ACL 2002, pages 311-318.
10. Greg Sanders, Sebastién Bronsart, Sherri Condon, and Craig Schlenoff. 2008. Odds of successful transfer of low-level concepts: A key metric for bidirectional speech-to-speech machine translation in DARPA‘s TRANSTAC program. Proceedings of LREC 2008. [find](https://scholar.google.com/scholar?q=Greg+Sanders%2C+Sebasti%C3%A9n+Bronsart%2C+Sherri+Condon%2C+and+Craig+Schlenoff%2E+2008%2E+Odds+of+successful+transfer+of+low%2Dlevel+concepts%3A+A+key+metric+for+bidirectional+speech%2Dto%2Dspeech+machine+translation+in+DA)
11. Wade Shen, Brian Delaney, Tim Anderson, and Ray Slyh. 2007. The MIT-LL/AFRL IWSLT-2007 MT System. Proceedings of the 2007 International Workshop on Spoken Language Translation, Trento, Italy. [find](https://scholar.google.com/scholar?q=Wade+Shen%2C+Brian+Delaney%2C+Tim+Anderson%2C+and+Ray+Slyh%2E+2007%2E+The+MIT%2DLL%2FAFRL+IWSLT%2D2007+MT+System%2E+Proceedings+of+the+2007+International+Workshop+on+Spoken+Language+Translation%2C+Trento%2C+Italy%2E)
12. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciula and John Makhoul. 2006. A Study of Translation Error Rate with Targeted Human Annotation. Proceedings of AMTA 2006, pages 223-231. Joseph Turian, Luke Shen, and I. Dan. [find](https://scholar.google.com/scholar?q=Matthew+Snover%2C+Bonnie+Dorr%2C+Richard+Schwartz%2C+Linnea+Micciula+and+John+Makhoul%2E+2006%2E+A+Study+of+Translation+Error+Rate+with+Targeted+Human+Annotation%2E+Proceedings+of+AMTA+2006%2C+pages+223%2D231%2E+Joseph)
13. Melamed. 2003. Evaluation of Machine Translation and Its Evaluation Proceedings of MT Summit 2003, pages 386-393. [find](https://scholar.google.com/scholar?q=Melamed%2E+2003%2E+Evaluation+of+Machine+Translation+and+Its+Evaluation+Proceedings+of+MT+Summit+2003%2C+pages+386%2D393%2E)
14. Brian Weiss, Craig Schlenoff, Greg Sanders, Michelle Potts Steves, Sherri Condon, Jon Phillips, and Dan Parvaz. 2008. Performance Evaluation of Speech Translation Systems. Proceedings of LREC 2008. [find](https://scholar.google.com/scholar?q=Brian+Weiss%2C+Craig+Schlenoff%2C+Greg+Sanders%2C+Michelle+Potts+Steves%2C+Sherri+Condon%2C+Jon+Phillips%2C+and+Dan+Parvaz%2E+2008%2E+Performance+Evaluation+of+Speech+Translation+Systems%2E+Proceedings+of+LREC+2008%2E)

*Updated 1 November 2026*