---
title: "Evaluation of Machine Translation Errors in English and Iraqi Arabic"
description: "The paper tackles a gap in machine translation evaluation. 12 references, cited by 6 papers here."
authors: ["Sherri Condon", "Dan Parvaz", "John Aberdeen", "Christy Doran", "Andrew Freeman", "Marwan Awad"]
year: 2010
venue: "Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)"
source: "https://aclanthology.org/L10-1065/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/condon-2010-evaluation-machine-translation-errors-english/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/condon-2010-evaluation-machine-translation-errors-english/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2010](https://md.research.elamur.ai/free/en/papers/2010/page/2/) / Evaluation of Machine Translation Errors in English and Iraqi Arabic

# Evaluation of Machine Translation Errors in English and Iraqi Arabic

**Authors:** [Sherri Condon](https://md.research.elamur.ai/free/en/authors/sherri-condon/), [Dan Parvaz](https://md.research.elamur.ai/free/en/authors/dan-parvaz/), [John Aberdeen](https://md.research.elamur.ai/free/en/authors/john-aberdeen/), [Christy Doran](https://md.research.elamur.ai/free/en/authors/christy-doran/), [Andrew Freeman](https://md.research.elamur.ai/free/en/authors/andrew-freeman/) and Marwan Awad · **Year:** 2010 · **Venue:** [Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)](https://md.research.elamur.ai/free/en/venues/lrec/) ([LREC 2010](https://md.research.elamur.ai/free/en/venues/lrec/2010/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/L10-1065/) · [PDF](https://aclanthology.org/L10-1065.pdf) · [OpenAlex](https://openalex.org/W1524845196)

**Citations:** 16 citations (OpenAlex, 2026-10-14) · [6](https://md.research.elamur.ai/free/en/papers/condon-2010-evaluation-machine-translation-errors-english/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper tackles a gap in machine translation evaluation. Human ratings and automatic scores say how good a translation is, but not what went wrong. MITRE linguists analyzed errors from 4 English–Iraqi Arabic speech translation systems built for DARPA's TRANSTAC program, at two evaluations in 2008. They used text transcripts, so speech recognition errors were excluded.

The method borrows from HTER. Annotators post-edit a reference so that it stays as close as possible to the system output while keeping the source meaning. TER deletions, insertions, substitutions and shifts are then labeled by word class, and harmless differences are marked as null.

The proportions of error types barely changed between the two evaluations, even though the systems produced fewer errors overall. Translations into English suffered mostly from pronoun and copula problems. Translations into Iraqi Arabic were dominated by substitutions, especially subject person inflection on verbs. Polarity flips were rare.

The authors trace most errors to structural contrasts between the languages, such as dropped subjects, an absent copula and head–modifier order. These contrasts force a translation to add or drop words. Some cases, such as pronoun gender and person, require discourse context to resolve.

## Key points
- Data: June 2008 and November 2008 TRANSTAC evaluations. The source utterances were:
  - 109 English→Iraqi Arabic and 97 Iraqi Arabic→English in June;
  - 93 English→Iraqi Arabic and 108 Iraqi Arabic→English in November.
- The translations analyzed came from 4 systems: 436/388 in June and 372/432 in November. Most systems were statistical; one English→Arabic engine was rule-based with a statistical fallback.
- The annotation combines a TER operation with a word class (pronoun, noun, verb, other), for example "ivp" for an inserted verb. Two extra categories were added for Arabic verb inflection:
  - **Pro-Verb**, for a single element combining pronoun and verb;
  - **Subject Person**, for wrong person inflection on the verb.
- Into English, pronouns and verbs together account for about half of all errors. In June, forms of *be* made up 38–52% of verb errors, depending on the system. Pronoun and copula errors together averaged over 40% of all Iraqi→English errors.
- Into Iraqi Arabic, nearly half of all errors are substitutions, and verb-related errors (including inflection) reach about 22%. Polarity reversals stay under 1% in both directions, and speech-act changes are very rare.

## Interesting ideas
- Build a "targeted" reference that is edited to sit as near as possible to the system output, so the remaining differences reflect real mistakes rather than legitimate paraphrase.
- Mark trivial mismatches as null instead of counting every difference, using a written list: articles, number inflection, repetitions, synonyms, and an omitted *and* that does not change meaning.
- Cross each edit operation (insert, delete, substitute, shift) with a word class to get a diagnostic error table instead of a single score.
- Many errors come from material that has no counterpart in the source, such as dropped subjects, an absent present-tense copula, or resumptive pronouns that English does not need.
- Choosing between "he", "she" and "it", or between first and second person on Iraqi past-tense verbs, depends on the referent and the dialogue context rather than on the sentence alone.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (back): When scoring text regenerated from the meaning graph, compare it against a minimally edited version of the output rather than the original sentence, so that paraphrase is not counted as error.
- **I2** (learn): Evaluation of witness parsers and silver data could use an explicit list of harmless differences, so that the agreement metrics reflect meaningful disagreements.
- **I3** (tree): Report parser errors as a matrix of UPOS × error type (attachment, label, missing or extra token), which points directly at which repair rules or training data are needed.
- **I4** (back): Translation through the meta-world should explicitly decide which implicit elements (a subject, a copula) to realize or suppress in the target language, rather than mapping words one to one.
- **I5** (frames): Coreference and speaker/addressee tracking in prag could supply this information, so that pronoun choice is grounded in the entity's identity rather than guessed locally.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Mitre](https://md.research.elamur.ai/free/en/companies/mitre/) — Company, Bedford, United States · 10 works · 2,631 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (4): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 1,087 works · [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Sherri Condon, Dan Parvaz, John Aberdeen, Christy Doran, Andrew Freeman and Marwan Awad. 2010. Evaluation of Machine Translation Errors in English and Iraqi Arabic. Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{condon2010evaluation,
  title = {{Evaluation of Machine Translation Errors in English and Iraqi Arabic}},
  author = {Sherri Condon, Dan Parvaz, John Aberdeen, Christy Doran, Andrew Freeman and Marwan Awad},
  booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)},
  year = {2010},
  doi = {10.21236/ada576234},
  url = {https://doi.org/10.21236/ada576234}
}
```

- [← previous Detecting Speculative Language Using Syntactic Dependencies and Logistic Regression](https://md.research.elamur.ai/free/en/papers/vlachos-2010-detecting-speculative-language-syntactic-dependencies/)
- [next → Exploring the Data-Driven Prediction of Prepositions in English](https://md.research.elamur.ai/free/en/papers/elghafari-2010-exploring-data-driven-prediction-prepositions/)
- [↑ back to papers of 2010](https://md.research.elamur.ai/free/en/papers/2010/page/2/)

## References

12 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Banerjee, S. and Lavie, A. (2005). [METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/). In Proceedings of the ACL 2005 Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization, pp. 65-73.
2. Condon, S., Sanders, G., Parvaz, D., Rubenstein, A., Doran, C., Aberdeen, J., and Oshika, B. (2009). [Normalization for Automated Metrics: English and Arabic Speech Translation](https://md.research.elamur.ai/free/en/papers/condon-2009-normalization-automated-metrics-english-arabic/). Proceedings of MT Summit XII.
3. Llitjós, A., Carbonell, J., and Lavie, A. (2005). A framework for interactive and automatic refinement of transfer-based machine translation. In Proc. Of the 10th Annual Conf. of the European Association for Machine Translation (EAMT), Budapest, Hungary,May. [find](https://scholar.google.com/scholar?q=Llitj%C3%B3s%2C+A%2E%2C+Carbonell%2C+J%2E%2C+and+Lavie%2C+A%2E+%282005%29%2E+A+framework+for+interactive+and+automatic+refinement+of+transfer%2Dbased+machine+translation%2E+In+Proc%2E+Of+the+10th+Annual+Conf%2E+of+the+European+Associat)
4. NIST. (2007). Post Editing Guidelines for GALE Machine Translation Evaluation Version 3.0.2. [http://projects.ldc.upenn.edu/gale/Translation/Editors/GALEpostedit_guidelines-3.0.2.pdf](http://projects.ldc.upenn.edu/gale/Translation/Editors/GALEpostedit_guidelines-3.0.2.pdf) [↗](http://projects.ldc.upenn.edu/gale/Translation/Editors/GALEpostedit_guidelines-3.0.2.pdf)
5. NIST. (2008). MTPostEditor_V1.2.0.jar. V1.2.2 available at Linguistic Data Consortium (LDC). GALE: Machine Translation Post-Editing Resource page for current post-editors. [http://projects.ldc.upenn.edu/gale/Translation/Editors/](http://projects.ldc.upenn.edu/gale/Translation/Editors/) [↗](http://projects.ldc.upenn.edu/gale/Translation/Editors/)
6. Popović, M. and Ney, H. (2007). Word Error Rates: Decomposition over POS Classes and Applications for Error Analysis. In Proceedings of the Second Workshop on Statistical Machine Translation, pp. 48–55. [find](https://scholar.google.com/scholar?q=Popovi%C4%87%2C+M%2E+and+Ney%2C+H%2E+%282007%29%2E+Word+Error+Rates%3A+Decomposition+over+POS+Classes+and+Applications+for+Error+Analysis%2E+In+Proceedings+of+the+Second+Workshop+on+Statistical+Machine+Translation%2C+pp%2E+48%E2%80%935)
7. Przybocki, M., Peterson, K., and Bronsart, S. (2008). Official results of the NIST 2008 "Metrics for MAchine TRanslation" Challenge (MetricsMATR08), [http://nist.gov/speech/tests/metricsmatr/2008/results/Sanders](http://nist.gov/speech/tests/metricsmatr/2008/results/Sanders), [↗](http://nist.gov/speech/tests/metricsmatr/2008/results/Sanders)
8. G., Bronsart, S., Condon, S., and Schlenoff, C. (2008). Odds of successful transfer of low-level concepts: A key metric for bidirectional speech-to-speech machine translation in DARPA‟s TRANSTAC program. Proceedings of LREC 2008. [find](https://scholar.google.com/scholar?q=G%2E%2C+Bronsart%2C+S%2E%2C+Condon%2C+S%2E%2C+and+Schlenoff%2C+C%2E+%282008%29%2E+Odds+of+successful+transfer+of+low%2Dlevel+concepts%3A+A+key+metric+for+bidirectional+speech%2Dto%2Dspeech+machine+translation+in+DARPA%E2%80%9Fs+TRANSTAC+progr)
9. Snover, M, Dorr, B., Schwartz, R., Micciula, L., and Makhoul, J. (2006). A Study of Translation Error Rate with Targeted Human Annotation. Proceedings of AMTA 2006, pp. 223-231. [find](https://scholar.google.com/scholar?q=Snover%2C+M%2C+Dorr%2C+B%2E%2C+Schwartz%2C+R%2E%2C+Micciula%2C+L%2E%2C+and+Makhoul%2C+J%2E+%282006%29%2E+A+Study+of+Translation+Error+Rate+with+Targeted+Human+Annotation%2E+Proceedings+of+AMTA+2006%2C+pp%2E+223%2D231%2E)
10. Tillmann , C., Vogel, S., Ney, H., Zubiaga, A., and Sawaf, H. (1997). Accelerated DP Based Search for Statistical Translation. In European Conference on Speech Communication and Technology, pp. 2667-2670. [find](https://scholar.google.com/scholar?q=Tillmann+%2C+C%2E%2C+Vogel%2C+S%2E%2C+Ney%2C+H%2E%2C+Zubiaga%2C+A%2E%2C+and+Sawaf%2C+H%2E+%281997%29%2E+Accelerated+DP+Based+Search+for+Statistical+Translation%2E+In+European+Conference+on+Speech+Communication+and+Technology%2C+pp%2E+2667%2D2)
11. Vilar, D., Xu, J., D‟Haro, L., and Ney, H. (2006). [Error Analysis of Statistical Machine Translation Output](https://md.research.elamur.ai/free/en/papers/vilar-2006-error-analysis-statistical-machine-translation/). In Proc. of the Fifth Int. Conf. on Language Resources and Evaluation (LREC), pp. 697–702.
12. Weiss, B., Schlenoff, C., Sanders, G., Steves, M., Condon, S., Phillips, J., and Parvaz, D. (2008). Performance Evaluation of Speech Translation Systems. Proceedings of LREC 2008. [find](https://scholar.google.com/scholar?q=Weiss%2C+B%2E%2C+Schlenoff%2C+C%2E%2C+Sanders%2C+G%2E%2C+Steves%2C+M%2E%2C+Condon%2C+S%2E%2C+Phillips%2C+J%2E%2C+and+Parvaz%2C+D%2E+%282008%29%2E+Performance+Evaluation+of+Speech+Translation+Systems%2E+Proceedings+of+LREC+2008%2E)

*Updated 31 October 2026*