---
title: "Combining Outputs from Multiple Machine Translation Systems"
description: "The paper asks whether translations from several machine translation systems built on… 36.9 BLEU. 16 references, cited by 75 papers here."
authors: ["Antti-Veikko Rosti", "Necip Fazil Ayan", "Bing Xiang", "Spyros Matsoukas", "Richard Schwartz", "Bonnie Dorr"]
year: 2007
venue: "Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference"
source: "https://aclanthology.org/N07-1029/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rosti-2007-combining-outputs-multiple-machine-translation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rosti-2007-combining-outputs-multiple-machine-translation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2007](https://md.research.elamur.ai/free/en/papers/2007/) / Combining Outputs from Multiple Machine Translation Systems

# Combining Outputs from Multiple Machine Translation Systems

**Authors:** [Antti-Veikko Rosti](https://md.research.elamur.ai/free/en/authors/antti-veikko-rosti/), Necip Fazil Ayan, [Bing Xiang](https://md.research.elamur.ai/free/en/authors/bing-xiang/), [Spyros Matsoukas](https://md.research.elamur.ai/free/en/authors/spyros-matsoukas/), [Richard Schwartz](https://md.research.elamur.ai/free/en/authors/richard-schwartz/) and [Bonnie Dorr](https://md.research.elamur.ai/free/en/authors/bonnie-dorr/) · **Year:** 2007 · **Venue:** [Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2007](https://md.research.elamur.ai/free/en/venues/naacl/2007/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/N07-1029/) · [PDF](https://aclanthology.org/N07-1029.pdf) · [OpenAlex](https://openalex.org/W2157512532)

**Citations:** 165 citations (OpenAlex, 2026-10-14) · [75](https://md.research.elamur.ai/free/en/papers/rosti-2007-combining-outputs-multiple-machine-translation/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper asks whether translations from several machine translation systems built on different paradigms can be merged into output that beats any single one of them. The authors combined six Arabic- and Chinese-to-English systems: three phrasal, two hierarchical and one syntax-based. They tested three combination strategies.
- **Sentence level:** re-rank a pooled N-best list using confidence scores and a 5-gram LM.
- **Phrase level:** build a per-sentence phrase table from the systems' source alignments and re-decode it with a phrasal decoder.
- **Word level:** apply consensus network decoding, using TER-based alignment and a skeleton chosen by minimum Bayes risk.

All weights are tuned with Powell's method, targeting TER or BLEU. Evaluation used NIST MT05 and the newsgroup part of the GALE 2006 dry-run. Sentence- and phrase-level combination gave small gains that mostly did not carry over to the test sets. Word-level combination was the most dependable and helped even with no tuned weights. The best results came from feeding the sentence- and phrase-level outputs into the word-level step as extra inputs. This matters because systems of similar accuracy make different errors, and those differences can be exploited.

## Key points
- Six systems were combined: three phrasal, two hierarchical and one syntax-based. Testing was done on NIST MT05 (newswire) together with the newsgroup data from the GALE 2006 dry-run.
- Tuning data and metrics:
  - Sentence- and phrase-level weights were tuned on NIST MT03.
  - Six-way word-level weights were tuned on MT03 plus MT04, and eight-way weights on MT04 only.
  - Arabic was tuned for TER and Chinese for BLEU, because optimizing TER on Chinese clearly lowered BLEU.
- On the tuning sets, word-level combination improved TER by about 2–3% and BLEU by about 2–4%. The other two methods improved both metrics by only about 0.5–1%.
- Chinese newswire results:
  - The 8-way BLEU-tuned combination ("BLEU 8") reached about 36.9 BLEU, against 35.3 for the strongest single system.
  - Unweighted 6-way consensus decoding gave the lowest TER, about 53.8.
- In internal HTER checks, BLEU 8 needed fewer edits after human post-editing than the unweighted 6-way output, even though the unweighted output had the better automatic TER.

## Interesting ideas
- **Stacking combiners.** The N-best output of one combination method is treated as an additional "system" for another combiner. Weak combiners can still add useful variety to a stronger downstream ensemble.
- **TER-based alignment with an MBR skeleton.** Hypotheses are aligned by edit distance, and the backbone is the hypothesis closest to all the others under TER. This handles word-order differences that a strict positional alignment cannot, and it could apply to any ensemble of sequence-generating models.
- **Unweighted voting as a strong baseline.** Plain consensus decoding with no tuned weights already beat the more elaborate methods. Simple agreement voting is worth checking first before building complex ensembles.
- **Graded agreement levels for phrase confidence.** Phrase confidence increases with how strongly systems agree, from exact matches on both source span and position down to merely overlapping target words. This graded notion of agreement could be reused for confidence estimation in other ensemble or annotation-merging settings.
- **Not optimizing one metric blindly.** Since it was unclear whether TER or BLEU better tracks the human-edited HTER, the authors watched both and picked the metric to tune for each language. This is a sensible guard whenever automatic proxies for a human judgement disagree.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 958 works

## Cite

Cite as: Antti-Veikko Rosti, Necip Fazil Ayan, Bing Xiang, Spyros Matsoukas, Richard Schwartz and Bonnie Dorr. 2007. Combining Outputs from Multiple Machine Translation Systems. Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{rosti2007combining,
  title = {{Combining Outputs from Multiple Machine Translation Systems}},
  author = {Antti-Veikko Rosti, Necip Fazil Ayan, Bing Xiang, Spyros Matsoukas, Richard Schwartz and Bonnie Dorr},
  booktitle = {Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference},
  year = {2007},
  url = {https://aclanthology.org/N07-1029/}
}
```

- [← previous A Comparison of Pivot Methods for Phrase-Based Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/utiyama-2007-comparison-pivot-methods-phrase-based/)
- [next → SemEval-2007 Task 07: Coarse-Grained English All-Words Task](https://md.research.elamur.ai/free/en/papers/navigli-2007-semeval-2007-task-07-coarse/)
- [↑ back to papers of 2007](https://md.research.elamur.ai/free/en/papers/2007/)

## References

16 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Srinivas Bangalore, German Bordel, and Giuseppe Riccardi. 2001. Computing consensus translation from multiple machine translation systems. In Proc. ASRU, pages 351–354. [find](https://scholar.google.com/scholar?q=Srinivas+Bangalore%2C+German+Bordel%2C+and+Giuseppe+Riccardi%2E+2001%2E+Computing+consensus+translation+from+multiple+machine+translation+systems%2E+In+Proc%2E+ASRU%2C+pages+351%E2%80%93354%2E)
2. Richard P. Brent. 1973. Algorithms for Minimization Without Derivatives. Prentice-Hall. [find](https://scholar.google.com/scholar?q=Richard+P%2E+Brent%2E+1973%2E+Algorithms+for+Minimization+Without+Derivatives%2E+Prentice%2DHall%2E)
3. David Chiang. 2005. [A hierarchical phrase-based model for statistical machine translation](https://md.research.elamur.ai/free/en/papers/chiang-2005-hierarchical-phrase-based-model-statistical/). In Proc. ACL, pages 263–270.
4. Jonathan G. Fiscus. 1997. A post-processing system to yield reduced word error rates: Recognizer output voting error reduction (ROVER). In Proc. ASRU, pages 347–354. [find](https://scholar.google.com/scholar?q=Jonathan+G%2E+Fiscus%2E+1997%2E+A+post%2Dprocessing+system+to+yield+reduced+word+error+rates%3A+Recognizer+output+voting+error+reduction+%28ROVER%29%2E+In+Proc%2E+ASRU%2C+pages+347%E2%80%93354%2E)
5. Robert Frederking and Sergei Nirenburg. 1994. Three heads are better than one. In Proc. ANLP, pages 95– 100. [find](https://scholar.google.com/scholar?q=Robert+Frederking+and+Sergei+Nirenburg%2E+1994%2E+Three+heads+are+better+than+one%2E+In+Proc%2E+ANLP%2C+pages+95%E2%80%93+100%2E)
6. Michel Galley, Jonathan Graehl, Kevin Knight, Daniel Marcu, Steve DeNeefe, Wei Wang, and Ignacio Thayer. 2006. Scalable inferences and training of context-rich syntax translation models. In Proc. COL-ING/ACL, pages 961–968. [find](https://scholar.google.com/scholar?q=Michel+Galley%2C+Jonathan+Graehl%2C+Kevin+Knight%2C+Daniel+Marcu%2C+Steve+DeNeefe%2C+Wei+Wang%2C+and+Ignacio+Thayer%2E+2006%2E+Scalable+inferences+and+training+of+context%2Drich+syntax+translation+models%2E+In+Proc%2E+COL%2D)
7. Shyamsundar Jayaraman and Alon Lavie. 2005. Multi-engine machine translation guided by explicit word matching. In Proc. EAMT. [find](https://scholar.google.com/scholar?q=Shyamsundar+Jayaraman+and+Alon+Lavie%2E+2005%2E+Multi%2Dengine+machine+translation+guided+by+explicit+word+matching%2E+In+Proc%2E+EAMT%2E)
8. Philipp Koehn. 2004. Pharaoh: a beam search decoder for phrase-based statistical machine translation models. In Proc. AMTA. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2E+2004%2E+Pharaoh%3A+a+beam+search+decoder+for+phrase%2Dbased+statistical+machine+translation+models%2E+In+Proc%2E+AMTA%2E)
9. Lidia Mangu, Eric Brill, and Andreas Stolcke. 2000. Finding consensus in speech recognition: Word error minimization and other applications of confusion networks. Computer Speech and Language, 14(4):373– 400. [find](https://scholar.google.com/scholar?q=Lidia+Mangu%2C+Eric+Brill%2C+and+Andreas+Stolcke%2E+2000%2E+Finding+consensus+in+speech+recognition%3A+Word+error+minimization+and+other+applications+of+confusion+networks%2E+Computer+Speech+and+Language%2C+14%284%29%3A3)
10. Evgeny Matusov, Nicola Ueffing, and Hermann Ney. 2006. Computing consensus translation from multiple machine translation systems using enhanced hypotheses alignment. In Proc. EACL, pages 33–40. [find](https://scholar.google.com/scholar?q=Evgeny+Matusov%2C+Nicola+Ueffing%2C+and+Hermann+Ney%2E+2006%2E+Computing+consensus+translation+from+multiple+machine+translation+systems+using+enhanced+hypotheses+alignment%2E+In+Proc%2E+EACL%2C+pages+33%E2%80%9340%2E)
11. Franz J. Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. [find](https://scholar.google.com/scholar?q=Franz+J%2E+Och+and+Hermann+Ney%2E+2003%2E+A+systematic+comparison+of+various+statistical+alignment+models%2E+Computational+Linguistics%2C+29%281%29%3A19%E2%80%9351%2E)
12. Franz J. Och. 2003. Minimum error rate training in statistical machine translation. In Proc. ACL, pages 160– 167. [find](https://scholar.google.com/scholar?q=Franz+J%2E+Och%2E+2003%2E+Minimum+error+rate+training+in+statistical+machine+translation%2E+In+Proc%2E+ACL%2C+pages+160%E2%80%93+167%2E)
13. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [BLEU: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proc. ACL, pages 311–318.
14. Khe Chai Sim, William J. Byrne, Mark J.F. Gales, Hichem Sahbi, and Phil C. Woodland. 2007. Consensus network decoding for statistical machine translation system combination. In Proc. ICASSP. [find](https://scholar.google.com/scholar?q=Khe+Chai+Sim%2C+William+J%2E+Byrne%2C+Mark+J%2EF%2E+Gales%2C+Hichem+Sahbi%2C+and+Phil+C%2E+Woodland%2E+2007%2E+Consensus+network+decoding+for+statistical+machine+translation+system+combination%2E+In+Proc%2E+ICASSP%2E)
15. Manhung Siu and Herbert Gish. 1999. Evaluation of word confidence for speech recognition systems. Computer Speech and Language, 13(4):299–319. [find](https://scholar.google.com/scholar?q=Manhung+Siu+and+Herbert+Gish%2E+1999%2E+Evaluation+of+word+confidence+for+speech+recognition+systems%2E+Computer+Speech+and+Language%2C+13%284%29%3A299%E2%80%93319%2E)
16. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciula, and John Makhoul. 2006. [A study of translation edit rate with targeted human annotation](https://md.research.elamur.ai/free/en/papers/snover-2006-study-translation-edit-rate-targeted/). In Proc. AMTA.

*Updated 24 October 2026*