---
title: "Edinburgh System Description for the 2005 IWSLT — Koehn 2005"
description: "The Edinburgh SMT group, whose earlier work targeted large open-domain text tasks such as Europarl and the DARPA Chinese/Arabic news challenges, entered…"
authors: ["Philipp Koehn", "Amittai Axelrod", "Alexandra Birch Mayne", "Chris Callison-Burch", "Miles Osborne", "David Talbot"]
year: 2005
venue: "Proceedings of the Second International Workshop on Spoken Language Translation"
source: "https://aclanthology.org/2005.iwslt-1.8/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/koehn-2005-edinburgh-system-description-2005-iwslt/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/koehn-2005-edinburgh-system-description-2005-iwslt/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2005](https://md.research.elamur.ai/free/papers/2005/) / Edinburgh System Description for the 2005 IWSLT Speech Translation Evaluation

# Edinburgh System Description for the 2005 IWSLT Speech Translation Evaluation

**Authors:** [Philipp Koehn](https://md.research.elamur.ai/free/authors/philipp-koehn/), [Amittai Axelrod](https://md.research.elamur.ai/free/authors/amittai-axelrod/), Alexandra Birch Mayne, [Chris Callison-Burch](https://md.research.elamur.ai/free/authors/chris-callison-burch/), Miles Osborne, [David Talbot](https://md.research.elamur.ai/free/authors/david-talbot/) · **Year:** 2005 · **Venue:** Proceedings of the Second International Workshop on Spoken Language Translation · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/2005.iwslt-1.8/) · [PDF](https://aclanthology.org/2005.iwslt-1.8.pdf) · [OpenAlex](https://openalex.org/W2401082558)

**Citations:** 359 citations (openalex-snapshot · 2026-10-14) · [186](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
The Edinburgh SMT group, whose earlier work targeted large open-domain text tasks such as Europarl and the DARPA Chinese/Arabic news challenges, entered the IWSLT 2005 evaluation, a limited-domain speech task. They competed only in the transcription track with supplied corpora, covering five pairs: Arabic, Japanese, Korean and Chinese into English, plus English into Chinese. Their system is a phrase-based log-linear model decoded with Pharaoh. It combines GIZA++ alignments symmetrised by heuristics, phrase and lexical translation scores, a new lexicalised reordering model and minimum error rate training. Because the corpora were small, a full training run took minutes, not days. This allowed many tuning experiments on the alignment heuristic, the reordering model and the reordering distance limit. The submission ranked first by BLEU for Korean–English and English–Chinese and was competitive elsewhere. A later analysis showed that the tuning criterion had made the output too short, which badly hurt NIST scores. The work shows that statistical MT can be moved quickly to new, data-poor language pairs, but settings tuned for large corpora must be revisited.

## Key points
- Pipeline: GIZA++ in both directions, then symmetrisation (intersect, grow, grow-diag, final, final-and). Phrase pairs consistent with the alignment are extracted and scored with relative frequency and lexical weighting. Feature weights are tuned by MERT on BLEU.
- Training on the supplied data took about 15 CPU minutes, against roughly a week for the Arabic–English NIST setup. A cluster of about 50 machines ran the parameter sweeps. The 2003 test set was used for tuning and the 2004 set for development.
- Sparser alignments produce much larger phrase tables. The intersection alignment has 79,200 alignment points and yields about 2.56M distinct phrase pairs, compared with 61,168 for the default grow-diag-final. Intersection gave the best dev BLEU for Japanese–English (45.1) and Chinese–English (34.6).
- Allowing reordering beyond the usual 4-word limit, up to 8 words, tended to help with Asian languages. Dropping unknown words instead of copying them to the output also raised BLEU, for example Arabic–English moved from about 50 to about 56.
- Official ranks by BLEU were 5th/8 for Arabic–English, 4th/7 for Japanese–English, 1st/4 for Korean–English, 3rd/10 for Chinese–English and 1st/2 for English–Chinese. After switching MERT to the average reference length, NIST for Japanese–English rose from 4.0784 to 8.1209.

## Interesting ideas
- **Re-check defaults when data shrinks.** The alignment heuristic that worked best on large corpora was often not the best on small ones. The best choice also differed by language pair. For low-resource Ukrainian pairs, the symmetrisation method is worth treating as a tunable hyperparameter.
- **Lexicalised orientation model.** Each phrase pair records how often it continues the previous phrase, swaps with it, or jumps away. These counts come from simple corner checks in the alignment matrix. This is a cheap way to learn word-specific reordering habits.
- **The tuning metric must match the evaluation metric.** Tuning BLEU against the shortest reference quietly pushed the system toward short outputs. That in turn damaged NIST. When optimising any system, make sure details like the length penalty match the official scorer exactly.
- **Fast training makes broad sweeps possible.** With small data and a modest cluster, they could test many configurations end to end within a month. This favours small, quickly retrainable baselines when exploring a new domain.
- **Small output-side heuristics can matter a lot.** Removing untranslatable unknown words, rather than copying them through, gave large BLEU gains on some pairs. It is a trivial post-processing choice that deserves to be tested explicitly.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/)

Topics: [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/)

## Cited by

Cited by

186 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 181 not yet.

- [Fill-up versus interpolation methods for phrase-based SMT adaptation](https://md.research.elamur.ai/free/papers/bisazza-2011-fill-up-versus-interpolation-methods/)
- [Linguistic realisation as machine translation: Comparing different MT models for AMR-to-text generation](https://md.research.elamur.ai/free/papers/ferreira-2017-linguistic-realisation-as-machine-translation/)
- [Anaphora Models and Reordering for Phrase-Based SMT](https://md.research.elamur.ai/free/papers/hardmeier-2014-anaphora-models-reordering-phrase-based/)
- [Measuring Machine Translation Errors in New Domains](https://md.research.elamur.ai/free/papers/irvine-2013-measuring-machine-translation-errors-new/)
- [Experiments in Domain Adaptation for Statistical Machine Translation](https://md.research.elamur.ai/free/papers/koehn-2007-experiments-domain-adaptation-statistical-machine/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{koehn2005edinburgh,
  title = {{Edinburgh System Description for the 2005 IWSLT Speech Translation Evaluation}},
  author = {Koehn, Philipp and Axelrod, Amittai and Mayne, Alexandra Birch and Callison-Burch, Chris and Osborne, Miles and Talbot, David},
  booktitle = {Proceedings of the Second International Workshop on Spoken Language Translation},
  year = {2005},
  url = {https://aclanthology.org/2005.iwslt-1.8/}
}
```

- [← previous Document Authoring the Bible for Minority Language Translation](https://md.research.elamur.ai/free/papers/beale-2005-document-authoring-bible-minority-language/)
- [next → Evaluating Machine Translation Output with Automatic Sentence Segmentation](https://md.research.elamur.ai/free/papers/matusov-2005-evaluating-machine-translation-output-automatic/)
- [↑ back to papers of 2005](https://md.research.elamur.ai/free/papers/2005/)

## References

12 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Collins, M., Koehn, P., and Kucerova, I. (2005). Clause restructuring for statistical machine translation. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL ’05), pages 531–540, Ann Arbor, Michigan. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Collins%2C+M%2E%2C+Koehn%2C+P%2E%2C+and+Kucerova%2C+I%2E+%282005%29%2E+Clause+restructuring+for+statistical+machine+translation%2E+In+Proceedings+of+the+43rd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28A) [our card](https://md.research.elamur.ai/free/papers/collins-2005-clause-restructuring-statistical-machine-translation/)
2. Koehn, P. (2004a). The foundation for statistical machine translation at MIT. In Proceedings of Machine Translation Evaluation Workshop 2004. [find](https://scholar.google.com/scholar?q=Koehn%2C+P%2E+%282004a%29%2E+The+foundation+for+statistical+machine+translation+at+MIT%2E+In+Proceedings+of+Machine+Translation+Evaluation+Workshop+2004%2E)
3. Koehn, P. (2004b). Pharaoh: a beam search decoder for statistical machine translation. In 6th Conference of the Association for Machine Translation in the Americas, AMTA, Lecture Notes in Computer Science. Springer. [find](https://scholar.google.com/scholar?q=Koehn%2C+P%2E+%282004b%29%2E+Pharaoh%3A+a+beam+search+decoder+for+statistical+machine+translation%2E+In+6th+Conference+of+the+Association+for+Machine+Translation+in+the+Americas%2C+AMTA%2C+Lecture+Notes+in+Computer+Sci)
4. Koehn, P., Axelrod, A., Mayne, A. B., Callison-Burch, C., Osborne, M., Talbot, D., and White, M. (2005). Edinburgh system description for the 2005 NIST MT evaluation. In Proceedings of Machine Translation Evaluation Workshop 2005. [find](https://scholar.google.com/scholar?q=Koehn%2C+P%2E%2C+Axelrod%2C+A%2E%2C+Mayne%2C+A%2E+B%2E%2C+Callison%2DBurch%2C+C%2E%2C+Osborne%2C+M%2E%2C+Talbot%2C+D%2E%2C+and+White%2C+M%2E+%282005%29%2E+Edinburgh+system+description+for+the+2005+NIST+MT+evaluation%2E+In+Proceedings+of+Machine+Transla)
5. Koehn, P., Och, F. J., and Marcu, D. (2003). Statistical phrase based translation. In Proceedings of the Joint Conference on Human Language Technologies and the Annual Meeting of the North American Chapter of the Association of Computational Linguistics (HLT-NAACL). WER 0.3902 0.5488 0.5570 0.4535 0.6197 PER 0.3462 0.4861 0.4797 0.3983 0.5286 METEOR 0.6893 0.5167 0.5585 0.6320 0.0955 GTM 0.6521 0.4748 0.4843 0.5988 0.5584 Rank 5th of 8 4th of 7 1st of 4 3rd of 10 1st of 2 WER 0.3860 0.5489 0.5617 0.5043 0.6037 PER 0.3323 0.4599 0.4559 0.4089 0.5149 METEOR 0.7270 0.5971 0.6221 0.6841 0.0955 GTM 0.6613 0.4890 0.4980 0.5914 0.5657 [↗](https://doi.org/10.21236/ada461156)
6. Lee, A. and Przybocki, M. (2005). NIST 2005 machine translation evaluation official results. Official release of automatic evaluation scores for all submissions. [find](https://scholar.google.com/scholar?q=Lee%2C+A%2E+and+Przybocki%2C+M%2E+%282005%29%2E+NIST+2005+machine+translation+evaluation+official+results%2E+Official+release+of+automatic+evaluation+scores+for+all+submissions%2E)
7. Marcu, D. and Wong, W. (2002). A phrase-based, joint probability model for statistical machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Marcu%2C+D%2E+and+Wong%2C+W%2E+%282002%29%2E+A+phrase%2Dbased%2C+joint+probability+model+for+statistical+machine+translation%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29) [our card](https://md.research.elamur.ai/free/papers/marcu-2002-phrase-based-joint-probability-model/)
8. Mayne, A. B. (2005). Scaling the joint probability phrase based statistical translation model. Master’s thesis, University of Edinburgh. [find](https://scholar.google.com/scholar?q=Mayne%2C+A%2E+B%2E+%282005%29%2E+Scaling+the+joint+probability+phrase+based+statistical+translation+model%2E+Master%E2%80%99s+thesis%2C+University+of+Edinburgh%2E)
9. Och, F. J. (2003). Minimum error rate training for statistical machine translation. In Proceedings of the 41st Annual Meeting of the Association of Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Och%2C+F%2E+J%2E+%282003%29%2E+Minimum+error+rate+training+for+statistical+machine+translation%2E+In+Proceedings+of+the+41st+Annual+Meeting+of+the+Association+of+Computational+Linguistics+%28ACL%29%2E)
10. Och, F. J. and Ney, H. (2003). A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–52. [↗](https://doi.org/10.1162/089120103321337421)
11. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association of Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Papineni%2C+K%2E%2C+Roukos%2C+S%2E%2C+Ward%2C+T%2E%2C+and+Zhu%2C+W%2E%2DJ%2E+%282002%29%2E+BLEU%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+Annual+Meeting+of+the+Association+of+Computational) [our card](https://md.research.elamur.ai/free/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
12. Tillmann, C. (2004). A unigram orientation model for statistical machine translation. In Proceedings of the Joint Conference on Human Language Technologies and the Annual Meeting of the North American Chapter of the Association of Computational Linguistics (HLT-NAACL). [↗](https://doi.org/10.21236/ada460258) [our card](https://md.research.elamur.ai/free/papers/tillmann-2004-unigram-orientation-model-statistical-machine/)