--- title: "Statistical Significance Tests for Machine — Koehn 2004" description: "The paper addresses the question of when a BLEU difference between two machine translation systems on a test set genuinely indicates a difference in…" authors: ["Philipp Koehn"] year: 2004 venue: "Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing" source: "https://aclanthology.org/W04-3250/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/koehn-2004-statistical-significance-tests-machine-translation/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/koehn-2004-statistical-significance-tests-machine-translation/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2004](https://md.research.elamur.ai/free/en/papers/2004/) / Statistical Significance Tests for Machine Translation Evaluation # Statistical Significance Tests for Machine Translation Evaluation **Authors:** [Philipp Koehn](https://md.research.elamur.ai/free/en/authors/philipp-koehn/) · **Year:** 2004 · **Venue:** Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/W04-3250/) · [PDF](https://aclanthology.org/W04-3250.pdf) · [OpenAlex](https://openalex.org/W222053410) **Citations:** 1395 citations (OpenAlex, 2026-10-14) · [1049](https://md.research.elamur.ai/free/en/papers/koehn-2004-statistical-significance-tests-machine-translation/cited-by/) in our library, of the 140884 papers whose bibliographies we read ## Summary The paper addresses the question of when a BLEU difference between two machine translation systems on a test set genuinely indicates a difference in quality rather than sampling chance. There is no convenient analytical way to estimate significance for BLEU. The author therefore proposes bootstrap resampling: sentences are repeatedly drawn at random with replacement from the test set, and conclusions are drawn from the resulting distribution of scores. The first variant of the method yields a confidence interval for a single system. The paired variant (paired bootstrap resampling) shows how often one system outperforms the other. For the experiments, a single phrase-based system was trained on the Europarl corpus for ten language pairs, each translating into English. A shared aligned test set of 30,000 sentences served as the reference for "true" quality. The evaluation showed that the significance estimates are accurate and even somewhat conservative. As few as 300 sentences suffice for reliable conclusions if the BLEU difference is at least 2–3 points. The author also shows that test sentences are better drawn from different parts of the corpus. This work made statistical significance testing standard practice in MT research. ## Key points - **Data:** Europarl covers 11 European languages, with 20–30 million words per language. The reference test set contains 30,000 sentences (about 900 thousand words). On it, BLEU ranges from 28.9% for Spanish to 20.2% for Finnish. - **Choice of test sentences:** if 300 consecutive sentences are taken, the BLEU of the Spanish–English system over 100 blocks varies from 21% to 37%. If instead the sample is spread across the entire corpus (broad sample), the spread narrows to 27–31%. - **Validation of the method:** 9 pairs of systems were compared on test sets ranging from 100 to 3000 sentences. Sets of 100 sentences produce noticeably more false conclusions. - **Accuracy of the estimates:** when the method estimates a significance level of 90–94.9%, the conclusion is in fact correct in 95% of cases. That is, the method tends to err on the side of caution. - **Example:** the Portuguese and Danish systems differ by only 0.5 BLEU. Over 100 samples of 300 sentences each, the paired bootstrap yielded 13 conclusions at 95% significance: 12 correct and 1 false. This is consistent with the expected error rate of 1 in 20. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) · [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/) Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · [Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) ## Institutions - [Massachusetts Institute of Technology](https://md.research.elamur.ai/free/en/universities/massachusetts-institute-of-technology/) — University, Cambridge, United States · 125 works · 60 031 citations ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{koehn2004statistical, title = {{Statistical Significance Tests for Machine Translation Evaluation}}, author = {Koehn, Philipp}, booktitle = {Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing}, year = {2004}, url = {https://aclanthology.org/W04-3250/} } ``` - [← previous RCV1: A New Benchmark Collection for Text Categorization Research](https://md.research.elamur.ai/free/en/papers/lewis-2004-rcv1-benchmark-collection-text-categorization/) - [next → Determining the Sentiment of Opinions](https://md.research.elamur.ai/free/en/papers/kim-2004-determining-sentiment-opinions/) - [↑ back to papers of 2004](https://md.research.elamur.ai/free/en/papers/2004/) ## References 15 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Brown, P., Cocke, J., Pietra, S. A. D., Pietra, V. J. D., Jelinek, F., Lafferty, J. D., Mercer, R. L., and Rossin, P. (1990). A statistical approach to machine translation. Computational Linguistics, 16(2):76–85. [find](https://scholar.google.com/scholar?q=Brown%2C+P%2E%2C+Cocke%2C+J%2E%2C+Pietra%2C+S%2E+A%2E+D%2E%2C+Pietra%2C+V%2E+J%2E+D%2E%2C+Jelinek%2C+F%2E%2C+Lafferty%2C+J%2E+D%2E%2C+Mercer%2C+R%2E+L%2E%2C+and+Rossin%2C+P%2E+%281990%29%2E+A+statistical+approach+to+machine+translation%2E+Computational+Linguistics%2C+) 2. Brown, P. F., Pietra, S. A. D., Pietra, V. J. D., and Mercer, R. L. (1993). The mathematics of statistical machine translation. Computational Linguistics, 19(2):263– 313. [find](https://scholar.google.com/scholar?q=Brown%2C+P%2E+F%2E%2C+Pietra%2C+S%2E+A%2E+D%2E%2C+Pietra%2C+V%2E+J%2E+D%2E%2C+and+Mercer%2C+R%2E+L%2E+%281993%29%2E+The+mathematics+of+statistical+machine+translation%2E+Computational+Linguistics%2C+19%282%29%3A263%E2%80%93+313%2E) 3. Efron, B. and Tibshirani, R. J. (1994). An Introduction to the Bootstrap. CRC Press. [find](https://scholar.google.com/scholar?q=Efron%2C+B%2E+and+Tibshirani%2C+R%2E+J%2E+%281994%29%2E+An+Introduction+to+the+Bootstrap%2E+CRC+Press%2E) 4. Germann, U. (2003). Greedy decoding for statistical machine translation in almost linear time. In Proceedings of HLT-NAACL. [find](https://scholar.google.com/scholar?q=Germann%2C+U%2E+%282003%29%2E+Greedy+decoding+for+statistical+machine+translation+in+almost+linear+time%2E+In+Proceedings+of+HLT%2DNAACL%2E) 5. Koehn, P. (2002). Europarl: A multilingual corpus for evaluation of machine translation. Unpublished, [http://www.isi.edu/](http://www.isi.edu/) koehn/europarl/. Sample Size (Sentences) 100 300 600 3000 33%/1% 60% 84% 100% 31% 65% 96% 100% 7%/2% 12%/1% 10% 30% 38% 68% 92% 100% 24% 48% 74% 100% 85% 100% 100% 100% 65% 97% 100% 100% 97% 100% 100% 100% 53% 91% 100% 100% Koehn, P. (2004). Pharaoh: a beam search decoder for phrase-based statistical machine translation models. In Proceedings of AMTA. [↗](http://www.isi.edu/) 6. Koehn, P., Och, F. J., and Marcu, D. (2003). Statistical phrase based translation. In Proceedings of HLT-NAACL. [find](https://scholar.google.com/scholar?q=Koehn%2C+P%2E%2C+Och%2C+F%2E+J%2E%2C+and+Marcu%2C+D%2E+%282003%29%2E+Statistical+phrase+based+translation%2E+In+Proceedings+of+HLT%2DNAACL%2E) 7. Kumar, S. and Byrne, W. (2004). Minimum bayes-risk decoding for statistical machine translation. In Proceedings of HLT-NAACL. [find](https://scholar.google.com/scholar?q=Kumar%2C+S%2E+and+Byrne%2C+W%2E+%282004%29%2E+Minimum+bayes%2Drisk+decoding+for+statistical+machine+translation%2E+In+Proceedings+of+HLT%2DNAACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/kumar-2004-minimum-bayes-risk-decoding-statistical/) 8. Melamed, I. D., Green, R., and Turian, J. P. (2003). Precision and recall of machine translation. In Proceedings of HLT-NAACL. [find](https://scholar.google.com/scholar?q=Melamed%2C+I%2E+D%2E%2C+Green%2C+R%2E%2C+and+Turian%2C+J%2E+P%2E+%282003%29%2E+Precision+and+recall+of+machine+translation%2E+In+Proceedings+of+HLT%2DNAACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/melamed-2003-precision-recall-machine-translation/) 9. Och, F. J. (2002). Statistical Machine Translation: From Single-Word Models to Alignment Templates. PhD thesis, RWTH Aachen, Germany. [find](https://scholar.google.com/scholar?q=Och%2C+F%2E+J%2E+%282002%29%2E+Statistical+Machine+Translation%3A+From+Single%2DWord+Models+to+Alignment+Templates%2E+PhD+thesis%2C+RWTH+Aachen%2C+Germany%2E) 10. Och, F. J. (2003). Minimum error rate training for statistical machine translation. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Och%2C+F%2E+J%2E+%282003%29%2E+Minimum+error+rate+training+for+statistical+machine+translation%2E+In+Proceedings+of+ACL%2E) 11. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: a method for automatic evaluation of machine translation. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Papineni%2C+K%2E%2C+Roukos%2C+S%2E%2C+Ward%2C+T%2E%2C+and+Zhu%2C+W%2E%2DJ%2E+%282002%29%2E+BLEU%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/) 12. Press, W. H., Teukolsky, S. A., Vetterling, W. T., and Flannery, B. P. (2002). Numerical Recipes in C++. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Press%2C+W%2E+H%2E%2C+Teukolsky%2C+S%2E+A%2E%2C+Vetterling%2C+W%2E+T%2E%2C+and+Flannery%2C+B%2E+P%2E+%282002%29%2E+Numerical+Recipes+in+C%2B%2B%2E+Cambridge+University+Press%2E) 13. Tillmann, C. (2003). A projection extension algorithm for statistical machine translation. In Collins, M. and Steedman, M., editors, Proceedings of EMNLP, pages 1–8. [find](https://scholar.google.com/scholar?q=Tillmann%2C+C%2E+%282003%29%2E+A+projection+extension+algorithm+for+statistical+machine+translation%2E+In+Collins%2C+M%2E+and+Steedman%2C+M%2E%2C+editors%2C+Proceedings+of+EMNLP%2C+pages+1%E2%80%938%2E) 14. Vogel, S., Zhang, Y., Huang, F., Tribble, A., Venugopal, A., Zhao, B., and Waibel, A. (2003). The CMU statistical machine translation system. In Proceedings of MT Summit IX. [find](https://scholar.google.com/scholar?q=Vogel%2C+S%2E%2C+Zhang%2C+Y%2E%2C+Huang%2C+F%2E%2C+Tribble%2C+A%2E%2C+Venugopal%2C+A%2E%2C+Zhao%2C+B%2E%2C+and+Waibel%2C+A%2E+%282003%29%2E+The+CMU+statistical+machine+translation+system%2E+In+Proceedings+of+MT+Summit+IX%2E) 15. Zens, R., Och, F. J., and Ney, H. (2002). Phrase-based statistical machine translation. In Proceedings of the German Conference on Artificial Intelligence (KI 2002). [find](https://scholar.google.com/scholar?q=Zens%2C+R%2E%2C+Och%2C+F%2E+J%2E%2C+and+Ney%2C+H%2E+%282002%29%2E+Phrase%2Dbased+statistical+machine+translation%2E+In+Proceedings+of+the+German+Conference+on+Artificial+Intelligence+%28KI+2002%29%2E)