--- title: "Automatic Evaluation of Machine Translation — Lin 2004" description: "BLEU and NIST, the standard automatic MT metrics, score translations by fixed-length consecutive n-gram overlap." authors: ["Chin-Yew Lin", "Franz Josef Och"] year: 2004 venue: "Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04)" source: "https://aclanthology.org/P04-1077/" doi: "10.3115/1218955.1219032" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/lin-2004-automatic-evaluation-machine-translation-quality/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lin-2004-automatic-evaluation-machine-translation-quality/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2004](https://md.research.elamur.ai/free/en/papers/2004/) / Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics # Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics **Authors:** [Chin-Yew Lin](https://md.research.elamur.ai/free/en/authors/chin-yew-lin/), [Franz Josef Och](https://md.research.elamur.ai/free/en/authors/franz-josef-och/) · **Year:** 2004 · **Venue:** Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/P04-1077/) · [PDF](https://aclanthology.org/P04-1077.pdf) · [DOI 10.3115/1218955.1219032](https://doi.org/10.3115/1218955.1219032) · [OpenAlex](https://openalex.org/W2108325777) **Citations:** 726 citations (openalex-snapshot · 2026-10-14) · [211](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary BLEU and NIST, the standard automatic MT metrics, score translations by fixed-length consecutive n-gram overlap. This has drawbacks. It ignores sentence-level word order beyond local windows and needs a preset maximum n. It uses a brevity penalty in place of genuine recall, and it often gives individual sentences a score of zero. Lin and Och propose order-aware alternatives that compare a candidate translation with one or more references: - **ROUGE-L** builds an F-measure from the longest common subsequence. It rewards words that appear in the same order even when gaps separate them. - **ROUGE-W** weights that subsequence so that consecutive runs count more. - **ROUGE-S** counts shared skip-bigrams, meaning ordered word pairs at any distance or within a limited one. When several references are available, the best-matching one is used. The authors compared these metrics with BLEU, NIST, GTM, PER and WER by correlating them with human adequacy and fluency judgments. Bootstrap confidence intervals were computed over 919 sentence segments. ROUGE-S* gave the strongest adequacy correlation. ROUGE-L and several ROUGE-S variants matched BLEU on fluency and clearly beat it on adequacy. They also behave sensibly on single sentences, which helps error diagnosis. ## Key points - ROUGE-S* (no skip-distance limit) reached a Pearson correlation of 0.95 with adequacy on lowercased, stemmed text, up from 0.85 when case was kept. WER was the weakest metric at 0.48. - BLEU12 had the highest Pearson correlation with fluency (0.93). However, it was statistically indistinguishable from most other metrics, except WER and GTM10. - Stemming raised correlation with adequacy, while preserving case raised correlation with fluency. For example, ROUGE-S* scored 0.84 on fluency with case kept and 0.78 with stemming. - By Spearman's coefficient, ROUGE-L was the best metric for both adequacy and fluency. In a toy example where BLEU-2 ties two candidates, ROUGE-L separates them (0.75 vs 0.5). - Planned extensions include soft matching of synonyms and paraphrases, for example via WordNet. The same metrics were also applied to summarization evaluation (Lin 2004). *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) ## Cited by Cited by 211 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 202 not yet. - [Findings of the 2009 Workshop on Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/burch-2009-findings-2009-workshop-statistical-machine/) - [A Systematic Comparison of Smoothing Techniques for Sentence-Level BLEU](https://md.research.elamur.ai/free/en/papers/chen-2014-systematic-comparison-smoothing-techniques-sentence/) - [GLUCOSE: GeneraLized and COntextualized Story Explanations](https://md.research.elamur.ai/free/en/papers/commonsense-2020-mostafazadeh-glucose-generalized-contextualized-story-explanations/) - [Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments](https://md.research.elamur.ai/free/en/papers/fomicheva-2019-taking-mt-evaluation-metrics-extremes/) - [ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation](https://md.research.elamur.ai/free/en/papers/lin-2004-orange-method-evaluating-automatic-evaluation/) - [Coreference-Aware Dialogue Summarization](https://md.research.elamur.ai/free/en/papers/liu-2021-coreference-aware-dialogue-summarization/) - [Paraphrasing Revisited with Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/mallinson-2017-paraphrasing-revisited-neural-machine-translation/) - [Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/) - [Recursively Summarizing Books with Human Feedback](https://md.research.elamur.ai/free/en/papers/wu-2021-recursively-summarizing-books-human-feedback/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{lin2004automatic, title = {{Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics}}, author = {Lin, Chin-Yew and Och, Franz Josef}, booktitle = {Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04)}, year = {2004}, doi = {10.3115/1218955.1219032}, url = {https://doi.org/10.3115/1218955.1219032} } ``` - [← previous Applying Conditional Random Fields to Japanese Morphological Analysis](https://md.research.elamur.ai/free/en/papers/kudo-2004-applying-conditional-random-fields-japanese/) - [next → Automatic Identification of Infrequent Word Senses](https://md.research.elamur.ai/free/en/papers/aclbulk-2004-mccarthy-automatic-identification-infrequent-word-senses/) - [↑ back to papers of 2004](https://md.research.elamur.ai/free/en/papers/2004/) ## References 19 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Akiba, Y., K. Imamura, and E. Sumita. 2001. Using Multiple Edit Distances to Automatically Rank Machine Translation Output. In Proceedings of the MT Summit VIII, Santiago de Compostela, Spain. [find](https://scholar.google.com/scholar?q=Akiba%2C+Y%2E%2C+K%2E+Imamura%2C+and+E%2E+Sumita%2E+2001%2E+Using+Multiple+Edit+Distances+to+Automatically+Rank+Machine+Translation+Output%2E+In+Proceedings+of+the+MT+Summit+VIII%2C+Santiago+de+Compostela%2C+Spain%2E) 2. Barzilay, R. and L. Lee. 2003. Learning to Paraphrase: An Unsupervised Approach Using Multiple-Sequence Alignmen. In Proceeding of NAACL-HLT 2003, Edmonton, Canada. [find](https://scholar.google.com/scholar?q=Barzilay%2C+R%2E+and+L%2E+Lee%2E+2003%2E+Learning+to+Paraphrase%3A+An+Unsupervised+Approach+Using+Multiple%2DSequence+Alignmen%2E+In+Proceeding+of+NAACL%2DHLT+2003%2C+Edmonton%2C+Canada%2E) 3. Leusch, G., N. Ueffing, and H. Ney. 2003. A Novel String-to-String Distance Measure with Applications to Machine Translation Evaluation. In Proceedings of MT Summit IX, New Orleans, U.S.A. [find](https://scholar.google.com/scholar?q=Leusch%2C+G%2E%2C+N%2E+Ueffing%2C+and+H%2E+Ney%2E+2003%2E+A+Novel+String%2Dto%2DString+Distance+Measure+with+Applications+to+Machine+Translation+Evaluation%2E+In+Proceedings+of+MT+Summit+IX%2C+New+Orleans%2C+U%2ES%2EA%2E) 4. Levenshtein, V. I. 1966. Binary codes capable of correcting deletions, insertions and reversals. Soviet Physics Doklady. [find](https://scholar.google.com/scholar?q=Levenshtein%2C+V%2E+I%2E+1966%2E+Binary+codes+capable+of+correcting+deletions%2C+insertions+and+reversals%2E+Soviet+Physics+Doklady%2E) 5. Lin, C.Y. 2004. R OUGE : A Package for Automatic Evaluation of Summaries. In Proceedings of the Workshop on Text Summarization Branches Out, post-conference workshop of ACL 2004, Barcelona, Spain. [find](https://scholar.google.com/scholar?q=Lin%2C+C%2EY%2E+2004%2E+R+OUGE+%3A+A+Package+for+Automatic+Evaluation+of+Summaries%2E+In+Proceedings+of+the+Workshop+on+Text+Summarization+Branches+Out%2C+post%2Dconference+workshop+of+ACL+2004%2C+Barcelona%2C+Spain%2E) 6. Lin, C.-Y. and F. J. Och. 2004. O RANGE : a Method for Evaluating Automatic Evaluation Metrics for Machine Translation. In Proceedings of 20 th International Conference on Computational Linguistic (COLING 2004), Geneva, Switzerland. [find](https://scholar.google.com/scholar?q=Lin%2C+C%2E%2DY%2E+and+F%2E+J%2E+Och%2E+2004%2E+O+RANGE+%3A+a+Method+for+Evaluating+Automatic+Evaluation+Metrics+for+Machine+Translation%2E+In+Proceedings+of+20+th+International+Conference+on+Computational+Linguistic+%28CO) 7. Miller, G. 1990. WordNet: An Online Lexical Database. International Journal of Lexicography, 3(4). [find](https://scholar.google.com/scholar?q=Miller%2C+G%2E+1990%2E+WordNet%3A+An+Online+Lexical+Database%2E+International+Journal+of+Lexicography%2C+3%284%29%2E) 8. Melamed, I.D. 1995. Automatic Evaluation and Uniform Filter Cascades for Inducing N-best Translation Lexicons. In Proceedings of the 3 rd Workshop on Very Large Corpora (WVLC3). Boston, U.S.A. [↗](https://doi.org/10.48550/arxiv.cmp-lg/9505044) 9. Melamed, I.D., R. Green and J. P. Turian. 2003. Precision and Recall of Machine Translation. In Proceedings of NAACL/HLT 2003, Edmonton, Canada. [find](https://scholar.google.com/scholar?q=Melamed%2C+I%2ED%2E%2C+R%2E+Green+and+J%2E+P%2E+Turian%2E+2003%2E+Precision+and+Recall+of+Machine+Translation%2E+In+Proceedings+of+NAACL%2FHLT+2003%2C+Edmonton%2C+Canada%2E) 10. Nießen S., F.J. Och, G, Leusch, H. Ney. 2000. An Evaluation Tool for Machine Translation: Fast Evaluation for MT Research. In Proceedings of the 2nd International Conference on Language Resources and Evaluation, Athens, Greece. [find](https://scholar.google.com/scholar?q=Nie%C3%9Fen+S%2E%2C+F%2EJ%2E+Och%2C+G%2C+Leusch%2C+H%2E+Ney%2E+2000%2E+An+Evaluation+Tool+for+Machine+Translation%3A+Fast+Evaluation+for+MT+Research%2E+In+Proceedings+of+the+2nd+International+Conference+on+Language+Resources+and+) [our card](https://md.research.elamur.ai/free/en/papers/en-2000-evaluation-tool-machine-translation-fast/) 11. NIST. 2002. Automatic Evaluation of Machine Translation Quality using N-gram Co-Occurrence Statistics. AAAAAAAAAAA [http://www.nist.gov/speech/tests/mt/doc/ngram-study.pdf](http://www.nist.gov/speech/tests/mt/doc/ngram-study.pdf) [↗](http://www.nist.gov/speech/tests/mt/doc/ngram-study.pdf) 12. Pantel, P. and Lin, D. 2002. Discovering Word Senses from Text. In Proceedings of SIGKDD-02. Edmonton, Canada. [find](https://scholar.google.com/scholar?q=Pantel%2C+P%2E+and+Lin%2C+D%2E+2002%2E+Discovering+Word+Senses+from+Text%2E+In+Proceedings+of+SIGKDD%2D02%2E+Edmonton%2C+Canada%2E) 13. Papineni, K., S. Roukos, T. Ward, and W.-J. Zhu. 2001. B LEU : a Method for Automatic Evaluation of Machine Translation. IBM Research Report RC22176 (W0109-022). [find](https://scholar.google.com/scholar?q=Papineni%2C+K%2E%2C+S%2E+Roukos%2C+T%2E+Ward%2C+and+W%2E%2DJ%2E+Zhu%2E+2001%2E+B+LEU+%3A+a+Method+for+Automatic+Evaluation+of+Machine+Translation%2E+IBM+Research+Report+RC22176+%28W0109%2D022%29%2E) 14. Porter, M.F. 1980. An Algorithm for Suffix Stripping. Program, 14, pp. 130-137. [↗](https://doi.org/10.1108/eb046814) 15. Saggion H., D. Radev, S. Teufel, and W. Lam. 2002. Meta-Evaluation of Summaries in a Cross-Lingual Environment Using Content-Based Metrics. In Proceedings of COLING-2002, Taipei, Taiwan. [find](https://scholar.google.com/scholar?q=Saggion+H%2E%2C+D%2E+Radev%2C+S%2E+Teufel%2C+and+W%2E+Lam%2E+2002%2E+Meta%2DEvaluation+of+Summaries+in+a+Cross%2DLingual+Environment+Using+Content%2DBased+Metrics%2E+In+Proceedings+of+COLING%2D2002%2C+Taipei%2C+Taiwan%2E) 16. Su, K.-Y., M.-W. Wu, and J.-S. Chang. 1992. A New Quantitative Quality Measure for Machine Translation System. In Proceedings of COLING-92, Nantes, France. [find](https://scholar.google.com/scholar?q=Su%2C+K%2E%2DY%2E%2C+M%2E%2DW%2E+Wu%2C+and+J%2E%2DS%2E+Chang%2E+1992%2E+A+New+Quantitative+Quality+Measure+for+Machine+Translation+System%2E+In+Proceedings+of+COLING%2D92%2C+Nantes%2C+France%2E) 17. Thompson, H. S. 1991. Automatic Evaluation of Translation Quality: Outline of Methodology and Report on Pilot Experiment. In Proceedings of the Evaluator’s Forum, ISSCO, Geneva, Switzerland. [find](https://scholar.google.com/scholar?q=Thompson%2C+H%2E+S%2E+1991%2E+Automatic+Evaluation+of+Translation+Quality%3A+Outline+of+Methodology+and+Report+on+Pilot+Experiment%2E+In+Proceedings+of+the+Evaluator%E2%80%99s+Forum%2C+ISSCO%2C+Geneva%2C+Switzerland%2E) 18. Turian, J. P., L. Shen, and I. D. Melamed. 2003. Evaluation of Machine Translation and its Evaluation. In Proceedings of MT Summit IX, New Orleans, U.S.A. [find](https://scholar.google.com/scholar?q=Turian%2C+J%2E+P%2E%2C+L%2E+Shen%2C+and+I%2E+D%2E+Melamed%2E+2003%2E+Evaluation+of+Machine+Translation+and+its+Evaluation%2E+In+Proceedings+of+MT+Summit+IX%2C+New+Orleans%2C+U%2ES%2EA%2E) [our card](https://md.research.elamur.ai/free/en/papers/turian-2003-evaluation-machine-translation-its-evaluation/) 19. Van Rijsbergen, C.J. 1979. Information Retrieval. Butterworths. London. [find](https://scholar.google.com/scholar?q=Van+Rijsbergen%2C+C%2EJ%2E+1979%2E+Information+Retrieval%2E+Butterworths%2E+London%2E)