--- title: "ORANGE: a Method for Evaluating Automatic — Lin 2004" description: "Metrics for automatic machine translation (MT) evaluation are usually validated by correlating their corpus-level scores with human adequacy and fluency…" authors: ["Chin-Yew Lin", "Franz Josef Och"] year: 2004 venue: "COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics" source: "https://aclanthology.org/C04-1072/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/lin-2004-orange-method-evaluating-automatic-evaluation/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/lin-2004-orange-method-evaluating-automatic-evaluation/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2004](https://md.research.elamur.ai/free/papers/2004/) / ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation # ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation **Authors:** [Chin-Yew Lin](https://md.research.elamur.ai/free/authors/chin-yew-lin/), [Franz Josef Och](https://md.research.elamur.ai/free/authors/franz-josef-och/) · **Year:** 2004 · **Venue:** COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/C04-1072/) · [PDF](https://aclanthology.org/C04-1072.pdf) · [OpenAlex](https://openalex.org/W7098597820) **Citations:** 0 citations (openalex-snapshot · 2026-10-14) · [141](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Metrics for automatic machine translation (MT) evaluation are usually validated by correlating their corpus-level scores with human adequacy and fluency ratings. Those ratings are costly and inconsistent, and they give only a few data points (one per system). The authors show that rankings from such correlations shift with the human criterion used and often overlap within confidence intervals. They propose ORANGE (Oracle Ranking for Gisting Evaluation), which needs only reference translations. For each source sentence, the references are mixed into an n-best list from ISI's AlTemp SMT system. Everything is scored with the metric under test, and ORANGE is the references' average rank divided by the list length, so lower is better. The paper also introduces ROUGE-W, a weighted-LCS extension of ROUGE-L, and compares it with ROUGE-L, ROUGE-S, BLEU variants, NIST, WER and PER. On 1024-best lists, ROUGE-S4 scores best, statistically tied with ROUGE-L and ROUGE-W, and PER scores worst. The method yields sentence-level diagnostics and fits naturally into minimum error rate training. ## Key points - Data: 2003 NIST Chinese-English MT evaluation. Correlation analysis has only 8 system-level data points, while ORANGE uses 919 sentence-level ones. - Correlation analysis (Table 1) is inconsistent across criteria: - Short-n-gram BLEU and low-exponent GTM correlate better with adequacy. - Longer BLEU (e.g., BLEU12) and GTM30 correlate better with fluency. - Many metrics are statistically indistinguishable under bootstrap 95% confidence intervals. - ORANGE scores on 1024-best lists (Table 6): - ROUGE-S4: 19.66% (average reference rank about 202) - ROUGE-W-1.1: 20.45% - ROUGE-L: 20.56% - BLEUS6: 22.91% (rank 235) - WER: 23.90% - NIST: 29.70% - PER: 36.84% - On 16384-best lists for 474 sentences (Table 7), the relative ordering holds, with ROUGE-S4 at 14.92% and PER at 28.95%. Plotting N from 1 to 16384 shows stable relative performance across the whole range. - ROUGE-W extends the LCS dynamic program with a table tracking the length of the current run of consecutive matches. A superadditive weighting function, e.g., f(k)=k², gives contiguous matches extra credit. ## Interesting ideas - **Rank-based self-check.** If you trust that human references beat system outputs, you can judge a scoring function by how high it ranks the references among candidates. This avoids collecting new human judgments and could transfer to evaluating metrics for summarization, paraphrase or other generation tasks. - **More data points per decision.** Making each sentence, rather than each system, an observation turns a tiny sample into hundreds of points and gives more meaningful confidence intervals. This is helpful whenever few competing systems exist. - **One objective instead of two.** Replacing separate adequacy and fluency correlations with a single number to minimize makes metric selection and tuning simpler. The same metric can then plug directly into n-best reranking or minimum error rate training. - **Error analysis from rank inversions.** Cases where a machine output outranks its reference form a ready-made confusion set. They show which features a metric is missing, creating a natural metric-development loop. - **Weighted LCS.** Tracking the length of consecutive matches inside the standard LCS table, and rewarding runs with a superadditive function, adds word-adjacency sensitivity at almost no cost. This could suit any sequence-similarity measure where in-order matches matter more than scattered ones. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/) Topics: [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/) ## Cited by Cited by 141 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 133 not yet. - [Unified Pre-training for Program Understanding and Generation](https://md.research.elamur.ai/free/papers/ahmad-2021-unified-pre-training-program-understanding/) - [MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response](https://md.research.elamur.ai/free/papers/deng-2024-musilingo-bridging-music-text-pre/) - [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://md.research.elamur.ai/free/papers/feng-2020-codebert-pre-trained-model-programming/) - [Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments](https://md.research.elamur.ai/free/papers/fomicheva-2019-taking-mt-evaluation-metrics-extremes/) - [Towards Heterogeneous Automatic MT Error Analysis](https://md.research.elamur.ai/free/papers/gimenez-2008-heterogeneous-automatic-mt-error-analysis/) - [Paraphrasing for Automatic Evaluation](https://md.research.elamur.ai/free/papers/kauchak-2006-paraphrasing-automatic-evaluation/) - [Analyzing Uncertainty in Neural Machine Translation](https://md.research.elamur.ai/free/papers/ott-2018-analyzing-uncertainty-neural-machine-translation/) - [Beyond BLEU: Training Neural Machine Translation with Semantic Similarity](https://md.research.elamur.ai/free/papers/wieting-2019-beyond-bleu-training-neural-machine/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{lin2004orange, title = {{ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation}}, author = {Lin, Chin-Yew and Och, Franz Josef}, booktitle = {COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics}, year = {2004}, url = {https://aclanthology.org/C04-1072/} } ``` - [← previous Open Text Semantic Parsing Using FrameNet and WordNet](https://md.research.elamur.ai/free/papers/shi-2004-open-text-semantic-parsing-framenet/) - [next → Parsing Ungrammatical Input: An Evaluation Procedure](https://md.research.elamur.ai/free/papers/lingbulk-2004-foster-parsing-ungrammatical-input-evaluation-procedure/) - [↑ back to papers of 2004](https://md.research.elamur.ai/free/papers/2004/) ## References 11 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Akiba, Y., K. Imamura, and E. Sumita. 2001. Using Multiple Edit Distances to Automatically Rank Machine Translation Output. In Proceedings of the MT Summit VIII, Santiago de Compostela, Spain. [find](https://scholar.google.com/scholar?q=Akiba%2C+Y%2E%2C+K%2E+Imamura%2C+and+E%2E+Sumita%2E+2001%2E+Using+Multiple+Edit+Distances+to+Automatically+Rank+Machine+Translation+Output%2E+In+Proceedings+of+the+MT+Summit+VIII%2C+Santiago+de+Compostela%2C+Spain%2E) 2. Cormen, T. R., C. E. Leiserson, and R. L. Rivest. 1989. Introduction to Algorithms. The MIT Press. [find](https://scholar.google.com/scholar?q=Cormen%2C+T%2E+R%2E%2C+C%2E+E%2E+Leiserson%2C+and+R%2E+L%2E+Rivest%2E+1989%2E+Introduction+to+Algorithms%2E+The+MIT+Press%2E) 3. Davison, A. C. and D. V. Hinkley. 1997. Bootstrap Methods and Their Application. Cambridge University Press. [↗](https://doi.org/10.1017/cbo9780511802843) 4. Leusch, G., N. Ueffing, and H. Ney. 2003. A Novel String-to-String Distance Measure with Applications to Machine Translation Evaluation. In Proceedings of MT Summit IX, New Orleans, U.S.A. ORANGE at Different NBEST Cutoff Length avg 0.4 0.35 0.3 ORANGE 0.25 0.2 0.15 0.1 6000 8000 10000 NBEST Cutoff Length = 1 to 16384 Figure 1. O RANGE scores for 6 metrics vs. length of n-best list from 1 to 16384 with increment of 64. Lin, C-Y. and F.J. [find](https://scholar.google.com/scholar?q=Leusch%2C+G%2E%2C+N%2E+Ueffing%2C+and+H%2E+Ney%2E+2003%2E+A+Novel+String%2Dto%2DString+Distance+Measure+with+Applications+to+Machine+Translation+Evaluation%2E+In+Proceedings+of+MT+Summit+IX%2C+New+Orleans%2C+U%2ES%2EA%2E+ORANGE+at+D) 5. Och. 2004. Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics. Submitted. [find](https://scholar.google.com/scholar?q=Och%2E+2004%2E+Automatic+Evaluation+of+Machine+Translation+Quality+Using+Longest+Common+Subsequence+and+Skip%2DBigram+Statistics%2E+Submitted%2E) [our card](https://md.research.elamur.ai/free/papers/lin-2004-automatic-evaluation-machine-translation-quality/) 6. Nießen S., F.J. Och, G, Leusch, H. Ney. 2000. An Evaluation Tool for Machine Translation: Fast Evaluation for MT Research. In Proceedings of the 2nd International Conference on Language Resources and Evaluation, Athens, Greece. [find](https://scholar.google.com/scholar?q=Nie%C3%9Fen+S%2E%2C+F%2EJ%2E+Och%2C+G%2C+Leusch%2C+H%2E+Ney%2E+2000%2E+An+Evaluation+Tool+for+Machine+Translation%3A+Fast+Evaluation+for+MT+Research%2E+In+Proceedings+of+the+2nd+International+Conference+on+Language+Resources+and+) [our card](https://md.research.elamur.ai/free/papers/en-2000-evaluation-tool-machine-translation-fast/) 7. NIST. 2002. Automatic Evaluation of Machine Translation Quality using N-gram Co-Occurrence Statistics. AAAAAAAAAAA [http://www.nist.gov/speech/tests/mt/doc](http://www.nist.gov/speech/tests/mt/doc) [↗](http://www.nist.gov/speech/tests/mt/doc) 8. Franz Josef Och. 2003. Minimum Error Rate Training for Statistical Machine Translation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (ACL-2003), Sapporo, Japan. [find](https://scholar.google.com/scholar?q=Franz+Josef+Och%2E+2003%2E+Minimum+Error+Rate+Training+for+Statistical+Machine+Translation%2E+In+Proceedings+of+the+41st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL%2D2003%29%2C+Sapporo%2C+) 9. Papineni, K., S. Roukos, T. Ward, and W.-J. Zhu. 2001. Bleu: a Method for Automatic Evaluation of Machine Translation. IBM Research Report RC22176 (W0109-022). [find](https://scholar.google.com/scholar?q=Papineni%2C+K%2E%2C+S%2E+Roukos%2C+T%2E+Ward%2C+and+W%2E%2DJ%2E+Zhu%2E+2001%2E+Bleu%3A+a+Method+for+Automatic+Evaluation+of+Machine+Translation%2E+IBM+Research+Report+RC22176+%28W0109%2D022%29%2E) [our card](https://md.research.elamur.ai/free/papers/papineni-2002-bleu-method-automatic-evaluation-machine/) 10. Su, K.-Y., M.-W. Wu, and J.-S. Chang. 1992. A New Quantitative Quality Measure for Machine Translation System. In Proceedings of COLING-92, Nantes, France. [find](https://scholar.google.com/scholar?q=Su%2C+K%2E%2DY%2E%2C+M%2E%2DW%2E+Wu%2C+and+J%2E%2DS%2E+Chang%2E+1992%2E+A+New+Quantitative+Quality+Measure+for+Machine+Translation+System%2E+In+Proceedings+of+COLING%2D92%2C+Nantes%2C+France%2E) 11. Turian, J. P., L. Shen, and I. D. Melamed. 2003. Evaluation of Machine Translation and its Evaluation. In Proceedings of MT Summit IX, New Orleans, U.S.A. BLEUS4 ROUGE−L PER ROUGE−S4 WER ROUGE−W−1−2 [find](https://scholar.google.com/scholar?q=Turian%2C+J%2E+P%2E%2C+L%2E+Shen%2C+and+I%2E+D%2E+Melamed%2E+2003%2E+Evaluation+of+Machine+Translation+and+its+Evaluation%2E+In+Proceedings+of+MT+Summit+IX%2C+New+Orleans%2C+U%2ES%2EA%2E+BLEUS4+ROUGE%E2%88%92L+PER+ROUGE%E2%88%92S4+WER+ROUGE%E2%88%92W%E2%88%921%E2%88%922) [our card](https://md.research.elamur.ai/free/papers/turian-2003-evaluation-machine-translation-its-evaluation/)