---
title: "deltaBLEU — paper by Galley et al. (2015) — summary and ideas"
description: "There, natural data offer only one reply per message, and judging relevance is subjective. 19 references, cited by 52 papers here."
authors: ["Michel Galley", "Chris Brockett", "Alessandro Sordoni", "Yangfeng Ji", "Michael Auli", "Chris Quirk", "Margaret Mitchell", "Jianfeng Gao", "Bill Dolan"]
year: 2015
venue: "arXiv:1506.06863"
source: "https://arxiv.org/abs/1506.06863"
doi: "10.48550/arxiv.1506.06863"
arxiv: "1506.06863"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/galley-2015-deltableu-discriminative-metric-generation-tasks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/galley-2015-deltableu-discriminative-metric-generation-tasks/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/) / deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets

# deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets

**Authors:** [Michel Galley](https://md.research.elamur.ai/free/en/authors/michel-galley/), [Chris Brockett](https://md.research.elamur.ai/free/en/authors/chris-brockett/), [Alessandro Sordoni](https://md.research.elamur.ai/free/en/authors/alessandro-sordoni/), [Yangfeng Ji](https://md.research.elamur.ai/free/en/authors/yangfeng-ji/), [Michael Auli](https://md.research.elamur.ai/free/en/authors/michael-auli/), [Chris Quirk](https://md.research.elamur.ai/free/en/authors/chris-quirk/), [Margaret Mitchell](https://md.research.elamur.ai/free/en/authors/margaret-mitchell/), [Jianfeng Gao](https://md.research.elamur.ai/free/en/authors/jianfeng-gao/) and [Bill Dolan](https://md.research.elamur.ai/free/en/authors/bill-dolan/) · **Year:** 2015 · **Venue:** arXiv:1506.06863 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1506.06863) · [PDF](https://arxiv.org/pdf/1506.06863) · [DOI 10.48550/arxiv.1506.06863](https://doi.org/10.48550/arxiv.1506.06863) · [arXiv 1506.06863](https://arxiv.org/abs/1506.06863) · [OpenAlex](https://openalex.org/W2951813108)

**Citations:** 83 citations (OpenAlex, 2026-10-14) · [52](https://md.research.elamur.ai/free/en/papers/galley-2015-deltableu-discriminative-metric-generation-tasks/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Automatic metrics like BLEU work reasonably well for machine translation. They struggle when many different outputs are acceptable, as in conversational response generation. There, natural data offer only one reply per message, and judging relevance is subjective.

Galley et al. propose ∆BLEU (Discriminative BLEU), a corpus-level multi-reference BLEU in which each reference carries a human-assigned quality weight in [−1, +1]. An n-gram match earns credit according to the highest-weighted reference that contains it, so overlap with poor references lowers the score. With all weights set to 1, the metric becomes standard BLEU, and its maximum stays at 1.

To build test data, the authors used BM25 retrieval to mine extra candidate replies from a large Twitter corpus. Crowd workers rated these replies, and all of them were kept with rescaled weights. Compared against human judgments of system pairs, ∆BLEU tracked human preferences clearly better than IBM BLEU and sentence-level BLEU. The approach moves human effort into a one-time rating of references, which can then be reused throughout development.

## Key points
- Data source: 29M Twitter context–message–response triples. About 33K were crowd-rated, and the 4232 triples averaging ≥4 were split into seed dev (2118) and test (2114) sets.
- For each seed pair, the top 15 BM25-retrieved responses were added. Each was rated by 5 annotators on a 5-point scale, and scores were mapped to [−1, +1] without any cutoff.
- The human evaluation covered outputs from 7 systems (12 system pairs), yielding about 126K pairwise ratings. All metrics used BLEU-2.
- With all references, ∆BLEU reached Spearman's ρ = .484 and Kendall's τ = .342. The best IBM BLEU scores were .343 and .232 (w ≥ 0.6), and sentence-level BLEU peaked at .330 and .222.
- IBM BLEU got worse as low-rated references were added, while ∆BLEU improved, even slightly benefiting from negatively weighted references. The code, multi-reference dev/test sets, and rated outputs were released by Microsoft Research.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 378 works · 161,630 citations
- [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 214 works · 241,492 citations
- [Université de Montréal](https://md.research.elamur.ai/free/en/universities/universite-de-montreal/) — University, Montreal, Canada · 87 works · 91,597 citations
- [Georgia Institute of Technology](https://md.research.elamur.ai/free/en/universities/georgia-institute-of-technology/) — University, Atlanta, United States · 152 works · 27,763 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Margaret Mitchell, Jianfeng Gao and Bill Dolan. 2015. deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets. arXiv:1506.06863.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{galley2015deltableu,
  title = {{deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets}},
  author = {Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Margaret Mitchell, Jianfeng Gao and Bill Dolan},
  year = {2015},
  doi = {10.48550/arxiv.1506.06863},
  eprint = {1506.06863},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1506.06863}
}
```

- [← previous Generating High Quality Proposition Banks for Multilingual Semantic Role Labeling](https://md.research.elamur.ai/free/en/papers/akbik-2015-generating-high-quality-proposition-banks/)
- [next → Machine Translation Experiments on PADIC: A Parallel Arabic DIalect Corpus](https://md.research.elamur.ai/free/en/papers/meftouh-2015-machine-translation-experiments-padic-parallel/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/)

## References

19 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Satanjeev Banerjee and Alon Lavie. 2005. M ETEOR : An automatic metric for MT evaluation with improved correlation with human judgments. In Proc. of ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72. [find](https://scholar.google.com/scholar?q=Satanjeev+Banerjee+and+Alon+Lavie%2E+2005%2E+M+ETEOR+%3A+An+automatic+metric+for+MT+evaluation+with+improved+correlation+with+human+judgments%2E+In+Proc%2E+of+ACL+Workshop+on+Intrinsic+and+Extrinsic+Evaluation+)
2. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the role of B LEU in machine translation research. In EACL, pages 249–256. [find](https://scholar.google.com/scholar?q=Chris+Callison%2DBurch%2C+Miles+Osborne%2C+and+Philipp+Koehn%2E+2006%2E+Re%2Devaluating+the+role+of+B+LEU+in+machine+translation+research%2E+In+EACL%2C+pages+249%E2%80%93256%2E)
3. Deborah Coughlin. 2003. Correlating automated and human assessments of machine translation quality. In Proc. of MT Summit IX, pages 63– 70. [find](https://scholar.google.com/scholar?q=Deborah+Coughlin%2E+2003%2E+Correlating+automated+and+human+assessments+of+machine+translation+quality%2E+In+Proc%2E+of+MT+Summit+IX%2C+pages+63%E2%80%93+70%2E)
4. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proc. of HLT, pages 138–145. [find](https://scholar.google.com/scholar?q=George+Doddington%2E+2002%2E+Automatic+evaluation+of+machine+translation+quality+using+n%2Dgram+co%2Doccurrence+statistics%2E+In+Proc%2E+of+HLT%2C+pages+138%E2%80%93145%2E)
5. Markus Dreyer and Daniel Marcu. 2012. HyTER: Meaning-equivalent semantics for translation evaluation. In Proc. of HLT-NAACL, pages 162–171. [find](https://scholar.google.com/scholar?q=Markus+Dreyer+and+Daniel+Marcu%2E+2012%2E+HyTER%3A+Meaning%2Dequivalent+semantics+for+translation+evaluation%2E+In+Proc%2E+of+HLT%2DNAACL%2C+pages+162%E2%80%93171%2E)
6. Yvette Graham and Timothy Baldwin. 2014. [Testing for significance of increased correlation with human judgment](https://md.research.elamur.ai/free/en/papers/graham-2014-testing-significance-increased-correlation-human/). In Proc. of EMNLP, pages 172–176.
7. Yvette Graham, Timothy Baldwin, and Nitika Mathur. 2015. [Accurate evaluation of segment-level machine translation metrics](https://md.research.elamur.ai/free/en/papers/graham-2015-accurate-evaluation-segment-level-machine/). In Proc. of NAACL-HLT, pages 1183–1191.
8. Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing image description as a ranking task: Data, models and evaluation metrics. J. Artif. Int. Res., 47(1):853–899. [find](https://scholar.google.com/scholar?q=Micah+Hodosh%2C+Peter+Young%2C+and+Julia+Hockenmaier%2E+2013%2E+Framing+image+description+as+a+ranking+task%3A+Data%2C+models+and+evaluation+metrics%2E+J%2E+Artif%2E+Int%2E+Res%2E%2C+47%281%29%3A853%E2%80%93899%2E)
9. Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proc. of the 22nd ACM International Conference on Information & Knowledge Management, pages 2333–2338. [find](https://scholar.google.com/scholar?q=Po%2DSen+Huang%2C+Xiaodong+He%2C+Jianfeng+Gao%2C+Li+Deng%2C+Alex+Acero%2C+and+Larry+Heck%2E+2013%2E+Learning+deep+structured+semantic+models+for+web+search+using+clickthrough+data%2E+In+Proc%2E+of+the+22nd+ACM+Internatio)
10. Alon Lavie and Abhaya Agarwal. 2007. M ETEOR : An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proc. of the Workshop on Statistical Machine Translation (StatMT), pages 228–231. [find](https://scholar.google.com/scholar?q=Alon+Lavie+and+Abhaya+Agarwal%2E+2007%2E+M+ETEOR+%3A+An+automatic+metric+for+mt+evaluation+with+high+levels+of+correlation+with+human+judgments%2E+In+Proc%2E+of+the+Workshop+on+Statistical+Machine+Translation+%28)
11. Preslav Nakov, Francisco Guzman, and Stephan Vogel. 2012. Optimizing for Sentence-Level BLEU+1 Yields Short Translations. In Proc. of COLING. [find](https://scholar.google.com/scholar?q=Preslav+Nakov%2C+Francisco+Guzman%2C+and+Stephan+Vogel%2E+2012%2E+Optimizing+for+Sentence%2DLevel+BLEU%2B1+Yields+Short+Translations%2E+In+Proc%2E+of+COLING%2E)
12. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proc. of ACL, pages 160–167. [find](https://scholar.google.com/scholar?q=Franz+Josef+Och%2E+2003%2E+Minimum+error+rate+training+in+statistical+machine+translation%2E+In+Proc%2E+of+ACL%2C+pages+160%E2%80%93167%2E)
13. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. B LEU : a method for automatic evaluation of machine translation. In Proc. of ACL, pages 311–318. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+B+LEU+%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proc%2E+of+ACL%2C+pages+311%E2%80%93318%2E)
14. M. Przybocki, K. Peterson, and S. Bronsart. 2008. Official results of the NIST 2008 ”Metrics for MAchine TRanslation” challenge (MetricsMATR08). [http://nist.gov/speech/tests/metricsmatr/2008/results/](http://nist.gov/speech/tests/metricsmatr/2008/results/). [↗](http://nist.gov/speech/tests/metricsmatr/2008/results/)
15. Alan Ritter, Colin Cherry, and William B. Dolan. 2011. [Data-driven response generation in social media](https://md.research.elamur.ai/free/en/papers/ritter-2011-data-driven-response-generation-social/). In Proc. of EMNLP, pages 583–593.
16. Stephen E Robertson, Steve Walker, Susan Jones, et al. 1995. [Okapi at TREC-3](https://md.research.elamur.ai/free/en/papers/robertson-1994-okapi-trec-3/). In TREC.
17. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Meg Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. [A neural network approach to context-sensitive generation of conversational responses](https://md.research.elamur.ai/free/en/papers/sordoni-2015-neural-network-approach-context-sensitive/). In Proc. of NAACL-HLT.
18. Hong Sun and Ming Zhou. 2012. Joint learning of a dual SMT system for paraphrase generation. In ACL, pages 38–42. [find](https://scholar.google.com/scholar?q=Hong+Sun+and+Ming+Zhou%2E+2012%2E+Joint+learning+of+a+dual+SMT+system+for+paraphrase+generation%2E+In+ACL%2C+pages+38%E2%80%9342%2E)
19. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. [CIDEr: Consensus-based image description evaluation](https://md.research.elamur.ai/free/en/papers/vedantam-2014-cider-consensus-based-image-description/). In CVPR.

*Updated 2 November 2026*