---
title: "Trading Off Diversity and Quality in Natural Language Generation"
description: "Open-ended generation tasks such as dialogue or storytelling need outputs that are both good and… 16 references, cited by 68 papers here."
authors: ["Hugh Zhang", "Daniel Duckworth", "Daphne Ippolito", "Arvind Neelakantan"]
year: 2021
venue: "Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval)"
source: "https://aclanthology.org/2021.humeval-1.3/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2021-trading-off-diversity-quality-natural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2021-trading-off-diversity-quality-natural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/) / Trading Off Diversity and Quality in Natural Language Generation

# Trading Off Diversity and Quality in Natural Language Generation

**Authors:** [Hugh Zhang](https://md.research.elamur.ai/free/en/authors/hugh-zhang/), [Daniel Duckworth](https://md.research.elamur.ai/free/en/authors/daniel-duckworth/), [Daphne Ippolito](https://md.research.elamur.ai/free/en/authors/daphne-ippolito/) and [Arvind Neelakantan](https://md.research.elamur.ai/free/en/authors/arvind-neelakantan/) · **Year:** 2021 · **Venue:** Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.humeval-1.3/) · [PDF](https://aclanthology.org/2021.humeval-1.3.pdf) · [OpenAlex](https://openalex.org/W3020264409)

**Citations:** 17 citations (OpenAlex, 2026-10-14) · [68](https://md.research.elamur.ai/free/en/papers/zhang-2021-trading-off-diversity-quality-natural/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Open-ended generation tasks such as dialogue or storytelling need outputs that are both good and varied, and the decoding algorithm largely determines this balance. Yet there has been no agreed way to compare decoders. The authors treat decoding as a quality–diversity tradeoff. They measure quality as the expected human rating of samples and diversity as conditional Shannon entropy. They then assess decoders across the whole tradeoff curve rather than at single settings. Using GPT-2 (774M), they ran a large crowdsourced study of temperature, top-k and nucleus sampling, each with five hyperparameter values. They compared the methods at matched entropy levels. Separately, they plotted human ratings against model log-likelihood. This confirmed a "likelihood trap": quality rises with likelihood only up to a certain point, after which the most probable texts are judged worse. In high-diversity regimes the three decoders perform about the same. When low entropy and quality are prioritized, nucleus sampling wins. The work shows that fair decoder comparisons must control for diversity, and that maximizing likelihood is a poor proxy for quality.

## Key points
- **Setup:** GPT-2 774M was prompted with the first 20 words of 48 manually filtered GPT-2 test-set examples. Every continuation was fixed at exactly 30 tokens.
- **Scale:** 146 qualified Amazon Mechanical Turk workers rated nearly 10,000 samples from 15 decoder configurations, producing over 38,000 pairwise ratings.
- **Scoring:** Pairwise scores (+1/−1/0) were derived from samples rated together, which reduced variance.
- **Diversity metric:** Entropy as a diversity measure correlates well with n-gram metrics. Spearman correlation was 0.80 with distinct-1 and 0.77 with distinct-2.
- **Ranking:** In the low-entropy, quality-focused regime, nucleus sampling outperformed top-k, which in turn beat temperature sampling. Pure random sampling from the model had the highest entropy and the lowest quality.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 353 works · 364,806 citations
- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 310 works · 284,480 citations
- [University of Pennsylvania](https://md.research.elamur.ai/free/en/universities/university-of-pennsylvania/) — University, Philadelphia, United States · 170 works · 60,739 citations
- [OpenAI](https://md.research.elamur.ai/free/en/companies/openai/) — Company, San Francisco, United States · 13 works · 18,282 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [✍️ Text generation](https://md.research.elamur.ai/free/en/topics/text-generation/) · 145 works

## Cite

Cite as: Hugh Zhang, Daniel Duckworth, Daphne Ippolito and Arvind Neelakantan. 2021. Trading Off Diversity and Quality in Natural Language Generation. Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2021trading,
  title = {{Trading Off Diversity and Quality in Natural Language Generation}},
  author = {Hugh Zhang, Daniel Duckworth, Daphne Ippolito and Arvind Neelakantan},
  booktitle = {Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval)},
  year = {2021},
  url = {https://aclanthology.org/2021.humeval-1.3/}
}
```

- [← previous Are References Really Needed? Unbabel-IST 2021 Submission for the Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/rei-2021-are-references-really-needed-unbabel/)
- [next → Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation](https://md.research.elamur.ai/free/en/papers/levy-2021-collecting-large-scale-gender-bias/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/4/)

## References

16 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein gan. arXiv preprint arXiv:1701.07875. — [arXiv:1701.07875](https://arxiv.org/abs/1701.07875) [find](https://scholar.google.com/scholar?q=Martin+Arjovsky%2C+Soumith+Chintala%2C+and+Le%CC%81on+Bottou%2E+2017%2E+Wasserstein+gan%2E+arXiv+preprint+arXiv%3A1701%2E07875%2E+%E2%80%94+%5BarXiv%3A1701%2E07875%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1701%2E07875%29)
2. Eldan Cohen and J Christopher Beck. 2018. (unconstrained) beam search is sensitive to large search discrepancies. [find](https://scholar.google.com/scholar?q=Eldan+Cohen+and+J+Christopher+Beck%2E+2018%2E+%28unconstrained%29+beam+search+is+sensitive+to+large+search+discrepancies%2E)
3. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+neural+information+processing+systems%2C+pages+3104%E2%80%933112%2E)
4. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833. [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Mike+Lewis%2C+and+Yann+Dauphin%2E+2018%2E+Hierarchical+neural+story+generation%2E+arXiv+preprint+arXiv%3A1805%2E04833%2E)
5. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2016. Show and tell: Lessons learned from the 2015 mscoco image captioning challenge. IEEE transactions on pattern analysis and machine intelligence, 39(4):652–663. [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+Alexander+Toshev%2C+Samy+Bengio%2C+and+Dumitru+Erhan%2E+2016%2E+Show+and+tell%3A+Lessons+learned+from+the+2015+mscoco+image+captioning+challenge%2E+IEEE+transactions+on+pattern+analysis+and+machine)
6. I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS). [find](https://scholar.google.com/scholar?q=I%2E+J%2E+Goodfellow%2C+J%2E+Pouget%2DAbadie%2C+M%2E+Mirza%2C+B%2E+Xu%2C+D%2E+Warde%2DFarley%2C+S%2E+Ozair%2C+A%2E+Courville%2C+and+Y%2E+Bengio%2E+2014%2E+Generative+adversarial+nets%2E+In+Advances+in+Neural+Information+Processing+Systems+%28Ne)
7. Yilin Yang, Liang Huang, and Mingbo Ma. 2018. [Breaking the beam search curse: A study of (re-) scoring methods and stopping criteria for neural machine translation](https://md.research.elamur.ai/free/en/papers/yang-2018-breaking-beam-search-curse-study/). arXiv preprint arXiv:1808.09582.
8. James D Hamilton. 1994. Time series analysis, volume 2. Princeton New Jersey. [find](https://scholar.google.com/scholar?q=James+D+Hamilton%2E+1994%2E+Time+series+analysis%2C+volume+2%2E+Princeton+New+Jersey%2E)
9. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019. [Dialogpt: Large-scale generative pre-training for conversational response generation](https://md.research.elamur.ai/free/en/papers/zhang-2019-dialogpt-large-scale-generative-pre/).
10. Tatsunori B Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying human and statistical evaluation for natural language generation. arXiv preprint arXiv:1904.02792. [find](https://scholar.google.com/scholar?q=Tatsunori+B+Hashimoto%2C+Hugh+Zhang%2C+and+Percy+Liang%2E+2019%2E+Unifying+human+and+statistical+evaluation+for+natural+language+generation%2E+arXiv+preprint+arXiv%3A1904%2E02792%2E)
11. Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. [find](https://scholar.google.com/scholar?q=Ari+Holtzman%2C+Jan+Buys%2C+Maxwell+Forbes%2C+and+Yejin+Choi%2E+2019%2E+The+curious+case+of+neural+text+degeneration%2E+arXiv+preprint+arXiv%3A1904%2E09751%2E)
12. Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. 2019a. [Human and automatic detection of generated text](https://md.research.elamur.ai/free/en/papers/ippolito-2019-automatic-detection-generated-text-easiest/). arXiv preprint arXiv:1911.00650.
13. Daphne Ippolito, Reno Kriz, Maria Kustikova, João Sedoc, and Chris Callison-Burch. 2019b. Comparison of diverse decoding methods from conditional language models. arXiv preprint arXiv:1906.06362. [find](https://scholar.google.com/scholar?q=Daphne+Ippolito%2C+Reno+Kriz%2C+Maria+Kustikova%2C+Joa%CC%83o+Sedoc%2C+and+Chris+Callison%2DBurch%2E+2019b%2E+Comparison+of+diverse+decoding+methods+from+conditional+language+models%2E+arXiv+preprint+arXiv%3A1906%2E06362%2E)
14. Diederik P Kingma and Max Welling. 2013. [Autoencoding variational bayes](https://md.research.elamur.ai/free/en/papers/kingma-2013-auto-encoding-variational-bayes/). arXiv preprint arXiv:1312.6114.
15. J. Li, M. Galley, C. Brockett, J. Gao, and W. B. Dolan. 2016. [A diversity-promoting objective function for neural conversation models](https://md.research.elamur.ai/free/en/papers/li-2016-diversity-promoting-objective-function-neural/). pages 110–119.
16. Clara Meister, Tim Vieira, and Ryan Cotterell. 2020. [If beam search is the answer, what was the question?](https://md.research.elamur.ai/free/en/papers/meister-2020-if-beam-search-is-answer/)

*Updated 24 October 2026*