---
title: "Unifying Human and Statistical Evaluation for Natural Language"
description: "Hashimoto, Zhang and Liang define one target quantity: twice the error of the best classifier that… 42 references, cited by 110 papers here."
authors: ["Tatsunori B. Hashimoto", "Hugh Zhang", "Percy Liang"]
year: 2019
venue: "arXiv:1904.02792"
source: "https://arxiv.org/abs/1904.02792"
doi: "10.48550/arxiv.1904.02792"
arxiv: "1904.02792"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hashimoto-2019-unifying-human-statistical-evaluation-natural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hashimoto-2019-unifying-human-statistical-evaluation-natural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) / Unifying Human and Statistical Evaluation for Natural Language Generation

# Unifying Human and Statistical Evaluation for Natural Language Generation

**Authors:** [Tatsunori B. Hashimoto](https://md.research.elamur.ai/free/en/authors/tatsunori-hashimoto/), [Hugh Zhang](https://md.research.elamur.ai/free/en/authors/hugh-zhang/) and [Percy Liang](https://md.research.elamur.ai/free/en/authors/percy-liang/) · **Year:** 2019 · **Venue:** arXiv:1904.02792 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1904.02792) · [PDF](https://arxiv.org/pdf/1904.02792) · [DOI 10.48550/arxiv.1904.02792](https://doi.org/10.48550/arxiv.1904.02792) · [arXiv 1904.02792](https://arxiv.org/abs/1904.02792) · [OpenAlex](https://openalex.org/W2941169998)

**Citations:** 33 citations (OpenAlex, 2026-10-14) · [110](https://md.research.elamur.ai/free/en/papers/hashimoto-2019-unifying-human-statistical-evaluation-natural/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Evaluating text generators is hard because the usual tools each miss something. Human ratings reward fluent output but cannot tell when a model just copies training data or always produces the same answer. Perplexity rewards coverage but barely punishes occasional bad samples. Hashimoto, Zhang and Liang define one target quantity: twice the error of the best classifier that tries to tell reference text from model output. A perfect model gets 1, and a model with no overlap with the reference gets 0. They prove this optimal classifier needs only two values per sentence: its probability under the model and its probability under the reference. Since the reference probability cannot be computed, they replace it with crowdsourced "typicality" ratings. HUSE is the leave-one-out error of a k-nearest-neighbour classifier over the length-normalised model log-probability and the human score. Experiments on summarization and chit-chat dialogue show two things:
- HUSE exposes diversity problems that pure human evaluation misses.
- Quality-boosting tricks like annealing actually lower HUSE.

## Key points
- The target metric L* equals 1 minus the total variation distance between the model and reference distributions. HUSE is an upper bound on it that is never looser than human-only evaluation: L(φ_hj) ≥ L(φ_huse) ≥ L*.
- The features are log p_model(y|x) divided by sentence length, plus HJ(x, y). HJ is the average 1–5 typicality rating from 20 Amazon Mechanical Turk workers.
- On the Reddit corpus, HJ correlated strongly with a sentence's log-frequency (r = 0.92), which supports using it as a proxy for log p_ref.
- The tasks studied were language modeling, chit-chat dialogue, story generation and summarization.
- Code, data and experiments are released on CodaLab.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 506 works · 330,053 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works · [✍️ Text generation](https://md.research.elamur.ai/free/en/topics/text-generation/) · 214 works

## Cite

Cite as: Tatsunori B. Hashimoto, Hugh Zhang and Percy Liang. 2019. Unifying Human and Statistical Evaluation for Natural Language Generation. arXiv:1904.02792.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{hashimoto2019unifying,
  title = {{Unifying Human and Statistical Evaluation for Natural Language Generation}},
  author = {Tatsunori B. Hashimoto, Hugh Zhang and Percy Liang},
  year = {2019},
  doi = {10.48550/arxiv.1904.02792},
  eprint = {1904.02792},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1904.02792}
}
```

- [← previous Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA](https://md.research.elamur.ai/free/en/papers/jiang-2019-avoiding-reasoning-shortcuts-adversarial-evaluation/)
- [next → Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking](https://md.research.elamur.ai/free/en/papers/broscheit-2019-investigating-entity-knowledge-bert-simple/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/)

## References

42 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. S. Balakrishnan and L. Wasserman. 2017. Hypothesis testing for high-dimensional multinomials: A selective review. arXiv preprint arXiv:1712.06120. — [arXiv:1712.06120](https://arxiv.org/abs/1712.06120) [find](https://scholar.google.com/scholar?q=S%2E+Balakrishnan+and+L%2E+Wasserman%2E+2017%2E+Hypothesis+testing+for+high%2Ddimensional+multinomials%3A+A+selective+review%2E+arXiv+preprint+arXiv%3A1712%2E06120%2E+%E2%80%94+%5BarXiv%3A1712%2E06120%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1712%2E06120)
2. S. R. Bowman, L. Vilnis, O. Vinyals, A. M. Dai, R. Jozefowicz, and S. Bengio. 2016. [Generating sentences from a continuous space](https://md.research.elamur.ai/free/en/papers/bowman-2016-generating-sentences-continuous-space/). In Computational Natural Language Learning (CoNLL), pages 10–21.
3. E. Bruni and R. Fernandez. 2017. Adversarial evaluation for open-domain dialogue generation. In Proceedings of the SIGDIAL 2017 Conference. [find](https://scholar.google.com/scholar?q=E%2E+Bruni+and+R%2E+Fernandez%2E+2017%2E+Adversarial+evaluation+for+open%2Ddomain+dialogue+generation%2E+In+Proceedings+of+the+SIGDIAL+2017+Conference%2E)
4. M. Caccia, L. Caccia, W. Fedus, H. Larochelle, J. Pineau, and L. Charlin. 2018. Language gans falling short. arXiv preprint arXiv:1811.02549. — [arXiv:1811.02549](https://arxiv.org/abs/1811.02549) [find](https://scholar.google.com/scholar?q=M%2E+Caccia%2C+L%2E+Caccia%2C+W%2E+Fedus%2C+H%2E+Larochelle%2C+J%2E+Pineau%2C+and+L%2E+Charlin%2E+2018%2E+Language+gans+falling+short%2E+arXiv+preprint+arXiv%3A1811%2E02549%2E+%E2%80%94+%5BarXiv%3A1811%2E02549%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1811%2E02549%29)
5. A. Chaganty, S. Mussmann, and P. Liang. 2018. The price of debiasing automatic metrics in natural language evaluation. In Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=A%2E+Chaganty%2C+S%2E+Mussmann%2C+and+P%2E+Liang%2E+2018%2E+The+price+of+debiasing+automatic+metrics+in+natural+language+evaluation%2E+In+Association+for+Computational+Linguistics+%28ACL%29%2E)
6. C. L. A. Clarke, M. Kolla, G. V. Cormack, O. Vechtomova, A. Ashkan, S. Büttcher, and I. MacKinnon. 2008. Novelty and diversity in information retrieval evaluation. In ACM SIGIR. [find](https://scholar.google.com/scholar?q=C%2E+L%2E+A%2E+Clarke%2C+M%2E+Kolla%2C+G%2E+V%2E+Cormack%2C+O%2E+Vechtomova%2C+A%2E+Ashkan%2C+S%2E+Bu%CC%88ttcher%2C+and+I%2E+MacKinnon%2E+2008%2E+Novelty+and+diversity+in+information+retrieval+evaluation%2E+In+ACM+SIGIR%2E)
7. M. Feder and N. Merhav. 1994. Relations between entropy and error probability. IEEE Transactions on Information Theory, 40:259–266. [find](https://scholar.google.com/scholar?q=M%2E+Feder+and+N%2E+Merhav%2E+1994%2E+Relations+between+entropy+and+error+probability%2E+IEEE+Transactions+on+Information+Theory%2C+40%3A259%E2%80%93266%2E)
8. M. Galley, C. Brockett, A. Sordoni, Y. Ji, M. Auli, C. Quirk, M. Mitchell, J. Gao, and B. Dolan. 2015. [deltableu: A discriminative metric for generation tasks with intrinsically diverse targets](https://md.research.elamur.ai/free/en/papers/galley-2015-deltableu-discriminative-metric-generation-tasks/). arXiv preprint arXiv:1506.06863.
9. J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin. 2017. [Convolutional sequence to sequence learning](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/). arXiv preprint arXiv:1705.03122.
10. S. Gehrmann, Y. Deng, and A. M. Rush. 2018. [Bottom-up abstractive summarization](https://md.research.elamur.ai/free/en/papers/gehrmann-2018-bottom-up-abstractive-summarization/). In Empirical Methods in Natural Language Processing (EMNLP).
11. I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. 2014. [Generative adversarial nets](https://md.research.elamur.ai/free/en/papers/goodfellow-2014-generative-adversarial-nets/). In Advances in Neural Information Processing Systems (NeurIPS).
12. T. Hashimoto, K. Guu, Y. Oren, and P. Liang. 2018. A retrieve-and-edit framework for predicting structured outputs. In Advances in Neural Information Processing Systems (NeurIPS). [find](https://scholar.google.com/scholar?q=T%2E+Hashimoto%2C+K%2E+Guu%2C+Y%2E+Oren%2C+and+P%2E+Liang%2E+2018%2E+A+retrieve%2Dand%2Dedit+framework+for+predicting+structured+outputs%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2E)
13. S. A. Hayati, R. Olivier, P. Avvaru, P. Yin, A. Tomasic, and G. Neubig. 2018. Retrieval-based neural code generation. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=S%2E+A%2E+Hayati%2C+R%2E+Olivier%2C+P%2E+Avvaru%2C+P%2E+Yin%2C+A%2E+Tomasic%2C+and+G%2E+Neubig%2E+2018%2E+Retrieval%2Dbased+neural+code+generation%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E)
14. M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. 2017. Gans trained by a two timescale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems (NeurIPS). [find](https://scholar.google.com/scholar?q=M%2E+Heusel%2C+H%2E+Ramsauer%2C+T%2E+Unterthiner%2C+B%2E+Nessler%2C+and+S%2E+Hochreiter%2E+2017%2E+Gans+trained+by+a+two+timescale+update+rule+converge+to+a+local+nash+equilibrium%2E+In+Advances+in+Neural+Information+Process)
15. R. Jozefowicz, O. Vinyals, M. Schuster, N. Shazeer, and Y. Wu. 2016. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410. [find](https://scholar.google.com/scholar?q=R%2E+Jozefowicz%2C+O%2E+Vinyals%2C+M%2E+Schuster%2C+N%2E+Shazeer%2C+and+Y%2E+Wu%2E+2016%2E+Exploring+the+limits+of+language+modeling%2E+arXiv+preprint+arXiv%3A1602%2E02410%2E)
16. A. Kannan and O. Vinyals. 2016. Adversarial evaluation of dialogue models. In NIPS 2016 Workshop on Adversarial Training. [find](https://scholar.google.com/scholar?q=A%2E+Kannan+and+O%2E+Vinyals%2E+2016%2E+Adversarial+evaluation+of+dialogue+models%2E+In+NIPS+2016+Workshop+on+Adversarial+Training%2E)
17. G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. M. Rush. 2017. [Opennmt: Open-source toolkit for neural machine translation](https://md.research.elamur.ai/free/en/papers/klein-2017-opennmt-open-source-toolkit-neural/). arXiv preprint arXiv:1701.02810.
18. A. Lavie and M. Denkowski. 2009. The meteor metric for automatic evaluation of machine translation. Machine Translation, 23. [find](https://scholar.google.com/scholar?q=A%2E+Lavie+and+M%2E+Denkowski%2E+2009%2E+The+meteor+metric+for+automatic+evaluation+of+machine+translation%2E+Machine+Translation%2C+23%2E)
19. J. Li, M. Galley, C. Brockett, J. Gao, and W. B. Dolan. 2016. [A diversity-promoting objective function for neural conversation models](https://md.research.elamur.ai/free/en/papers/li-2016-diversity-promoting-objective-function-neural/). In Human Language Technology and North American Association for Computational Linguistics (HLT/NAACL), pages 110–119.
20. J. Li, R. Jia, H. He, and P. Liang. 2018. [Delete, retrieve, generate: A simple approach to sentiment and style transfer](https://md.research.elamur.ai/free/en/papers/li-2018-delete-retrieve-generate-simple-approach/). In North American Association for Computational Linguistics (NAACL).
21. J. Li, W. Monroe, T. Shi, A. Ritter, and D. Jurafsky. 2017. [Adversarial learning for neural dialogue generation](https://md.research.elamur.ai/free/en/papers/li-2017-adversarial-learning-neural-dialogue-generation/). arXiv preprint arXiv:1701.06547.
22. C. Lin and M. Rey. 2004. Looking for a few good metrics: ROUGE and its evaluation. In NTCIR Workshop. [find](https://scholar.google.com/scholar?q=C%2E+Lin+and+M%2E+Rey%2E+2004%2E+Looking+for+a+few+good+metrics%3A+ROUGE+and+its+evaluation%2E+In+NTCIR+Workshop%2E)
23. T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll’ar, and C. L. Zitnick. 2014. [Microsoft COCO: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In European Conference on Computer Vision (ECCV), pages 740–755.
24. C. Liu, R. Lowe, I. V. Serban, M. Noseworthy, L. Charlin, and J. Pineau. 2016. [How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation](https://md.research.elamur.ai/free/en/papers/liu-2016-how-not-evaluate-your-dialogue/). In Empirical Methods in Natural Language Processing (EMNLP).
25. R. Lowe, M. Noseworthy, I. V. Serban, N. Angelard-Gontier, Y. Bengio, and J. Pineau. 2017. [Towards an automatic turing test: Learning to evaluate dialogue responses](https://md.research.elamur.ai/free/en/papers/lowe-2017-towards-automatic-turing-test-learning/). In Association for Computational Linguistics (ACL).
26. N. Mostafazadeh, N. Chambers, X. He, D. Parikh, D. Batra, L. Vanderwende, P. Kohli, and J. Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In North American Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=N%2E+Mostafazadeh%2C+N%2E+Chambers%2C+X%2E+He%2C+D%2E+Parikh%2C+D%2E+Batra%2C+L%2E+Vanderwende%2C+P%2E+Kohli%2C+and+J%2E+Allen%2E+2016%2E+A+corpus+and+cloze+evaluation+for+deeper+understanding+of+commonsense+stories%2E+In+North+American)
27. R. Nallapati, B. Zhou, C. Gulcehre, B. Xiang, et al. 2016. [Abstractive text summarization using sequence-to-sequence rnns and beyond](https://md.research.elamur.ai/free/en/papers/nallapati-2016-abstractive-text-summarization-sequence-sequence/). arXiv preprint arXiv:1602.06023.
28. A. Nenkova, R. J. Passonneau, and K. McKeown. 2007. The pyramid method: Incorporating human content selection variation in summarization evaluation. In ACM Transactions on Speech and Language Processing. [find](https://scholar.google.com/scholar?q=A%2E+Nenkova%2C+R%2E+J%2E+Passonneau%2C+and+K%2E+McKeown%2E+2007%2E+The+pyramid+method%3A+Incorporating+human+content+selection+variation+in+summarization+evaluation%2E+In+ACM+Transactions+on+Speech+and+Language+Processi)
29. J. Novikova, O. Dušek, A. C. Curry, and V. Rieser. 2017. [Why we need new evaluation metrics for NLG](https://md.research.elamur.ai/free/en/papers/novikova-2017-why-we-need-new-evaluation/). In Empirical Methods in Natural Language Processing (EMNLP).
30. C. Olsson, S. Bhupatiraju, T. Brown, A. Odena, and I. Goodfellow. 2018. Skill rating for generative models. arXiv preprint arXiv:1808.04888. — [arXiv:1808.04888](https://arxiv.org/abs/1808.04888) [find](https://scholar.google.com/scholar?q=C%2E+Olsson%2C+S%2E+Bhupatiraju%2C+T%2E+Brown%2C+A%2E+Odena%2C+and+I%2E+Goodfellow%2E+2018%2E+Skill+rating+for+generative+models%2E+arXiv+preprint+arXiv%3A1808%2E04888%2E+%E2%80%94+%5BarXiv%3A1808%2E04888%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1808%2E04888%29)
31. K. Papineni, S. Roukos, T. Ward, and W. Zhu. 2002. [BLEU: A method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Association for Computational Linguistics (ACL).
32. M. Roemmele. 2016. Writing stories with help from recurrent neural networks. In Association for the Advancement of Artificial Intelligence (AAAI). [find](https://scholar.google.com/scholar?q=M%2E+Roemmele%2E+2016%2E+Writing+stories+with+help+from+recurrent+neural+networks%2E+In+Association+for+the+Advancement+of+Artificial+Intelligence+%28AAAI%29%2E)
33. M. S. M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and S. Gelly. 2018. Assessing generative models via precision and recall. arXiv preprint arXiv:1806.00035. — [arXiv:1806.00035](https://arxiv.org/abs/1806.00035) [find](https://scholar.google.com/scholar?q=M%2E+S%2E+M%2E+Sajjadi%2C+O%2E+Bachem%2C+M%2E+Lucic%2C+O%2E+Bousquet%2C+and+S%2E+Gelly%2E+2018%2E+Assessing+generative+models+via+precision+and+recall%2E+arXiv+preprint+arXiv%3A1806%2E00035%2E+%E2%80%94+%5BarXiv%3A1806%2E00035%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
34. H. Shima and T. Mitamura. 2011. Diversity-aware evaluation for paraphrase patterns. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=H%2E+Shima+and+T%2E+Mitamura%2E+2011%2E+Diversity%2Daware+evaluation+for+paraphrase+patterns%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E)
35. S. Singh, A. Uppal, B. Li, C. Li, M. Zaheer, and B. Poczos. 2018. Nonparametric density estimation under adversarial losses. In Advances in Neural Information Processing Systems (NeurIPS), pages 246–257. [find](https://scholar.google.com/scholar?q=S%2E+Singh%2C+A%2E+Uppal%2C+B%2E+Li%2C+C%2E+Li%2C+M%2E+Zaheer%2C+and+B%2E+Poczos%2E+2018%2E+Nonparametric+density+estimation+under+adversarial+losses%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2C+pages+246%E2%80%932)
36. A. Sordoni, M. Galley, M. Auli, C. Brockett, Y. Ji, M. Mitchell, J. Nie, J. Gao, and B. Dolan. 2015. [A neural network approach to context-sensitive generation of conversational responses](https://md.research.elamur.ai/free/en/papers/sordoni-2015-neural-network-approach-context-sensitive/). In North American Association for Computational Linguistics (NAACL).
37. H. Sun and M. Zhou. 2012. Joint learning of a dual SMT system for paraphrase generation. In Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=H%2E+Sun+and+M%2E+Zhou%2E+2012%2E+Joint+learning+of+a+dual+SMT+system+for+paraphrase+generation%2E+In+Association+for+Computational+Linguistics+%28ACL%29%2E)
38. J. Surowiecki. 2004. The wisdom of crowds: Why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations. Doubleday and Co. [find](https://scholar.google.com/scholar?q=J%2E+Surowiecki%2E+2004%2E+The+wisdom+of+crowds%3A+Why+the+many+are+smarter+than+the+few+and+how+collective+wisdom+shapes+business%2C+economies%2C+societies%2C+and+nations%2E+Doubleday+and+Co%2E)
39. L. Theis, A. van den Oord, and M. Bethge. 2015. A note on the evaluation of generative models. arXiv preprint arXiv:1511.01844. — [arXiv:1511.01844](https://arxiv.org/abs/1511.01844) [find](https://scholar.google.com/scholar?q=L%2E+Theis%2C+A%2E+van+den+Oord%2C+and+M%2E+Bethge%2E+2015%2E+A+note+on+the+evaluation+of+generative+models%2E+arXiv+preprint+arXiv%3A1511%2E01844%2E+%E2%80%94+%5BarXiv%3A1511%2E01844%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1511%2E01844%29)
40. I. Tolstikhin, B. K. Sriperumbudur, and B. Scholkopf. 2016. Minimax estimation of maximum mean discrepancy with radial kernels. In Advances in Neural Information Processing Systems (NeurIPS), pages 1930–1938. [find](https://scholar.google.com/scholar?q=I%2E+Tolstikhin%2C+B%2E+K%2E+Sriperumbudur%2C+and+B%2E+Scholkopf%2E+2016%2E+Minimax+estimation+of+maximum+mean+discrepancy+with+radial+kernels%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2C+pages+19)
41. A. M. Turing. 1950. Computing machinery and intelligence. Mind, 49:433–460. [find](https://scholar.google.com/scholar?q=A%2E+M%2E+Turing%2E+1950%2E+Computing+machinery+and+intelligence%2E+Mind%2C+49%3A433%E2%80%93460%2E)
42. L. Ungar, B. Mellors, V. Satopää, J. Baron, P. Tetlock, J. Ramos, and S. Swift. 2012. The good judgment project: A large scale test of different methods of combining expert predictions. In Association for the Advancement of Artificial Intelligence (AAAI). [find](https://scholar.google.com/scholar?q=L%2E+Ungar%2C+B%2E+Mellors%2C+V%2E+Satopa%CC%88a%CC%88%2C+J%2E+Baron%2C+P%2E+Tetlock%2C+J%2E+Ramos%2C+and+S%2E+Swift%2E+2012%2E+The+good+judgment+project%3A+A+large+scale+test+of+different+methods+of+combining+expert+predictions%2E+In+Associati)

*Updated 2 November 2026*