---
title: "Moving Beyond the Turing Test with the Allen AI Science Challenge"
description: "The authors argue that the Turing Test no longer works as a benchmark for AI. Its rules are loosely… 17 references, cited by 16 papers here."
authors: ["Carissa Schoenick", "Peter Clark", "Oyvind Tafjord", "Peter Turney", "Oren Etzioni"]
year: 2016
venue: "arXiv:1604.04315"
source: "https://arxiv.org/abs/1604.04315"
doi: "10.48550/arxiv.1604.04315"
arxiv: "1604.04315"
licence: "arXiv non-exclusive distribution license"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/schoenick-2016-moving-turing-test-allen/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/schoenick-2016-moving-turing-test-allen/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/) / Moving Beyond the Turing Test with the Allen AI Science Challenge

# Moving Beyond the Turing Test with the Allen AI Science Challenge

**Authors:** [Carissa Schoenick](https://md.research.elamur.ai/free/en/authors/carissa-schoenick/), [Peter Clark](https://md.research.elamur.ai/free/en/authors/peter-clark/), [Oyvind Tafjord](https://md.research.elamur.ai/free/en/authors/oyvind-tafjord/), [Peter Turney](https://md.research.elamur.ai/free/en/authors/peter-turney/) and [Oren Etzioni](https://md.research.elamur.ai/free/en/authors/oren-etzioni/) · **Year:** 2016 · **Venue:** arXiv:1604.04315 · **Licence:** [arXiv non-exclusive distribution license](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1604.04315) · [PDF](https://arxiv.org/pdf/1604.04315) · [DOI 10.48550/arxiv.1604.04315](https://doi.org/10.48550/arxiv.1604.04315) · [arXiv 1604.04315](https://arxiv.org/abs/1604.04315) · [OpenAlex](https://openalex.org/W2949201464)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [16](https://md.research.elamur.ai/free/en/papers/schoenick-2016-moving-turing-test-allen/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The authors argue that the Turing Test no longer works as a benchmark for AI. Its rules are loosely specified, chatbots can game it by fooling judges, and it yields only a pass/fail verdict instead of a graded score. As a measurable alternative, AI2 ran the Allen AI Science Challenge on Kaggle from October 2015 to February 2016. Systems had to answer four-option multiple-choice 8th grade science questions taken from licensed exam material. To discourage manual answering, the genuine questions were mixed with many machine-generated decoys. All three winning entries depended mostly on information retrieval over carefully curated corpora. They fed signals such as word2vec/GloVe similarity, PMI and features of the question's form into gradient boosting or logistic regression models. The top score was just under 60%. That clearly beats a Lucene-over-Wikipedia baseline but is far from mastering the exam. The paper concludes that retrieval-based methods have hit a ceiling. Further progress will require deeper semantic reasoning and commonsense knowledge, which is the direction of AI2's Project Aristo.

## Key points
- **Participation:** 780 teams took part and 170 submitted final models. The top three received $50,000, $20,000 and $10,000, and winners had to open-source their code.
- **Data:** 5,083 questions were licensed in total, with 2,500 used for training.
  - The validation set had 8,132 items, of which only 800 were real.
  - The test set had 21,298 items, of which 2,583 were real.
  - AI2 later published some subsets on its website.
- **Scores:** Random guessing gives 25%, and the Lucene/Wikipedia baseline reached 40.7% on the test set. The winners scored 59.31% (Chaim Linhart, "Cardal"), 58.34% (Talkwalker, "poweredByTalkwalker") and 58.26% (Alejandro Mosquera), a spread of only 1.05%.
- **Winning methods:**
  - Linhart averaged 15 small gradient boosting models that scored each option separately; by his estimate, IR features alone gave about 55%.
  - Talkwalker built a 180GB Lucene index and used XGBoost pairwise ranking plus hashed term–definition features.
  - Mosquera classified the 12 ordered answer pairs into three classes with logistic regression over Elastic Search features.
- **Open problem:** Questions that every winner missed, such as the earthquakes/plate tectonics and star-brightness examples, require chained reasoning. Existing QA systems do not handle this kind of reasoning, including those from TREC, MUC and FreeBase-based work, as well as Wolfram Alpha, Evi and START.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 353 works · 200,043 citations
- [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) — Research institute, Seattle, United States · 275 works · 97,634 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

💡 Concepts (1): [💡 Artificial intelligence](https://md.research.elamur.ai/free/en/concepts/artificial-intelligence/) · 49 works

## Cite

Cite as: Carissa Schoenick, Peter Clark, Oyvind Tafjord, Peter Turney and Oren Etzioni. 2016. Moving Beyond the Turing Test with the Allen AI Science Challenge. arXiv:1604.04315.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{schoenick2016moving,
  title = {{Moving Beyond the Turing Test with the Allen AI Science Challenge}},
  author = {Carissa Schoenick, Peter Clark, Oyvind Tafjord, Peter Turney and Oren Etzioni},
  year = {2016},
  doi = {10.48550/arxiv.1604.04315},
  eprint = {1604.04315},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1604.04315}
}
```

- [← previous A Character-level Convolutional Neural Network for Distinguishing Similar Languages and Dialects](https://md.research.elamur.ai/free/en/papers/belinkov-2016-character-level-convolutional-neural-network/)
- [next → Understanding the Lexical Simplification Needs of Non-Native Speakers of English](https://md.research.elamur.ai/free/en/papers/paetzold-2016-understanding-lexical-simplification-needs-non/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/)

## References

17 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alan M. Turing. Computing machinery and intelligence. Mind, LIX(236):433–460, October 1950. [find](https://scholar.google.com/scholar?q=Alan+M%2E+Turing%2E+Computing+machinery+and+intelligence%2E+Mind%2C+LIX%28236%29%3A433%E2%80%93460%2C+October+1950%2E)
2. BBC. Computer AI passes Turing test in 'world first' Turing Test. BBC News. 9 June 2014. [http://www.bbc.com/news/technology-27762088](http://www.bbc.com/news/technology-27762088) [↗](http://www.bbc.com/news/technology-27762088)
3. Aron, J. Software tricks people into thinking it is human. New Scientist (Issue 2829), Sept 2011. [find](https://scholar.google.com/scholar?q=Aron%2C+J%2E+Software+tricks+people+into+thinking+it+is+human%2E+New+Scientist+%28Issue+2829%29%2C+Sept+2011%2E)
4. Marcus, G., Rossi, F., Veloso, M. (Eds), Beyond the Turing Test (AI Magazine Special Edition), AI Magazine, 37 (1), Spring 2016. [find](https://scholar.google.com/scholar?q=Marcus%2C+G%2E%2C+Rossi%2C+F%2E%2C+Veloso%2C+M%2E+%28Eds%29%2C+Beyond+the+Turing+Test+%28AI+Magazine+Special+Edition%29%2C+AI+Magazine%2C+37+%281%29%2C+Spring+2016%2E)
5. Turk, V. The Plan to Replace the Turing Test with a ‘Turing Olympics’. Motherboard. 28 January 2015. [http://motherboard.vice.com/read/the-plan-to-replace-the-turing-test-with-a-turing-olympics](http://motherboard.vice.com/read/the-plan-to-replace-the-turing-test-with-a-turing-olympics) [↗](http://motherboard.vice.com/read/the-plan-to-replace-the-turing-test-with-a-turing-olympics)
6. Clark, P., Etzioni, O. My Computer is an Honor Student - But how Intelligent is it? Standardized Tests as a Measure of AI. In AI Magazine 37 (1), Spring 2016. [find](https://scholar.google.com/scholar?q=Clark%2C+P%2E%2C+Etzioni%2C+O%2E+My+Computer+is+an+Honor+Student+%2D+But+how+Intelligent+is+it%3F+Standardized+Tests+as+a+Measure+of+AI%2E+In+AI+Magazine+37+%281%29%2C+Spring+2016%2E)
7. "The Allen AI Science Challenge." Kaggle, 13 Feb. 2016, [https://www.kaggle.com/c/the-allen-ai-science-challenge](https://www.kaggle.com/c/the-allen-ai-science-challenge). [↗](https://www.kaggle.com/c/the-allen-ai-science-challenge)
8. "Data - Allen Institute for Artificial Intelligence." Allen Institute for Artificial Intelligence, [http://allenai.org/data](http://allenai.org/data). Accessed 20 Oct. 2016. [↗](http://allenai.org/data)
9. Grishman, R., Sundheim, B. Message Understanding Conference-6: A Brief History. In COLING (Vol. 96, pp. 466-471), 1996. — [Message Understanding Conference- 6: A Brief History](https://md.research.elamur.ai/free/en/papers/grishman-1996-message-understanding-conference-6-brief/)
10. Voorhees, E., Ellis, A. (Eds)Proc. 24th Text REtrieval Conference (TREC 2015), Publication SP 500-319, NIST (http://trec.nist.gov/ ), 2015. [find](https://scholar.google.com/scholar?q=Voorhees%2C+E%2E%2C+Ellis%2C+A%2E+%28Eds%29Proc%2E+24th+Text+REtrieval+Conference+%28TREC+2015%29%2C+Publication+SP+500%2D319%2C+NIST+%28http%3A%2F%2Ftrec%2Enist%2Egov%2F+%29%2C+2015%2E)
11. Yao, X., Van Durme, B. Information Extraction over Structured Data: Question Answering with Freebase. In ACL (1) (pp. 956-966), 2014. [find](https://scholar.google.com/scholar?q=Yao%2C+X%2E%2C+Van+Durme%2C+B%2E+Information+Extraction+over+Structured+Data%3A+Question+Answering+with+Freebase%2E+In+ACL+%281%29+%28pp%2E+956%2D966%29%2C+2014%2E)
12. Berant, J., Chou, A., Frostig, R., Liang, P. [Semantic Parsing on Freebase from Question-Answer Pairs](https://md.research.elamur.ai/free/en/papers/berant-2013-semantic-parsing-freebase-question-answer/). In EMNLP (Vol. 2, No. 5, p. 6), 2013.
13. Fader, A., Zettlemoyer, L., & Etzioni, O. Open question answering over curated and extracted knowledge bases. In Proc 20th ACM SIGKDD Int Conf on Knowledge Discovery and Data Mining (pp. 1156-1165). ACM, 2014. [find](https://scholar.google.com/scholar?q=Fader%2C+A%2E%2C+Zettlemoyer%2C+L%2E%2C+%26+Etzioni%2C+O%2E+Open+question+answering+over+curated+and+extracted+knowledge+bases%2E+In+Proc+20th+ACM+SIGKDD+Int+Conf+on+Knowledge+Discovery+and+Data+Mining+%28pp%2E+1156%2D1165%29%2E+A)
14. Wolfram, S. Making the World's Data Computable. Proc. Wolfram Data Summit, 2010. (http://blog.stephenwolfram.com/2010/09/making-the-worlds-data-computable/ ) [find](https://scholar.google.com/scholar?q=Wolfram%2C+S%2E+Making+the+World%27s+Data+Computable%2E+Proc%2E+Wolfram+Data+Summit%2C+2010%2E+%28http%3A%2F%2Fblog%2Estephenwolfram%2Ecom%2F2010%2F09%2Fmaking%2Dthe%2Dworlds%2Ddata%2Dcomputable%2F+%29)
15. Simmons, J. True Knowledge: The Natural Language Question Answering Wikipedia for Facts. In: Semantic Focus, Feb 2008. [find](https://scholar.google.com/scholar?q=Simmons%2C+J%2E+True+Knowledge%3A+The+Natural+Language+Question+Answering+Wikipedia+for+Facts%2E+In%3A+Semantic+Focus%2C+Feb+2008%2E)
16. Katz, B., Borchardt, G., Felshin, S. Natural Language Annotations for Question Answering. Proc 19th Int FLAIRS Conference (FLAIRS 2006), 2006. (http://start.csail.mit.edu) [find](https://scholar.google.com/scholar?q=Katz%2C+B%2E%2C+Borchardt%2C+G%2E%2C+Felshin%2C+S%2E+Natural+Language+Annotations+for+Question+Answering%2E+Proc+19th+Int+FLAIRS+Conference+%28FLAIRS+2006%29%2C+2006%2E+%28http%3A%2F%2Fstart%2Ecsail%2Emit%2Eedu%29)
17. Sachan, M., Dubey, A., Xing, E. Science Question Answering using Instructional Materials. arXiv preprint at arXiv:1602.04375 [http://arxiv.org/pdf/1602.04375.pdf](http://arxiv.org/pdf/1602.04375.pdf) [↗](http://arxiv.org/pdf/1602.04375.pdf)

*Updated 29 October 2026*