---
title: "GPQA: A Graduate-Level Google-Proof Q&A — Rein 2023"
description: "GPQA is a benchmark for scalable oversight: methods that let people reliably supervise AI systems that may know more than they do."
authors: ["David Rein", "Betty Li Hou", "Asa Cooper Stickland", "Jackson Petty", "Richard Yuanzhe Pang", "Julien Dirani", "Julian Michael", "Samuel R. Bowman"]
year: 2023
venue: "arXiv:2311.12022"
source: "https://arxiv.org/abs/2311.12022"
doi: "10.48550/arxiv.2311.12022"
arxiv: "2311.12022"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/) / GPQA: A Graduate-Level Google-Proof Q&A Benchmark

# GPQA: A Graduate-Level Google-Proof Q&A Benchmark

**Authors:** David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, [Richard Yuanzhe Pang](https://md.research.elamur.ai/free/en/authors/richard-yuanzhe-pang/), Julien Dirani, [Julian Michael](https://md.research.elamur.ai/free/en/authors/julian-michael/), [Samuel R. Bowman](https://md.research.elamur.ai/free/en/authors/samuel-r-bowman/) · **Year:** 2023 · **Venue:** arXiv:2311.12022 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2311.12022) · [PDF](https://arxiv.org/pdf/2311.12022) · [DOI 10.48550/arxiv.2311.12022](https://doi.org/10.48550/arxiv.2311.12022) · [arXiv 2311.12022](https://arxiv.org/abs/2311.12022) · [OpenAlex](https://openalex.org/W4388926373)

**Citations:** 28 citations (openalex-snapshot · 2026-10-14) · [936](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
GPQA is a benchmark for scalable oversight: methods that let people reliably supervise AI systems that may know more than they do. Such research needs questions with an answer that specialists can verify but that capable outsiders cannot work out, even with web search. The authors hired 61 PhD-level contractors through Upwork to write 448 multiple-choice questions in biology, physics and chemistry. Each question passed through a staged process. The writer drafted it with explanations, a same-field expert answered and critiqued it, and the writer revised it. A second expert then checked it, and three non-experts from other fields tried to solve it using the internet but no LLM assistants. Pay depended mostly on quality bonuses. In-domain experts answered correctly 65% of the time, or roughly 74% after excluding errors they later acknowledged. Non-experts reached only 34% despite spending about 37 minutes per question, and a GPT-4 baseline reached 39%. This gap makes GPQA a realistic testbed for oversight experiments on future, more capable models.

## Key points
- The dataset has 448 four-option questions across biology, physics and chemistry subdomains. Random guessing gives 25% accuracy. Data and code are released on GitHub (idavidrein/gpqa), with a canary string so the questions can be filtered out of training corpora.
- The pipeline has four steps: question writing, a first expert validation, revision by the writer, and a final round with a second expert plus three non-expert validators. The non-experts spent at least 15 minutes per question, about 37 minutes on average.
- The stricter "Diamond" subset keeps questions where both expert validators agree with the intended answer and at most one of three non-experts gets it right.
- Accuracy results:
  - Experts: 65%, or 74% under a conservative objectivity estimate.
  - Skilled non-experts with web access: 34%.
  - GPT-4 with few-shot chain-of-thought prompting: 39%.
- Contributors earned an estimated average of about $95 per hour. The authors note several limitations:
  - The small size limits use for training and gives low statistical power.
  - The non-experts were unusually skilled.
  - Contributors were recruited through Upwork, which may introduce bias.
  - The benchmark is only an imperfect proxy for supervising truly superhuman systems.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Mathematics & statistics](https://md.research.elamur.ai/free/en/shelves/mathematics/)

## Cited by

Cited by

936 papers of our library cite it (of the 140673 whose bibliographies we read); 35 of them have a page here, 901 not yet.

- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/)
- [Phi-4 Technical Report](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-4-technical-report/)
- [Many-Shot In-Context Learning](https://md.research.elamur.ai/free/en/papers/agarwal-2024-many-shot-context-learning/)
- [L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/aggarwal-2025-l1-controlling-long-reasoning-model/)
- [LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks](https://md.research.elamur.ai/free/en/papers/bai-2024-longbench-v2-deeper-understanding/)
- [Qwen2.5-VL Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen2-5-vl-technical/)
- [Qwen3-VL Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen3-vl-technical-report/)
- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2025-towards-reasoning-era-survey-long/)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deepseek-2025-ai-deepseek-r1-incentivizing-reasoning-capability/)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://md.research.elamur.ai/free/en/papers/deepseek-2025-ai-deepseek-v3-2-pushing/)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://md.research.elamur.ai/free/en/papers/deepseek-2026-ai-deepseek-v4-highly-efficient/)
- [Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs](https://md.research.elamur.ai/free/en/papers/gandhi-2025-cognitive-behaviors-that-enable-self/)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-2024-team-gemini-1-5-unlocking-multimodal/)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://md.research.elamur.ai/free/en/papers/glm-2024-chatglm-family-large-language-models/)
- [GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models](https://md.research.elamur.ai/free/en/papers/glm-2025-4-glm-4-5-agentic/)
- [OpenThoughts: Data Recipes for Reasoning Models](https://md.research.elamur.ai/free/en/papers/guha-2025-openthoughts-data-recipes-reasoning-models/)
- [Kimi K2: Open Agentic Intelligence](https://md.research.elamur.ai/free/en/papers/kimi-2025-team-kimi-k2-open-agentic/)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://md.research.elamur.ai/free/en/papers/lambert-2024-tulu-3-pushing-frontiers-open/)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://md.research.elamur.ai/free/en/papers/li-2024-crowdsourced-data-high-quality-benchmarks/)
- [DataComp-LM: In search of the next generation of training sets for language models](https://md.research.elamur.ai/free/en/papers/li-2024-datacomp-lm-search-next-generation/)
- [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://md.research.elamur.ai/free/en/papers/li-2025-search-o1-agentic-search-enhanced/)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://md.research.elamur.ai/free/en/papers/microsoft-2025-phi-4-mini-technical-report/)
- [s1: Simple test-time scaling](https://md.research.elamur.ai/free/en/papers/muennighoff-2025-s1-simple-test-time-scaling/)
- [Large Language Diffusion Models](https://md.research.elamur.ai/free/en/papers/nie-2025-large-language-diffusion-models/)
- [Humanity's Last Exam](https://md.research.elamur.ai/free/en/papers/phan-2025-humanity-s-last-exam/)
- [To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning](https://md.research.elamur.ai/free/en/papers/sprague-2024-cot-cot-chain-thought-helps/)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://md.research.elamur.ai/free/en/papers/wang-2024-openhands-open-platform-ai/)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://md.research.elamur.ai/free/en/papers/wang-2025-internvl3-5-advancing-open/)
- [Qwen2.5-Omni Technical Report](https://md.research.elamur.ai/free/en/papers/xu-2025-qwen2-5-omni-technical/)
- [Qwen3-Omni Technical Report](https://md.research.elamur.ai/free/en/papers/xu-2025-qwen3-omni-technical-report/)
- [Qwen3 Technical Report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/)
- [LIMO: Less is More for Reasoning](https://md.research.elamur.ai/free/en/papers/ye-2025-limo-less-more-reasoning/)
- [Demystifying Long Chain-of-Thought Reasoning in LLMs](https://md.research.elamur.ai/free/en/papers/yeo-2025-demystifying-long-chain-thought-reasoning/)
- [MAmmoTH2: Scaling Instructions from the Web](https://md.research.elamur.ai/free/en/papers/yue-2024-mammoth2-scaling-instructions-web/)
- [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://md.research.elamur.ai/free/en/papers/zeng-2025-simplerl-zoo-investigating-taming-zero/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{rein2023gpqa,
  title = {{GPQA: A Graduate-Level Google-Proof Q\&A Benchmark}},
  author = {Rein, David and Hou, Betty Li and Stickland, Asa Cooper and Petty, Jackson and Pang, Richard Yuanzhe and Dirani, Julien and Michael, Julian and Bowman, Samuel R.},
  year = {2023},
  doi = {10.48550/arxiv.2311.12022},
  eprint = {2311.12022},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.12022}
}
```

- [← previous Gorilla: Large Language Model Connected with Massive APIs](https://md.research.elamur.ai/free/en/papers/patil-2023-gorilla-large-language-model-connected/)
- [next → GPT detectors are biased against non-native English writers](https://md.research.elamur.ai/free/en/papers/liang-2023-gpt-detectors-biased-against-non/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/)

## References

39 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016. URL https://arxiv.org/abs/1606.06565. [↗](https://arxiv.org/abs/1606.06565) [our card](https://md.research.elamur.ai/free/en/papers/amodei-2016-concrete-problems-ai-safety/)
2. S. R. Bowman and G. Dahl. What will it take to fix benchmarking in natural language understanding? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4843–4855, Online, June 2021. Association for Co… [find](https://scholar.google.com/scholar?q=S%2E+R%2E+Bowman+and+G%2E+Dahl%2E+What+will+it+take+to+fix+benchmarking+in+natural+language+understanding%3F+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Association+for+Computatio)
3. S. R. Bowman, J. Hyun, E. Perez, E. Chen, C. Pettit, S. Heiner, K. Lukošiūtė, A. Askell, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, C. Olah, D. Amodei, D. Amodei, D. Drain, D. Li, E. Tran-Johnson, J. Kernion, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, L. Lovitt, N. Elhag… [find](https://scholar.google.com/scholar?q=S%2E+R%2E+Bowman%2C+J%2E+Hyun%2C+E%2E+Perez%2C+E%2E+Chen%2C+C%2E+Pettit%2C+S%2E+Heiner%2C+K%2E+Luko%C5%A1iu%CC%84te%CC%87%2C+A%2E+Askell%2C+A%2E+Jones%2C+A%2E+Chen%2C+A%2E+Goldie%2C+A%2E+Mirhoseini%2C+C%2E+McKinnon%2C+C%2E+Olah%2C+D%2E+Amodei%2C+D%2E+Amodei%2C+D%2E+Drain%2C+D%2E+Li%2C+E%2E+)
4. P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran As… [find](https://scholar.google.com/scholar?q=P%2E+F%2E+Christiano%2C+J%2E+Leike%2C+T%2E+Brown%2C+M%2E+Martic%2C+S%2E+Legg%2C+and+D%2E+Amodei%2E+Deep+reinforcement+learning+from+human+preferences%2E+In+I%2E+Guyon%2C+U%2E+V%2E+Luxburg%2C+S%2E+Bengio%2C+H%2E+Wallach%2C+R%2E+Fergus%2C+S%2E+Vishwanath)
5. A. Cotra. The case for aligning narrowly superhuman models, 2021. URL [https://www.alignmentforum.org/posts/PZtsoaoSLpKjjbMqM/the-case-for-aligning-narrowly-superhuman-models](https://www.alignmentforum.org/posts/PZtsoaoSLpKjjbMqM/the-case-for-aligning-narrowly-superhuman-models). [↗](https://www.alignmentforum.org/posts/PZtsoaoSLpKjjbMqM/the-case-for-aligning-narrowly-superhuman-models)
6. D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=D%2E+Dua%2C+Y%2E+Wang%2C+P%2E+Dasigi%2C+G%2E+Stanovsky%2C+S%2E+Singh%2C+and+M%2E+Gardner%2E+DROP%3A+A+reading+comprehension+benchmark+requiring+discrete+reasoning+over+paragraphs%2E+In+Proceedings+of+the+2019+Conference+of+the+N) [our card](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)
7. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations, 2 Oct. 2020. URL [https://openreview.net/pdf?id=d7KBjmI3GmQ](https://openreview.net/pdf?id=d7KBjmI3GmQ). [↗](https://openreview.net/pdf?id=d7KBjmI3GmQ) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
8. E. Hubinger. AI safety via market making. [https://www.alignmentforum.org/posts/](https://www.alignmentforum.org/posts/) YWwzccGbcHMJMpT45/ai-safety-via-market-making, 26 June 2020. Accessed: 2023-10-24. [↗](https://www.alignmentforum.org/posts/)
9. G. Irving and A. Askell. AI safety needs social scientists. Distill, 4(2):e14, 2019. doi: 10.23915/ distill.00014. URL [https://distill.pub/2019/safety-needs-social-scientists](https://distill.pub/2019/safety-needs-social-scientists). [↗](https://distill.pub/2019/safety-needs-social-scientists)
10. G. Irving, P. Christiano, and D. Amodei. AI safety via debate. arXiv preprint arXiv:1805.00899, 2018. URL https://arxiv.org/abs/1805.00899. [↗](https://arxiv.org/abs/1805.00899)
11. A. Jacovi, A. Caciularu, J. Herzig, R. Aharoni, B. Bohnet, and M. Geva. A comprehensive evaluation of tool-assisted generation strategies. arXiv preprint arXiv:2310.10062, 2023. URL https://arxiv.org/abs/2310.10062. [↗](https://arxiv.org/abs/2310.10062)
12. M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver, Canada, July 2017.… [find](https://scholar.google.com/scholar?q=M%2E+Joshi%2C+E%2E+Choi%2C+D%2E+Weld%2C+and+L%2E+Zettlemoyer%2E+TriviaQA%3A+A+large+scale+distantly+supervised+challenge+dataset+for+reading+comprehension%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+f) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
13. T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 22199–22213. Curran Associates, Inc., 2022. URL h… [find](https://scholar.google.com/scholar?q=T%2E+Kojima%2C+S%2E+S%2E+Gu%2C+M%2E+Reid%2C+Y%2E+Matsuo%2C+and+Y%2E+Iwasawa%2E+Large+language+models+are+zero%2Dshot+reasoners%2E+In+S%2E+Koyejo%2C+S%2E+Mohamed%2C+A%2E+Agarwal%2C+D%2E+Belgrave%2C+K%2E+Cho%2C+and+A%2E+Oh%2C+editors%2C+Advances+in+Neura) [our card](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/)
14. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M.-W. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: A benchmark for question answering research. Transactions of th… [find](https://scholar.google.com/scholar?q=T%2E+Kwiatkowski%2C+J%2E+Palomaki%2C+O%2E+Redfield%2C+M%2E+Collins%2C+A%2E+Parikh%2C+C%2E+Alberti%2C+D%2E+Epstein%2C+I%2E+Polosukhin%2C+J%2E+Devlin%2C+K%2E+Lee%2C+K%2E+Toutanova%2C+L%2E+Jones%2C+M%2E+Kelcey%2C+M%2E%2DW%2E+Chang%2C+A%2E+M%2E+Dai%2C+J%2E+Uszkoreit%2C+Q%2E+L) [our card](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
15. J. Leike, D. Krueger, T. Everitt, M. Martic, V. Maini, and S. Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871, 2018. URL https://arxiv.org/abs/1811.07871. [↗](https://arxiv.org/abs/1811.07871)
16. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. [find](https://scholar.google.com/scholar?q=Y%2E+Liu%2C+M%2E+Ott%2C+N%2E+Goyal%2C+J%2E+Du%2C+M%2E+Joshi%2C+D%2E+Chen%2C+O%2E+Levy%2C+M%2E+Lewis%2C+L%2E+Zettlemoyer%2C+and+V%2E+Stoyanov%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2C+2019%2E)
17. C. Malaviya, S. Lee, S. Chen, E. Sieber, M. Yatskar, and D. Roth. Expertqa: Expert-curated questions and attributed answers. arXiv preprint arXiv:2309.07852, 2023. URL https://arxiv.org/abs/2309. 07852. [↗](https://arxiv.org/abs/2309)
18. J. Michael, S. Mahdi, D. Rein, J. Petty, J. Dirani, V. Padmakumar, and S. R. Bowman. Debate helps supervise unreliable experts. arXiv preprint arXiv:2311.08702, 2023. URL https://arxiv.org/abs/ 2311.08702. — [arXiv:2311.08702](https://arxiv.org/abs/2311.08702) [↗](https://arxiv.org/abs/)
19. T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. URL https://arxiv.org/abs/1301.3781. [↗](https://arxiv.org/abs/1301.3781)
20. OpenAI. Model index for researchers. [https://platform.openai.com/docs/model-index-for-researchers](https://platform.openai.com/docs/model-index-for-researchers). Accessed: 2023-11-18. [↗](https://platform.openai.com/docs/model-index-for-researchers)
21. OpenAI. GPT-4 technical report, 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2E+GPT%2D4+technical+report%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
22. R. Y. Pang, A. Parrish, N. Joshi, N. Nangia, J. Phang, A. Chen, V. Padmakumar, J. Ma, J. Thompson, H. He, and S. Bowman. QuALITY: Question answering with long input texts, yes! In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human… [find](https://scholar.google.com/scholar?q=R%2E+Y%2E+Pang%2C+A%2E+Parrish%2C+N%2E+Joshi%2C+N%2E+Nangia%2C+J%2E+Phang%2C+A%2E+Chen%2C+V%2E+Padmakumar%2C+J%2E+Ma%2C+J%2E+Thompson%2C+H%2E+He%2C+and+S%2E+Bowman%2E+QuALITY%3A+Question+answering+with+long+input+texts%2C+yes%21+In+Proceedings+of+the+2) [our card](https://md.research.elamur.ai/free/en/papers/pang-2022-quality-question-answering-long-input/)
23. R. Y. Pang, A. Parrish, N. Joshi, N. Nangia, J. Phang, A. Chen, V. Padmakumar, J. Ma, J. Thompson, H. He, and S. Bowman. QuALITY: Question answering with long input texts, yes! In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human… [find](https://scholar.google.com/scholar?q=R%2E+Y%2E+Pang%2C+A%2E+Parrish%2C+N%2E+Joshi%2C+N%2E+Nangia%2C+J%2E+Phang%2C+A%2E+Chen%2C+V%2E+Padmakumar%2C+J%2E+Ma%2C+J%2E+Thompson%2C+H%2E+He%2C+and+S%2E+Bowman%2E+QuALITY%3A+Question+answering+with+long+input+texts%2C+yes%21+In+Proceedings+of+the+2) [our card](https://md.research.elamur.ai/free/en/papers/pang-2022-quality-question-answering-long-input/)
24. E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3419–3448, Abu Dhabi, United Arab Emirates, Dec. 2022a… [find](https://scholar.google.com/scholar?q=E%2E+Perez%2C+S%2E+Huang%2C+F%2E+Song%2C+T%2E+Cai%2C+R%2E+Ring%2C+J%2E+Aslanides%2C+A%2E+Glaese%2C+N%2E+McAleese%2C+and+G%2E+Irving%2E+Red+teaming+language+models+with+language+models%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+) [our card](https://md.research.elamur.ai/free/en/papers/perez-2022-red-teaming-language-models-language/)
25. E. Perez, S. Ringer, K. Lukošiūtė, K. Nguyen, E. Chen, S. Heiner, C. Pettit, C. Olsson, S. Kundu, S. Kadavath, A. Jones, A. Chen, B. Mann, B. Israel, B. Seethor, C. McKinnon, C. Olah, D. Yan, D. Amodei, D. Amodei, D. Drain, D. Li, E. Tran-Johnson, G. Khundadze, J. Kernion, J. Landis, J. Kerr, J. M… [find](https://scholar.google.com/scholar?q=E%2E+Perez%2C+S%2E+Ringer%2C+K%2E+Luko%C5%A1iu%CC%84te%CC%87%2C+K%2E+Nguyen%2C+E%2E+Chen%2C+S%2E+Heiner%2C+C%2E+Pettit%2C+C%2E+Olsson%2C+S%2E+Kundu%2C+S%2E+Kadavath%2C+A%2E+Jones%2C+A%2E+Chen%2C+B%2E+Mann%2C+B%2E+Israel%2C+B%2E+Seethor%2C+C%2E+McKinnon%2C+C%2E+Olah%2C+D%2E+Yan%2C+D%2E+Amo) [our card](https://md.research.elamur.ai/free/en/papers/perez-2022-discovering-language-model-behaviors-model/)
26. E. Perez, S. Ringer, K. Lukosiute, K. Nguyen, E. Chen, S. Heiner, C. Pettit, C. Olsson, S. Kundu, S. Kadavath, A. Jones, A. Chen, B. Mann, B. Israel, B. Seethor, C. McKinnon, C. Olah, D. Yan, D. Amodei, D. Amodei, D. Drain, D. Li, E. Tran-Johnson, G. Khundadze, J. Kernion, J. Landis, J. Kerr, J. Mue… [find](https://scholar.google.com/scholar?q=E%2E+Perez%2C+S%2E+Ringer%2C+K%2E+Lukosiute%2C+K%2E+Nguyen%2C+E%2E+Chen%2C+S%2E+Heiner%2C+C%2E+Pettit%2C+C%2E+Olsson%2C+S%2E+Kundu%2C+S%2E+Kadavath%2C+A%2E+Jones%2C+A%2E+Chen%2C+B%2E+Mann%2C+B%2E+Israel%2C+B%2E+Seethor%2C+C%2E+McKinnon%2C+C%2E+Olah%2C+D%2E+Yan%2C+D%2E+Amode) [our card](https://md.research.elamur.ai/free/en/papers/perez-2022-discovering-language-model-behaviors-model/)
27. O. Press, M. Zhang, S. Min, L. Schmidt, N. A. Smith, and M. Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022. URL http://arxiv.org/abs/2210.03350. [↗](http://arxiv.org/abs/2210.03350)
28. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020. URL [http://jmlr.org/papers/v21/20-074.html](http://jmlr.org/papers/v21/20-074.html). [↗](http://jmlr.org/papers/v21/20-074.html)
29. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas, Nov. 2016. Association for Computational Linguistics. doi: 10.1865… [find](https://scholar.google.com/scholar?q=P%2E+Rajpurkar%2C+J%2E+Zhang%2C+K%2E+Lopyrev%2C+and+P%2E+Liang%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Processing) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
30. P. Roit, A. Klein, D. Stepanov, J. Mamou, J. Michael, G. Stanovsky, L. Zettlemoyer, and I. Dagan. Controlled crowdsourcing for high-quality QA-SRL annotation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7008–7013, Online, July 2020. Association f… [find](https://scholar.google.com/scholar?q=P%2E+Roit%2C+A%2E+Klein%2C+D%2E+Stepanov%2C+J%2E+Mamou%2C+J%2E+Michael%2C+G%2E+Stanovsky%2C+L%2E+Zettlemoyer%2C+and+I%2E+Dagan%2E+Controlled+crowdsourcing+for+high%2Dquality+QA%2DSRL+annotation%2E+In+Proceedings+of+the+58th+Annual+Meeting) [our card](https://md.research.elamur.ai/free/en/papers/roit-2020-controlled-crowdsourcing-high-quality-qa/)
31. P. Sen and A. Saffari. What do models learn from question answering datasets? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2429–2438, Stroudsburg, PA, USA, Nov. 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-ma… [find](https://scholar.google.com/scholar?q=P%2E+Sen+and+A%2E+Saffari%2E+What+do+models+learn+from+question+answering+datasets%3F+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pages+2429%E2%80%932438%2C+Stroud)
32. M. Sharma, M. Tong, T. Korbak, D. Duvenaud, A. Askell, S. R. Bowman, N. Cheng, E. Durmus, Z. Hatfield-Dodds, S. R. Johnston, S. Kravec, T. Maxwell, S. McCandlish, K. Ndousse, O. Rausch, N. Schiefer, D. Yan, M. Zhang, and E. Perez. Towards understanding sycophancy in language models. arXiv preprint a… [find](https://scholar.google.com/scholar?q=M%2E+Sharma%2C+M%2E+Tong%2C+T%2E+Korbak%2C+D%2E+Duvenaud%2C+A%2E+Askell%2C+S%2E+R%2E+Bowman%2C+N%2E+Cheng%2C+E%2E+Durmus%2C+Z%2E+Hatfield%2DDodds%2C+S%2E+R%2E+Johnston%2C+S%2E+Kravec%2C+T%2E+Maxwell%2C+S%2E+McCandlish%2C+K%2E+Ndousse%2C+O%2E+Rausch%2C+N%2E+Schiefer%2C+D) [our card](https://md.research.elamur.ai/free/en/papers/sharma-2023-towards-understanding-sycophancy-language-models/)
33. A. Srivastava et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2023. URL https://arxiv.org/abs/2206.04615. [↗](https://arxiv.org/abs/2206.04615) [our card](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
34. A. Talmor, J. Herzig, N. Lourie, and J. Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short P… [find](https://scholar.google.com/scholar?q=A%2E+Talmor%2C+J%2E+Herzig%2C+N%2E+Lourie%2C+and+J%2E+Berant%2E+CommonsenseQA%3A+A+question+answering+challenge+targeting+commonsense+knowledge%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+th)
35. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan,… [find](https://scholar.google.com/scholar?q=H%2E+Touvron%2C+L%2E+Martin%2C+K%2E+Stone%2C+P%2E+Albert%2C+A%2E+Almahairi%2C+Y%2E+Babaei%2C+N%2E+Bashlykov%2C+S%2E+Batra%2C+P%2E+Bhargava%2C+S%2E+Bhosale%2C+D%2E+Bikel%2C+L%2E+Blecher%2C+C%2E+C%2E+Ferrer%2C+M%2E+Chen%2C+G%2E+Cucurull%2C+D%2E+Esiobu%2C+J%2E+Fernandes%2C)
36. A. Wang, R. Y. Pang, A. Chen, J. Phang, and S. R. Bowman. SQuALITY: Building a long-document summarization dataset the hard way. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1139–1156, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Com… [find](https://scholar.google.com/scholar?q=A%2E+Wang%2C+R%2E+Y%2E+Pang%2C+A%2E+Chen%2C+J%2E+Phang%2C+and+S%2E+R%2E+Bowman%2E+SQuALITY%3A+Building+a+long%2Ddocument+summarization+dataset+the+hard+way%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+L)
37. Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, Brussels, Belgium, Oct.-Nov. 2018… [find](https://scholar.google.com/scholar?q=Z%2E+Yang%2C+P%2E+Qi%2C+S%2E+Zhang%2C+Y%2E+Bengio%2C+W%2E+Cohen%2C+R%2E+Salakhutdinov%2C+and+C%2E+D%2E+Manning%2E+HotpotQA%3A+A+dataset+for+diverse%2C+explainable+multi%2Dhop+question+answering%2E+In+Proceedings+of+the+2018+Conference+on+) [our card](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/)
38. W. Yu, Z. Jiang, Y. Dong, and J. Feng. ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning. In International Conference on Learning Representations, 25 Sept. 2019. URL [https://openreview.net/pdf?id=HJgJtT4tvB](https://openreview.net/pdf?id=HJgJtT4tvB). [↗](https://openreview.net/pdf?id=HJgJtT4tvB) [our card](https://md.research.elamur.ai/free/en/papers/yu-2020-reclor-reading-comprehension-dataset-requiring/)
39. Y. Zhang, Y. Li, L. Cui, D. Cai, L. Liu, T. Fu, X. Huang, E. Zhao, Y. Zhang, Y. Chen, et al. Siren’s song in the ai ocean: A survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023. URL https://arxiv.org/abs/2309.01219. *Figure 3: The average of the three non-expert v… [↗](https://arxiv.org/abs/2309.01219) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2023-siren-s-song-ai-ocean/)