---
title: "SimpleQA Verified — paper by Haas et al. (2025)"
description: "OpenAI's SimpleQA became a standard test of whether LLMs can recall short facts from their own… 29 references, cited by 11 papers here."
authors: ["Lukas Haas", "Gal Yona", "Giovanni D'Antonio", "Sasha Goldshtein", "Dipanjan Das"]
year: 2025
venue: "arXiv:2509.07968"
source: "https://arxiv.org/abs/2509.07968"
doi: "10.48550/arxiv.2509.07968"
arxiv: "2509.07968"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/haas-2025-simpleqa-verified-reliable-factuality-benchmark/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/haas-2025-simpleqa-verified-reliable-factuality-benchmark/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/) / SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge

# SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge

**Authors:** [Lukas Haas](https://md.research.elamur.ai/free/en/authors/lukas-haas/), [Gal Yona](https://md.research.elamur.ai/free/en/authors/gal-yona/), Giovanni D'Antonio, Sasha Goldshtein and [Dipanjan Das](https://md.research.elamur.ai/free/en/authors/dipanjan-das/) · **Year:** 2025 · **Venue:** arXiv:2509.07968 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2509.07968) · [PDF](https://arxiv.org/pdf/2509.07968) · [DOI 10.48550/arxiv.2509.07968](https://doi.org/10.48550/arxiv.2509.07968) · [arXiv 2509.07968](https://arxiv.org/abs/2509.07968) · [OpenAlex](https://openalex.org/W4416068860)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [11](https://md.research.elamur.ai/free/en/papers/haas-2025-simpleqa-verified-reliable-factuality-benchmark/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
OpenAI's SimpleQA became a standard test of whether LLMs can recall short facts from their own weights. It has flaws, though. Some reference answers are wrong, individual raters contributed clusters of near-identical questions, and dates, person names and science/technology topics dominate.

Researchers at Google DeepMind and Google Research rebuilt it as SimpleQA Verified, a 1,000-question subset. Their pipeline:
- kept a single question per source URL,
- removed near-duplicates using Gemini Embeddings and TF-IDF similarity,
- dropped items whose sources block AI-training crawlers via robots.txt,
- rebalanced topics and answer types,
- resolved conflicting sources and gave numeric answers tolerance ranges,
- preferred questions that frontier models get wrong.

They also revised the grading prompt so that responses listing several hedged guesses count as not attempted.

Models were evaluated without tools. Gemini 2.5 Pro leads with an F1 of 55.6, ahead of GPT-5 (52.3) and o3 (51.9). The released dataset, code and Kaggle leaderboard give a cleaner signal of real progress in parametric factuality and make it harder to gain by overfitting to dataset quirks.

## Key points
- Curation shrank the dataset from 4,326 to 1,000 items (−76.9%). The robots.txt publisher filter alone removed about 30% (2,664 → 1,855).
- De-duplication used a cosine cutoff of 0.77 on Gemini Embeddings, then TF-IDF similarity above 0.4 with manual review. The original set had, for example, 119 questions about founding dates of Colombian municipalities.
- Numeric answers received acceptable error margins (roughly ±1% or ±5% depending on magnitude). The autorater prompt (run with gpt-4.1-2025-04-14) grew from 963 to 1,124 words.
- 13 models were compared. GPT-4o, Claude Opus 4 and Claude Sonnet 4 score significantly lower than on SimpleQA, while o4-mini improves. Claude models abstain often; Opus 4 attempts only 35.5% of questions.
- With search tools enabled, scores approach perfection, so the benchmark is intended strictly for tool-free evaluation. It is hosted on Kaggle (deepmind/simpleqa-verified).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 430 works · 424,363 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Lukas Haas, Gal Yona, Giovanni D'Antonio, Sasha Goldshtein and Dipanjan Das. 2025. SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge. arXiv:2509.07968.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{haas2025simpleqa,
  title = {{SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge}},
  author = {Lukas Haas, Gal Yona, Giovanni D'Antonio, Sasha Goldshtein and Dipanjan Das},
  year = {2025},
  doi = {10.48550/arxiv.2509.07968},
  eprint = {2509.07968},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2509.07968}
}
```

- [← previous SAEs Are Good for Steering – If You Select the Right Features](https://md.research.elamur.ai/free/en/papers/arad-2025-saes-are-good-steering-if/)
- [next → SKIntern: Internalizing Symbolic Knowledge for Distilling Better CoT Capabilities into Small Language Models](https://md.research.elamur.ai/free/en/papers/liao-2025-skintern-internalizing-symbolic-knowledge-distilling/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/)

## References

29 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anthropic. Introducing claude 4, 2025a. URL [https://www.anthropic.com/news/claude-4](https://www.anthropic.com/news/claude-4) . [↗](https://www.anthropic.com/news/claude-4)
2. Anthropic. Claude 3.7 Sonnet and Claude Code, 2025b. URL [https://www.anthropic.com/](https://www.anthropic.com/) news/claude-3-7-sonnet . [↗](https://www.anthropic.com/)
3. S. Chen, Y. Zhao, J. Zhang, I.-C. Chern, S. Gao, P. Liu, and J. He. Felm: Benchmarking factuality evaluation of large language models. arXiv preprint arXiv:2310.00741, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Chen%2C+Y%2E+Zhao%2C+J%2E+Zhang%2C+I%2E%2DC%2E+Chern%2C+S%2E+Gao%2C+P%2E+Liu%2C+and+J%2E+He%2E+Felm%3A+Benchmarking+factuality+evaluation+of+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E00741%2C+2023%2E)
4. DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Lu… — [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/)
5. Gemini Team, Google. [Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/). arXiv preprint arXiv:2507.06261, 2025.
6. M. Geva, D. Khashabi, E. Segal, T. Khot, D. Roth, and J. Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. arXiv preprint arXiv:2101.02235, 2021. [find](https://scholar.google.com/scholar?q=M%2E+Geva%2C+D%2E+Khashabi%2C+E%2E+Segal%2C+T%2E+Khot%2C+D%2E+Roth%2C+and+J%2E+Berant%2E+Did+aristotle+use+a+laptop%3F+a+question+answering+benchmark+with+implicit+reasoning+strategies%2E+arXiv+preprint+arXiv%3A2101%2E02235%2C+2021%2E)
7. A. Jacovi, A. Wang, C. Alberti, C. Tao, J. Lipovetz, K. Olszewska, L. Haas, M. Liu, N. Keating, A. Bloniarz, C. Saroufim, C. Fry, D. Marcus, D. Kukliansky, G. S. Tomar, J. Swirhun, J. Xing, L. Wang, M. Gurumurthy, M. Aaron, M. Ambar, R. Fellinger, R. Wang, Z. Zhang, S. Goldshtein, and D. Das. The fa… — [The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input](https://md.research.elamur.ai/free/en/papers/jacovi-2025-facts-grounding-leaderboard-benchmarking-llms/)
8. K. S. Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28:11–21, 1972. [find](https://scholar.google.com/scholar?q=K%2E+S%2E+Jones%2E+A+statistical+interpretation+of+term+specificity+and+its+application+in+retrieval%2E+Journal+of+Documentation%2C+28%3A11%E2%80%9321%2C+1972%2E)
9. M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. [TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611. Association for Computational…
10. J. Kasai, K. Sakaguchi, Y. Takahashi, R. L. Bras, A. Asai, X. Yu, D. Radev, N. A. Smith, Y. Choi, and K. Inui. Realtime qa: What’s the answer right now? arXiv preprint arXiv:2207.[13332, 2024](https://md.research.elamur.ai/free/en/papers/kasai-2022-realtime-qa-s-answer-right/).
11. S. Krishna, K. Krishna, A. Mohananey, S. Schwarcz, A. Stambler, S. Upadhyay, and M. Faruqui. Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation. arXiv preprint arXiv:2409.12941, 2025. [find](https://scholar.google.com/scholar?q=S%2E+Krishna%2C+K%2E+Krishna%2C+A%2E+Mohananey%2C+S%2E+Schwarcz%2C+A%2E+Stambler%2C+S%2E+Upadhyay%2C+and+M%2E+Faruqui%2E+Fact%2C+fetch%2C+and+reason%3A+A+unified+evaluation+of+retrieval%2Daugmented+generation%2E+arXiv+preprint+arXiv%3A2409%2E)
12. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M.-W. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov. [Natural questions: A benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of th…
13. J. Lee, F. Chen, S. Dua, D. Cer, M. Shanbhogue, I. Naim, G. H. Ábrego, Z. Li, K. Chen, H. S. Vera, X. Ren, S. Zhang, D. Salz, M. Boratko, J. Han, B. Chen, S. Huang, V. Rao, P. Suganthan, F. Han, A. Doumanoglou, N. Gupta, F. Moiseev, C. Yip, A. Jain, S. Baumgartner, S. Shahi, F. P. Gomez, S. Mariserl… — [Gemini Embedding: Generalizable Embeddings from Gemini](https://md.research.elamur.ai/free/en/papers/lee-2025-gemini-embedding-generalizable-embeddings-gemini/)
14. S. Lin, J. Hilton, and O. Evans. [Truthfulqa: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). arXiv preprint arXiv:2109.07958, 2022.
15. S. Min, K. Krishna, X. Lyu, M. Lewis, W.-t. Yih, P. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi. [FActScore: Fine-grained atomic evaluation of factual precision in long form text generation](https://md.research.elamur.ai/free/en/papers/min-2023-factscore-fine-grained-atomic-evaluation/). In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in…
16. OpenAI. [Gpt-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). arXiv preprint arXiv:2410.21276, 2024a.
17. OpenAI. Learning to reason with LLMs, 2024b. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Learning+to+reason+with+LLMs%2C+2024b%2E)
18. URL [https://openai.com/index/learning-to-reason-with-llms](https://openai.com/index/learning-to-reason-with-llms) . OpenAI. Introducing gpt-4.1 in the api, 2025a. URL https://openai.com/index/gpt-4-1/ . [↗](https://openai.com/index/learning-to-reason-with-llms)
19. OpenAI. Gpt-5 system card, 2025b. URL — [OpenAI GPT-5 System Card](https://md.research.elamur.ai/free/en/papers/singh-2025-openai-gpt-5-system-card/) [https://openai.com/index/gpt-5-system-card/](https://openai.com/index/gpt-5-system-card/) . [↗](https://openai.com/index/gpt-5-system-card/)
20. L. Pan, X. Wu, X. Lu, A. T. Luu, W. Y. Wang, M.-Y. Kan, and P. Nakov. [Fact-checking complex claims with program-guided reasoning](https://md.research.elamur.ai/free/en/papers/pan-2023-fact-checking-complex-claims-program/). arXiv preprint arXiv:2305.12744, 2023.
21. S. Pichai, D. Hassabis, and K. Kavukcuoglu. Introducing Gemini 2.0: our new ai model for the agentic era, 2024. URL [https://blog.google/technology/google-deepmind/](https://blog.google/technology/google-deepmind/) google-gemini-ai-update-december-2024 . [↗](https://blog.google/technology/google-deepmind/)
22. H. Rashkin, V. Nikolaev, M. Lamm, L. Aroyo, M. Collins, D. Das, S. Petrov, G. S. Tomar, I. Turc, and D. Reitter. [Measuring attribution in natural language generation models](https://md.research.elamur.ai/free/en/papers/rashkin-2023-measuring-attribution-natural-language-generation/). arXiv preprint arXiv:2112.12870, 2022.
23. Y. Song, Y. Kim, and M. Iyyer. [Veriscore: Evaluating the factuality of verifiable claims in long-form text generation](https://md.research.elamur.ai/free/en/papers/song-2024-veriscore-evaluating-factuality-verifiable-claims/). arXiv preprint arXiv:2406.19276, 2024.
24. T. Vu, M. Iyyer, X. Wang, N. Constant, J. Wei, J. Wei, C. Tar, Y.-H. Sung, D. Zhou, Q. Le, and T. Luong. [Freshllms: Refreshing large language models with search engine augmentation](https://md.research.elamur.ai/free/en/papers/vu-2023-freshllms-refreshing-large-language-models/). arXiv preprint arXiv:2310.03214, 2023.
25. J. Wei, N. Karina, H. W. Chung, Y. J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus. [Measuring short-form factuality in large language models](https://md.research.elamur.ai/free/en/papers/wei-2024-measuring-short-form-factuality-large/). arXiv preprint arXiv:2411.04368, 2024a.
26. J. Wei, C. Yang, X. Song, Y. Lu, N. Hu, J. Huang, D. Tran, D. Peng, R. Liu, D. Huang, C. Du, and Q. V. Le. [Long-form factuality in large language models](https://md.research.elamur.ai/free/en/papers/wei-2024-long-form-factuality-large-language/). arXiv preprint arXiv:2403.18802, 2024b.
27. X. Yang, K. Sun, H. Xin, Y. Sun, N. Bhalla, X. Chen, S. Choudhary, R. D. Gui, Z. W. Jiang, Z. Jiang, L. Kong, B. Moran, J. Wang, Y. E. Xu, A. Yan, C. Yang, E. Yuan, H. Zha, N. Tang, L. Chen, N. Scheffer, Y. Liu, N. Shah, R. Wanga, A. Kumar, W. tau Yih, and X. L. Dong. Crag – comprehensive rag benchm… — [arXiv:2406.04744](https://arxiv.org/abs/2406.04744) [find](https://scholar.google.com/scholar?q=X%2E+Yang%2C+K%2E+Sun%2C+H%2E+Xin%2C+Y%2E+Sun%2C+N%2E+Bhalla%2C+X%2E+Chen%2C+S%2E+Choudhary%2C+R%2E+D%2E+Gui%2C+Z%2E+W%2E+Jiang%2C+Z%2E+Jiang%2C+L%2E+Kong%2C+B%2E+Moran%2C+J%2E+Wang%2C+Y%2E+E%2E+Xu%2C+A%2E+Yan%2C+C%2E+Yang%2C+E%2E+Yuan%2C+H%2E+Zha%2C+N%2E+Tang%2C+L%2E+Chen%2C+N%2E+Scheff)
28. Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning. [Hotpotqa: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). arXiv preprint arXiv:1809.09600, 2018.
29. A. SimpleQA Verified Evaluation Prompt The following is the updated SimpleQA Verified evaluation prompt, where differences from the original SimpleQA (Wei et al., 2024a) prompt are highlighted in red. [find](https://scholar.google.com/scholar?q=A%2E+SimpleQA+Verified+Evaluation+Prompt+The+following+is+the+updated+SimpleQA+Verified+evaluation+prompt%2C+where+differences+from+the+original+SimpleQA+%28Wei+et+al%2E%2C+2024a%29+prompt+are+highlighted+in+red%2E)

*Updated 31 October 2026*